Hi, can everyone hear me? If you can hear me in the back, can you raise your hand? Great. Thank you. Wow Now ten jumping jacks, please Well, good afternoon everyone the title of our session today is changing the tires at 60 miles per hour how Martha Stewart living Omni Media migrated to Drupal My name is Ira Tao. I'm the VP of Internet Technology at Martha Stewart living Omni Media and In today's presentation We are going to talk about how we spent the last two years migrating from our old legacy CMS system Which was on vignette v7 or as some people like to refer to it as vcm To a brand new infrastructure and implementation built on Drupal 6 and Drupal 7 So we're going to be talking about moving data deploying code our content model workflow customizations specialized theme management and Dealing with a high-traffic website on Drupal 6 We're going to be doing a broad technical overview So we're not going to be delving too deeply in any one specific area because obviously we want to cover as much as possible But we are here all week and we certainly love to talk about this stuff We like lots of questions. So anything that you want to learn more about Please approach any of us and feel free to ask us anything you'd like And if you feel you're in the wrong session, which happens We are not insulted. Please feel free to go and find another session while you still have the chance so I'm going to put up our Twitter hashtag while I introduce my team to my right is John Reynolds John is the AVP of Internet and CMS Internet CMS and application development Next to him is Christopher Whitholm who is our manager of CMS development Mark Dorison is our manager of application development Terrence Petrie or TJ as we like to call him is our manager of front-end development and Our special guest star today is James Sandsbury also known as Corbin of lullabot fame So to be clear, we are talking about three websites today Martha Stewart comm Martha Stewart weddings comm and whole living comm and each of them are running on a press flow installation using Drupal 6 multi sites Just some quick stats we serve During our off-peak season, which basically is January through October Approximately seven to ten million page views a day During our high season, which as you can imagine begins like two weeks before Halloween and goes through I guess the day after Christmas. We serve somewhere between ten and fourteen million page views a day So the first question one would ask Why why would we undertake such an endeavor? Well, it really boiled down to our old CMS and I'm not gonna, you know Go into too much detail around that, but I will tell you that there's a whole laundry list of reasons Why we didn't like our old CMS? But the bottom line is is that it couldn't support our business It couldn't support the initiatives that we needed to undertake to keep up with the changing world and the changing environment out there and of course Drupal is awesome so How did we wrap our heads around a project of this magnitude? The first question that we needed to ask ourselves really was what are we building? When we started we had no idea what the front-end vision was to be Our business owners were still trying to figure that out our design people were still trying to figure that out We knew that there would be some design tweaks from the existing website But we weren't sure what those would be what we did know was that it wouldn't be a full-on redesign So we thought okay great. We're at a terrific advantage here We can build something really cool something totally abstract from the inside out with absolutely no requirements that can handle anything Well the disadvantage of that is we didn't know what we were building so It was very difficult to contain the scope and we still needed to prioritize Strategize organize dice and slice this project into digestible and implementable units that actually made sense We also knew that we wanted to embrace Agile methodologies as much as possible which was very very new to the company We had no experience at Martha Stewart with agile whatsoever So while everyone was deciding what they wanted We decided to go ahead and just take some action and we started by looking at our old CMS and Documenting all of our content types and taking the full content inventory of what existed on the site We also Documented the functionality at a high level what each page did what each content type was responsible for We analyzed our internal processes. We looked at how people worked We looked at how data flowed around the organization and how it was organized and how people managed it on a day-to-day basis We also engaged the experts Not only has James been with our project since the very start, but a big shout-out goes to the Wonderful and talented Karen Stevenson who was with us from day one as well as Eric Duran Seth Brown Matt Westgate and Jeff Robbins and We also need to Acknowledge that North Point Solutions was our partner as well in this project So a big thank you goes to Jeff Penner, Rich Cooley and Matt Dorman So the way we proceeded from this point was to create what we called Vertical and horizontal sub projects now. What does that mean? So what we consider a vertical project is something that is related to very focused functionality or a content type recipes Photo galleries channel landing pages things that can be described in very discrete ways What we considered Horizontal sub projects were things that we knew had to span across each and every one of those other sub projects So add integration Omniture integration Social actions workflow. These are just examples and we had this huge grid that represented all of the sub projects of our entire larger replatforming project So the next really critical part of our analysis phase was drawing a roadmap towards Actually doing this actually getting content functionality off of one legacy CMS platform and onto the new one So In this part of the analysis we we regarded these these considerations and facts One is we wanted to show continuous progress We wanted to and we were we were compelled to by our constituents They we didn't want to go off into a cave and build something and emerge You know a few years later as heroes and liberators that generally doesn't work too well So we wanted to continuously put stuff into production and let everybody know it publicize it We also wanted to very early on socialized Drupal internally We wanted the editors to see the interface we wanted to be completely transparent This was again part of you know sort of a cultural coup to get buy-in on on the project We need to mitigate risk so I were mentioned the page views we get there's ads on those pages We're really not allowed to have any downtime. It can obviously impact revenue Impact the user experience impact the editorial flow So we needed to have some kind of strategy that would make sure that that happened as little as possible or not at all We want to keep the developers psyched again This goes into the going to cave is is no fun for developers Nobody likes to think they're working on vaporware So we wanted to continually kind of buys into the whole idea of continuous progress keeping developers energized by keeping their work going into production And we wanted to keep the people are writing the checks happy. This is part of any project. I think So all of this led us to an inevitable conclusion, which is we had no illusions that we could simply forklift Forklift the forklift One site from one platform to another and this picture of a forklift illustrates that point So what we decided to do Was migrate iteratively and this kind of became the rallying cry for the project iterative iterative iterative So what does that mean that means we resolved to maintain both platforms Drupal and vignette We resolved to move one content at a time from the old platform to the new while keeping both stacks up and running And Finally all the stuff we don't know we knew that there would be the load balancer would play a critical role in Balancing between those two platforms. We knew redirects would be involved Did we have everything figured out the very beginning as much as we could but no and we just decided that was okay Hey, we're agile now. So We we had enough faith in the premise that we wanted to move it forward. So with that We were ready to start actually building the site migrating data and my colleagues are going to talk about that I'm going to come back a little bit later and talk about how we actually what happened when we really start routing traffic to that dual platform system Hi, so when I first joined the team There was a lot of homework that had already been done and I can't stress that enough that even though we were working iteratively and weren't really quite sure what our final product was going to be There was a lot of research that had gone into this a lot of inventory taking the content types had been studied And I was part of a team that worked on Documenting all that what fields do we have in each content type? What fields do we want in our final model as far as we know now although that's going to change? so it came time to actually start to dig in and When you're ready to go start going right away Don't wait till the time that you have to like actually be in your new apartment start figuring out how you want to move Way before so I would say start migrating this data as soon as you can and What we ended up doing was we would start migrating the data differently as the needs of the developers changed We would find that we needed a feature to be different on the front end or we would need a piece of data that we didn't think we needed anymore and What we did by using the migrate module was we were able to just re migrate content With the data transformations and changes as we needed them So we were able to actually migrate the data alongside the code iterations The other thing we really wanted to do was we wanted to keep some kind of view Both for the editors and for the developers into the legacy data without necessarily having to rely on keeping Our old BCM system alive and kicking in some back room somewhere forever and ever and ever So we wanted to in some way be able to hold on to the vital parts of that Old data model and also be able to understand some of the logic and thinking that had gone into constructing those data models even though we didn't want them anymore so we set out as Ira and John said to come and sort of come up with a big picture the big dream How do we really want this to be and we just went in and created the content types the way that we thought we would want them and then We built what we call a Drupal 6 data bridge We called it the ECL the enterprise content library and so we set up a Separate Drupal 6 instance that was going to sit in between the old legacy content system and the Drupal 6 instance that was going to be our actual production sites for our users and We relied heavily on the migrate module and a lot of the work I want to give a shout out to my grind here who back ported some XML functionality from D7 into D6 for us and Taught me maybe one one-hundredth of what he knows about migrate and migrations So the schema of this I don't want to spend too much time on this is we see too many charts in these kinds of conferences all the time Basically on the left side here. We have two services that present XML data one from our recipe database in the middle there We had about 60,000 recipes We set up a service from the old BCM data also an XML format And using the migrate module we mapped those into this Intermediate enterprise content library Drupal 6 instance We also had a you know years and years and years and years of high quality photographs that we wanted to migrate in as well and keep tack of and through Scanning the IPTC data of that. This is something that's in progress right now We're trying to get a lot of the metadata for those images into this Drupal 6 instance what this Middle layer was a lot us to do would be basically to hold on to a lot of the unique IDs Old legacy data about when something was created Who created it things that we didn't necessarily think we would need in the final product and actually would rather remove But it allowed us to keep it and refer back to it and it was a resource for in future iterations of our migration We might decide actually we do need that thing And then we had another whole set of migrations that we wrote from that Drupal 6 instance out to the Drupal 6 Magazines platform where we serve Martha Stewart calm whole living calm and Martha Stewart weddings calm So some of the things that we learned about this was we spent a lot of time going backwards and forwards trying to figure out Where did this come from where did this go? And we really needed a way to very easily without having to fire off sequel queries every day 20 30 times a day to figure out where did this come from? Where is it going? Why didn't it go where I thought it would go So what we ended up doing was we created little widgets in our UI that allowed editors and Developers to very quickly be able to see where something came from and where it went So here you can see a little widget in the enterprise content library system that Would tell us what node was created and on which site from a certain piece of content and On the front-end system in the production websites We've created a little widget with the UUID of the item That had been migrated in from the ECL and that's of course a clickable link that would take you back To that item in the ECL if you wanted to view it there There's also a link underneath it that you can't quite see which would actually take you all the way in this case This is a recipe it would take you all the way back to our original recipe content database So the second thing is that we learned we sort of knew this from the outset But it was certainly made very clear to me as we worked along and I think Mike can concur with this that the content types are very iterative The demands of your data what you think you're going to need Something that you thought you didn't need at all is going to suddenly become vital Something that you thought was vital to have is going to become junk So don't hold on to that original data model too much in the very beginning a Real blessing that revealed itself to us through this was by using the real data as soon as possible bugs and bad assumptions in our Consumer-facing application logic became apparent much quicker Real data is much messier and dirtier than anything that the develop module can generate for you Or any sample data that you're going to type in because you're going to type in any data with the assumptions of the program are already in mind But real data is messy. It's you know, you got ten years of people trying to work around the content management system By customizing titles or introducing line breaks and that stuff is going to be really helpful to you The sooner you can see it. It's going to be really helpful to you in sort of resolving issues that you didn't expect coming along And the migrate modules are wonderful. They're really powerful But they're very complicated to work with you definitely want people who are very familiar with object-oriented programming and their resources intent resource Intensive we were definitely bumping up the memory limit of PHP often To make these things wrong So we've got a process now for iterating our data models through our launch But we also are going to be deploying this code very frequently The target would be you know once a week or once every two weeks. So we needed a framework for deploying this stuff So we wanted to do it the same way every time Because we wanted to have faith and trust and not have to feel like we were reinventing each deployment every time we did it And we also want to de-minimize the steps that we would have to take After deploying the new code to the website, you know, like good don't go to this admin form and click on this button and save this We wanted to get rid of all that stuff We also wanted to be able to deploy content and configuration without changing the content since our editors were going to be working Directly on the production site So we looked at a bunch of tools that are out there and without wanting to disparage any of them None of them really we're going to meet our needs and that's a long topic that I'm happy to talk about at some other point And Drupal is making a lot of progress in general in this direction But we ended up sort of throwing together our own deployment solution Based on some tools out there and also writing our own tools. So we wrote a tool called Bower Which is a bash shell script which basically is a wrapper That calls on thing which is an ant-like XML application deployment tool But it's written in PHP which allowed us to use the resources and the knowledge that we already had on our PHP expert staff To customize it and then we wrote all a bunch of custom drush commands And also I might mention we named Bower It's Dutch for builder in a little tribute to Dries so We decided to use features in order to encapsulate changes in the data model changes in permissions changes in views and And also to be able to export configuration and deploy configuration We tried to use c-tools and exportable as often as possible everywhere. We could We reserve the update hooks for schema changes and Then of course source code management in our enterprise we decided that get Would be the right solution and we ended up using the get flow model which John Reynolds here We'll be happy to talk to you about later So how do we create a Bower release? Basically, it's three very simple commands First you create the release package you type in Bower seed and I have platform and version platform Well, we had three Drupal instances running. We had the enterprise content library. We had the consumer-facing Drupal instance and then we also had a d7 instance which James will talk about later for our user content So we needed to know which platform we were deploying so Bower seed Let's say the name of the platform and the version that you want to seed and what that would do is it would create a release package Which would be a release branch in git and a state for modules enabled which modules were disabled Some other sort of relevant information about that sort of meta information about that release package Then you can tweak that and change those say and make those changes say those changes commit those back into git And when you say Bower save all that is wrapped back up and sent up back into the origin source code management tool And then you are ready to deploy And when you deploy on a dev environment or when you deploy in a QA environment Something different happens than when you deploy on live There are certain special hooks that are enacted when you go into the live deployment So it's it's context-aware it always knows if I'm on the live server I have to just disable the develop modules for example And that's that and we would continue to use the same The same deployment model for all of our releases throughout the cycle So one of the things we learned from this is by by sticking to using the same Release cycle and the same deployment tools. There was a lot more confidence in our releases Less nail biting less nervousness The second thing that was good about it was it allowed our developers to actually develop their code towards a deployment They knew what was expected of a finished feature. They're working on a new Part of the site and it's not just I've written my functions. No, have you also wrapped this up in features? Have you enabled the permissions and saved those? Is this something that is deployable with our deployment model? So it really kind of enforced a tighter coding standard on our developers and I think it also gave them more confidence that what they did In their code was actually going to be represented on the live site correctly Unfortunately with Drupal 6 we still can't capture every configuration detail on some of our releases We did have to you know do some manual poise deployment steps So it's not perfect, but we I would say nine times out of ten. We were able to do our deployments Without any other further interaction And unfortunately still the you know the holy grail is to do a real true rollback in Drupal We gave up on that we tried And so we're we do a database backup before we do any deployment So we knew that there were two main structural problems that we needed to solve when we were building out the structure of our site The first thing was the hierarchical nature of our content Whereas many sites might have a Article node with a image file attached to it What's more likely in our case is we have a gallery node which node references a recipe node Which node references an image node and then that image node would have many different image crops attached to it So here we can see an example of a recipe node and you see a delicious looking cookie on the left hand side So that's actually we're looking at a recipe node which is referencing an image node and there's an image file attached to that Here we're looking at the same Recipe node, but this time displayed in a gallery And so we built a number of tools for our editors to be able to you know see where all this content was being used on the top We have that first recipe node and we're seeing all the different places that that content is being referenced throughout the site And on the bottom we see the same interface for the gallery node that we just looked at So the editors have a clear picture of where all this stuff is being presented to our users and can and make you good use of that So the other problem that we needed to solve Was something that most of us learned very quickly when we started at Martha Stewart that there's no breaking news at Martha Stewart All of our content is evergreen. So, you know, most of us have built plenty of Drupal sites where we've created lots of views and Most of them or many of them were sorting by date descending But here that's not really the case that recipe that cookie recipe is going to be just as delicious today as it was six months ago And we want to make sure that our editors have the power to use that so What we did was we created promotional content items So these are actual separate content types whose sole purpose is to reference and present other content Here's an example of three different ones displayed on our home page And I should mention that each one of those links on the left hand column and many in the center are node references to our What we consider real content or recipes or articles or galleries Here's a channel page our holiday channel getting ready for Easter and Another promotional content item we call them touts. This one just themed differently as a gallery and One more the same thing just themed differently So when we were trying to figure out, how do we give the editors control to place this stuff on the page? The first thing we did was we looked at how our pages are laid out. It's pretty straightforward three column system common to many sites so we made a grid and We created blocks we place blocks using context and those blocks are powered through views For each one of those sections in the grid Then we needed to have the ability for editors to actually put content in those items in those Blocks in the grid So we looked at three different ways to do this. The first was node references. We use them everywhere else on the site So why not here? It seemed like a straightforward idea But the problem that we ran into is that for every time an editor wanted to place a piece of content there They'd have to edit two nodes They'd have to create the content item save it and then they would have to go and edit this channel page and Change the node reference to point to that content piece of content that they just created So we threw this idea out pretty quickly because we didn't want to have that two-step workflow The second thing that we looked at was panels every time that we described this problem to someone They said this sounds like panels. So we prototyped it and it worked very well for what we wanted to do But the one shortcoming that we found was that some of our editors are very non-technical So we weren't a hundred percent comfortable with putting them in the panels UI So the solution that we came up with was what we call weak references So if a node reference is a strong reference you have a Direct point piece of data in the database pointing from one node to another a weak reference is is just One piece of data that two content items share. So in this case we use taxonomy So as an example of that you can see here the edit screen for one of our touts the promo content and We move in and we can look at the two Important pieces to get one of those tout pieces of content onto the page So an editor selects a channel Which it shares that piece of content with the channel page and then they select a position So which spot do they want to show up in center column position two right column position three? So once they do that and save views takes care of the rest There's a display for that each individual position and it pulls in the correct one that they want to they want to show up there So what do we take away from all this node references are super powerful They allow you to build amazing experiences through those connections to your content So we can bubble things up and on a recipe show all of the galleries that that recipe is referenced in They're also very expensive Not such a big deal when you're loading a recipe node We reference the image and we load that no big deal But when you're loading one of our gallery nodes You could potentially be having to load hundreds of nodes just to generate that gallery experience the pieces of content that it's Referencing and the images that those are referencing all the way down the line They also create caching issues. So just like it's very simple, you know previously for a recipe For node references also very straightforward you edit a recipe clear that recipe from cash But where else is that recipe being displayed? Is it being referenced on the home page a channel page? Is it in a gallery? So we had to figure out how do we handle the caching challenges with this and Lastly blocks and contexts Require a bit of strategic planning. We you know experimented with it. We found that the best solution for us was to have many More specific contexts than larger ones. So we have many different specific contexts that all ladies blocks out on the page So next we're going to talk about how our editors and producers work with the content on a day-to-day basis As Mark mentioned all of our content is curated. It's evergreen that article From last Thanksgiving about how to carve a turkey is going to be just as relevant this coming Thanksgiving as it was last year also our brand is known for attention to detail and quality and It was incumbent on us to enable our editors and producers to create those visual Experiences as they saw fit and as they felt their special needs required so Content at Martha Stewart as I'm sure many other places But content goes through many many different rounds of edits and is touched by many people before it is actually published The pipeline that we created needed to have the ability to edit create an edit content and Have that piece of content be touched by editors photo editors Video editors copy editors Producers senior editors and anyone else you might be able to think of so we needed a way to keep it organized and Provide the correct visibility to our senior editorial staff so that they could keep track of what was going on and what was scheduled To publish or what was moving around the system? we also took on the You know the responsibility on our own to make sure that the permissions for the nodes Changed at every step along the way because we wanted to make sure that nobody was stepping on each other's work we also needed the ability to schedule publish in advance and To make sure that things could go out on an automated schedule but a little bit more You know tricky was Creating the ability to edit content that was already live Leave it as it is have it go through that same workflow again And then ultimately when it's published replace the live version So there was a lot of requirements around our editorial UI and our workflow system so To accomplish this we basically went to the community as many do and started out with the workflow suite of modules We started out with workflow we did lots of customization and The important point that I would like to make about that is that all the customization that we did around workflow And it's probably the most customization that we've done across Any contrib modules that we are that we're using is that we did it the Drupal way We did not change a single line of code from the contrib modules. We did everything with alters We made sure to have an upgrade path that we weren't boxing ourselves out from future versions of The of the workflow module so the modules that we used were workflow workflow access Module grants Revisioning and we also wrote a little custom module of our own that we call the workman So to illustrate this I'm gonna show a typical note This is happens to be a photo gallery that we have on our site But let's look behind this node at the node edit screen So I'd like you to take notice of the upper right and see that this user is logged in as editor editor sample and The tabs across the top of this node edit screen Represent the workflow that this node will probably travel through as it moves from draft to production Also, this node is in draft state at this moment So when the user is done editing this content and ready to save the node and they move on to the workflow screen You'll notice that the that the user has basically two choices here They can push it forward to another workflow state like top edit or copy edit or they can basically leave it in draft But the editor does not have the ability to publish this node and that's by design Now let's look at the same exact node But this time the editor is logged in as copy editor Well, the copy editor has a different set of permissions associated with his or her role And the copy editor on the workflow tab therefore has more choices They can push it backward in the workflow or they can push it to be published We also Wanted to create the ability to assign nodes to individuals rather than just roles So what we're seeing here is one of our custom UIs which is the ability to assign this node to a user rather than just a role and What this what this interface is does it basically allows you to sign it to any user that has permissions to the role I'm sorry. It has permissions to the node That is currently in this workflow state Send a message send an email and and save the node. So then what happens? The node goes into what we call the work bin And the work bin is the bird's eye view of all the content that's in our system That's neither in draft nor in published state So anything that's in between Appears on this screen and if you notice the tab on top says in workflow And the exposed filters allow you to search based on lots of different criteria so that you can actually locate the content that you're looking looking for The next tab is the work bin So this basically assigns this shows content that is assigned to the logged in user and So so as an editor or a producer I come to work I turn on my computer I go directly to this screen and I see what content is waiting for me to work on The third tab The third tab is the schedule to publish tab and this shows work that has gone through its workflow and is about to be made live on the site And about to be made about to be published So, you know, this is used by our editors a lot so that they understand what changes are going to appear on the live site So what we learned through all of this is that first and foremost Drupal permissions are very very complex and they're very complex most specifically when they're dealing with the differences between Authenticated versus anonymous users So if you're attempting or if you want to attempt to do any type of customization around workflow I really recommend that you look at the node access function in the node module and Understand it very very precisely because the way Drupal treats nodes and their permissions can get a little unruly The workflow module is very powerful, and I'm using the word unruly again, but you'll forgive me for that We solved special cases with hooks There's just no other way to do it in in Drupal and You will save yourself hours and hours of heartache if you follow this one simple rule The workflow module itself happens to use Drupal actions a lot We decided to work around that and instead of using Drupal actions We put everything in code simply because that enabled us to have a greater degree of granular control Over what would happen across workflow transitions? But most importantly We suggest to learn how people work Find out what they need versus what they want get continuous input from your users on what you are building mitigate risk by not Making too many assumptions Because after all they know their jobs better than you know their jobs So they are the experts in what they need and As we all know happy users that feel like they've been heard at the end of the day make us all feel happy Thanks, Ira So we could see there's a whole lot of great stuff about what's going on in the back end, but what about the front end? As our head mentioned earlier, there really wasn't much of an overall design strategy at the time when we went and set forth on this this replatforming project so the collective decision was made to Maintain a look and feel because after all this was a replatforming effort not a not necessarily a redesign effort So to set forth on this the easiest way that that we could go about it was or the way that I could think of it Was just to recycle the old code the markup and the CSS has already been created for us. It didn't seem Didn't seem like it was worth our while to actually recreate all of the markup and all of the CSS So basically we just use the same CSS the the files still live somewhere So we could access those files and just point them used our use to point them to those same files and then Using the the the templates that were created in vignette We use those as a as the starting point to to pretty much try to duplicate that output within Drupal It wasn't a super easy effort, but along the way Our design team found some opportunities to be able to kind of freshen up some of the designs that we've had from the older legacy system In the midst of that this is actually this is an example of an of a one of our older recipe pages This is our recipe content type and what when I say An example of a fresh and up design. This is the Same recipe page just with that the fresh and up version So you can see that a lot of the styles are exactly the same when it comes to the the main nav and all the global elements Some of the colors are the same the backgrounds the same So this kind of this kind of became a little difficult We had to figure out a way that we could shoehorn this this fresh and up design into The the idea that we've been going along with the whole time So the best way that we could do this was to introduce something that we would call the the transitional theme now The transitional theme isn't exactly the theme that you're normally used to in the Drupal world The transitional theme is something that we had to use in order to We have we have content types that needed to exist in both our old legacy format still But we also needed to utilize the same content types in this new transitional theme version On the left right here is a is one of the legacy designs with the blue header And on the right is is the exact same content type This is one of the content types that mark was discussing earlier. These are one of our tout blocks So this needed to exist in both the fresh and up designs in both in the in the legacy designs So what this transitional theme does is basically it's just a function it Uh, it checks the node type and it returns whether or not it's transitional If it is transitional, it'll add and remove a lot of front-end components or switch certain things that need to happen along the way we did explore the The idea of creating an actual theme for it, but at the time we had so many themes that were juggling It didn't seem it didn't seem it seemed like a little bit too much to manage for this little bit of functionality That we needed to incorporate Another directive that came along was that we needed to have forward compatible images We have a great image team that that works tirelessly to to crop and optimize in in in work on all of the different Enormous giant image library that we have And we needed a way to take care of all of that stuff that was already being created And also find a way to automate some of the the images that needed to be created in the future When we first began in the in the vignette content management system each image person would would would Crop and optimize all of these types of images So we began with like small medium and large type images and then we moved on to Some video images with different size video images and as these grew we actually moved into iPhone and some shop images and then HD images came along and then after a while ended up being over 14 image crops for each image and these are all hand done by our image team. So that didn't that didn't really seem like that it was that That was that easy it wasn't that it wasn't scalable It wasn't something that we can continue doing so we needed to find a way to make that a little bit easier for our image team So we introduced the the image content type and a little bit of custom image caching Right now. We're down to only three crops And and this makes our image team a lot happier But basically what we've used the the image content type So now we're able to reference all of those legacy images that have been imported and we can also use the we can also reference some of the newer images that have also been created The custom image caching basically what it does is it determines the image type So if it determines that that image type is a legacy image type We can go ahead and serve that image because it's already been cropped and optimized for viewing In if it's if it's if it hasn't been then we can use some custom image caching to take one of these crops and Image cache it down to the the size and specs that we need to display it as Some of the takeaways from this is that pre-processing is pretty expensive To mimic a lot of the markup that came out from the vignette content management system We had to take a lot of that data and kind of crunch it down and mold it and do it What we needed to do with it to copy a lot of that idea in class structure into to Segway into to a lot of that a lot of that read a lot of the stuff that we have in template that php that we've done in Pre-processors is pretty complex and it's it probably should have belonged in the in the module layer so I would I would advocate for avoiding a lot of complex theme logic and The third one I that's I believe is probably kind of unavoidable But but managing a lot of parent on a lot of multiple parent and sub themes with multiple inherited styles and still supporting our legacy system can be tricky and I don't really know any other way around of how we did it, but but I'm pretty happy with the way it turned out We're not done yet because We've done a lot of great stuff like we've migrated all this data out of this old system And we created a new Drupal stacks with new content types pushed all the data in there And we created amazing workflows for our editors and we made it all beautiful But only we care because no one has seen it yet We haven't presented any of this any of the Drupal pages to the public So now it was time to actually light it up and let people see our our great work on Drupal So I'm gonna harken back to my earlier what I was saying earlier about our roadmap our early plan we were gonna maintain those two CMS is in parallel and Move stuff from the old to the new well as we're doing that in any given situation How do we know if a given URL is supposed to be served from Drupal or from vignette? Even that guy doesn't know So the first the first piece the puzzle was obviously the load balancer was critically important and for us This was this was actually really simple I'll tell you why we got a little lucky because the What you're looking at here is an actual an actual URL of a Martha Stewart recipe on the old system the vignette URL structure We had was content type slash title Our SEO business analysts wanted us wants to change that for reasons I won't enumerate here to node ID slash title when we move stuff over to Drupal so We got this is why we got lucky and you might not get so lucky But in the vignette system no URL started with a number So we just created a load balancer rule that said if the URL starts with a number sent to Drupal Otherwise like in the case of this one. It's a vignette URL by default so sent to vignette So the load balancer part was pretty easy. There were some fine-tuning But that rule got us like a huge part of the way so the next piece of the puzzle was redirects obviously we're gonna have to create a lot of redirects and As we rolled say a content type like recipes It's time to serve that from Drupal instead of vignette. We were faced with okay We've got to create a redirect for every vignette recipe URL We have to create a redirect to the corresponding Drupal URL. So we use the path redirect module. Anybody know this module? popular module Anybody love this module as much as we do So the path redirect module is an incredibly simple and powerful module It stores the source and the target of your redirect and the and the the status code usually 301 or 302 and it'll and then Drupal actually perform the redirect and you actually Also, you know, you can go in the note edit form and create a redirect right there so we started with that but that wasn't good enough and here we have an example of a Drupal your Drupal URL being or vignette URL being redirected to Drupal URL But we had a problem we didn't want all of those redirects to be processed by Drupal because We have a lot of traffic and we didn't want to have the performance problem that we incurred from all those redirect requests coming to our Drupal infrastructure, so we partnered up with Akamai who is our content delivery network partner and We said what can you do to help us here? And they crafted a solution for us where we export all those redirect rules from path redirect Convert them and our sync them up to Akamai and Akamai actually ends up performing the redirect and Path redirect module then just becomes the way we actually manage The redirect rules themselves. So bring the load balancer back in the picture As we created those recipe redirect rules We push them up to Akamai and now our vignette URL gets redirected by Akamai turn into a Drupal URL and a load balancer recognizes Drupal and We launched all the recipes hooray and as we did each content type We would just generate another batch of redirect rules and that was like the big ceremonial Thing that we did to launch a content type. So here comes the special sauce custom link rendering so All right The first page so when this is this is funny So you know when you get like carpet cleaner and it says test on an inconspicuous area and you think then that's really good advice Right, what page do you think we launched first on Drupal? The home page we launched the home page first on Drupal. You know why? Because go hard or go home. That's why so right so So but we had a problem right because we launched the home page on Drupal But everything else is on vignette so all the links to all the atomic content on that homepage have to point to vignette What are we going to do and what's more as we start moving content types over to Drupal those links that were vignette Have to matchly turn into Drupal links. So here's what we did We started with a serve from vignette list literally a list of the content types We were serving from vignette. This was an array because it's Drupal got have an array So we made an array listing all the content types It was a comp variable that we stored in each site settings dot PHP file And we just called it with variable get from module and theme code anytime We needed to check which content types are served from vignette And as we started to move stuff off of Drupal we go and take that one right out And then we as we moved them over that array eventually became an empty array, which is extremely gratifying The second ingredient was a cck field that we put it was a shared field on all the content types That contained the legacy vignette url that we actually ported over from migration So Christopher made part of the migration grab the vignette url and port it over into this cck field on each node You can see why this is going to be useful and finally we've had our own custom functions for rendering links our own custom Drop-in replacements for url and l temporary replacements, but replacements nonetheless. So here's another example of a tout You've seen we've been talking about touts all day long So mark mentioned these touts the links in these towers created by node references, right? So here's the node that's being referenced. Okay, it's this I'm not even sure what content type it is but it's there's the node ID and there's the early alias we're using path modules everybody does we have early alias is and The vignette url, which we've captured over migration is also carried through with the node in that cck field So our old friend the url function We all know this function right if you put in if you've got early alias And you put in a string like node slash and id you're gonna get the url alias back to generate your link if you're using l or url We simply just made our own custom url function that added another check in there It's before you checked it check that serve from vignette list. Is this node that's being referenced is its type on that list Oh, then grab that path It's in that cck field and run that through url instead of node slash and id if it wasn't on the list Just run node slash and id through and get the droopal url And by doing this when we took out those those content types out of that that serve from vignette array The links would just change to droopal was automatic So that was real slick takeaways from all of this dual platform stuff It required a lot of care I'm making it sound like you know it was we thought of this in an afternoon But we spent a lot of time mulling over all of the exceptions and weird edge cases and you really have to think about that stuff You don't want to be controlled by the edge cases, but you need to factor them in and know what you're up against It took takes a lot of time to do it right I mentioned showing continuous project huge win I recommend if you're doing a project like this make sure you're not doing it in a cave and that you're Benefiting production either the way that we've described or some other way. That's a huge win all the way around Having those two things those two platforms was not easy. It was expensive We had to pay for the infrastructure for both droopal and vignette and and there was complexity added as a result of that We had to write some disposable code like the custom url function I just mentioned and some other things we knew this was inevitable When possibly you want to try to abstract that disposable code into modules or functions that you can find easily later and clean up But have no illusions that you can write some you can't you can do it without writing some disposable code And finally the gradual transition from one platform to the other had pluses In the the editors enjoyed it because they got to see the and work with the Drupal interface early and Affect its evolution because you can change it with form alters and stuff pretty easily editors love that They couldn't really do that with vignette So they got to have a say and how that UI worked But the other side was the editors were working into CMS's so they'd have to do some stuff in Drupal And then if they're working on a content type, we haven't moved yet. They have to work in vignette Painpoint, but you know getting them involved with Drupal early was really effective at getting their buy-in and getting their enthusiasm energizing the project Hold your ears. I'm gonna move the mic Okay, that wasn't that painful So as you've seen this process has been iterative Pretty much touching every layer of the stack and once we started working on the authenticated experience There was really no exception there either The first thing we needed to create was a seamless experience So if someone were to navigate to one of those vignette pages For instance if a recipe was still on vignette, they go there they log in and then if they were to navigate to One of the other content types that had been served from Drupal. They should at least appear logged in right? This was priority number one. We had to get this figured out before we could launch even that first content type The way we did this at the beginning is on our Drupal 6 side We had some JavaScript in the page that would perform an Ajax request back to vignette to retrieve the information about the user And then in that on that Drupal page, it would just pop that in there You know so if it says hi James up at the top You know it knows my name because it's made that Ajax request back to vignette So it at least appears that I'm logged in Because we did this this is really awesome in hindsight because it forced us to start thinking about The authenticated experience as a completely separate application And so once we started working on implementing this in Drupal We made a really exciting decision to actually use a completely separate Drupal instance for our authenticated experience And if you guys familiar with discuss So discuss is like a commenting system It's kind of works the same way where you've got this separate application for all your comments and everything like that This was a the instance basically so that we had this instance it contained all the users all the comments All the user profile data complete separation of user generated content or UGC and This was a huge win for us not just in the in the aspect of being able to shard the data So we have this completely separate database that has all these users and all the content and comments and all that stuff But also just being able to kind of custom tailor these two environments from a security perspective and a performance perspective Another huge win here was that because we had this single instance for all users We could really easily implement single sign-on. So say you go to Martha Stewart weddings.com you sign in you then go to Martha Stewart Dot-com you're signed in you've got all the same user profile data It's across all the different environments if Martha Stewart adds another brand to this BAM. It's all the data is already there One thing we knew we needed to do is to create a really feature rich comment experience And while we didn't need to do this right away eventually we needed to Have a way for users to be able to upload images to comments for users to be able to rate comments for users to say be able to post their own variation of a recipe and Because of this we made another really exciting decision to use Drupal 7 So Drupal 7 comments are fieldable so we can create this really customized comment experience Really easily in Drupal 7 if we were to try and do the same thing in Drupal 6 not only would we have been adding? Probably using nodes to do all this for comments and therefore adding even more nodes into our node table But it you know it would just be extremely extremely complex So we stood up this separate instance, but how did we get that into our Drupal 6 site? Right, we've still got to solve that problem The way we did this was with iframes and ESIs or edge side includes In our Drupal 6 site We have a module there that just simply prints out ESI tags for all the user bits user authenticated experience bits in the page and ESI tag looks something like this and What happens at the edge as a CDN is referred to is the request for that URL goes back to our Drupal 7 instance And what the Drupal 7 instance is return it returns is just simple markup like this containing links to Drupal 7 pages a Perfect example of this is the sign-in and register now links that you see here So if I were to click on one of those links You'd actually get a modal window popped up and that would be the iframe That I the reason we used iframes here is that way we could completely leverage Drupal 7's Ajax API Without having to do any of like cross JavaScript between Drupal 6 and Drupal 7 that the Ajax API between Drupal 6 and Drupal 7 is completely Different and the Drupal 7 Ajax API is awesome. So Just so you know if you're clicking around on Martha Stewart comm right now and trying to replicate this We haven't actually like I said, this is iterative. We haven't actually launched this yet But it's it's a work in progress and hopefully in a couple weeks. We'll have this completely done. So What did we learn from all this? We created a very complex environment for ourselves So we've got developers. We've got QA testers We've got designers all trying to work with and understand this really really complicated environment, right? So we needed to calculate in the time for documentation for communication and most importantly for a test-driven development workflow We you know, it's really easy to add like add Just have regressive bugs with this complicated environment. So Another thing that we learned is how important it was for the developers and the designers to communicate Early and often, you know, it's kind of developers kind of want to go off in their own cave and build things in Drupal the way They want to build them and the designers kind of want to go in their room and just dream up these beautiful user experiences You know just beautiful workflows and everything like that And if there's no communication from the developers saying hey, this is how Drupal wants to do these things. This is what we can get out of the box The designers are really just building these things or you know dreaming these things up And there's there's there's a disconnect there, right? And it's just gonna completely lengthen the the how long the project takes So the developers can communicate the designers you make compromises and then we start working towards that goal of what the designers want to do So like Ira said We're we've only got a minute here, so we don't really have time for questions, but we're gonna be here Oh, we do have time for questions, sorry So, yeah, if you guys have questions, please come up come up to the microphone Hi, congratulations. First of all, it sounds like a really successful project Thank you I'm curious. You mentioned you're doing about 10 million page views a day on the and that's For people logged in and not logged in is that true so the the question is is the the 10 million page visits a day Is that overall or is that only for anonymous? Well, that's a combination of both Yes, that's that's that's a combination of authenticated as well as non authenticated users The problem with our old CMS was that it was kind of difficult to create these customized Authenticated dynamically driven experiences and that was one of the main drivers that led us to Drupal So we rely on Akamai a lot for our cash content and our non Authenticated users our anonymous users, but we don't expect that we're gonna experience a tremendous amount of scaling when we Begin introducing more dynamically driven experiences because of this separate Drupal 7 instance that we've set up Exclusively to handle those experiences and the way Akamai with the ESI is allows us to reach back into that Infrastructure will will help us scale in a in a much easier and streamlined way One one quick follow-up question. What percentage of your traffic do you expect to come from? authenticated users when you launch that Right now. We're we're looking at approximately seven to eight percent Obviously, we'd like to increase that You know, we're thinking 25 to 35 percent might be authenticated. Thank you Thanks for your presentation. I had a small question Can you explain the SEO reason for the node ID? title and then also the bigger question was you talked about the the enterprise content library and that sounded kind of static the Way you had it set up We're doing something kind of similar, but it's it's it's sort of to be for our different channels where All of our different people Weren't technical people can update our digital assets and that's more of an active thing to let to like push out to our various Sites to feeds. Are you guys looking at doing anything like that or okay? So there are two questions here The first is our URL structure and why we decided to use an ID slash title rather than content type slash title There's not a lot of you know Rocket science in that answer Everyone knows that SEO has a little, you know black magic in it and you know, you can't really can't figure it out to be perfect We were advised by our SEO experts that putting the content type in the URL really did nothing to boost SEO And that it was more important to have the title rather than two conflicting strings that that the SEO engines wouldn't Necessarily or the SEO Processes in the search engines weren't necessarily understand the second question as they understand it relates to our continuous migration process So I'm going to allow Christopher to to answer that That has to do with our enterprise kind of like content library and how data flowed from our legacy system into our Drupal system Yeah, you mentioned that you thought that the enterprise content library was somewhat static Yeah, we didn't consider the ECL as a you know a staging environment Drupal natively is built so that the editors and content managers are actually working on the live stack And we actually thought that was a win for us. I know a lot of people feel differently But our old system was a preview live system where we had preview environment You see what it's going to look like in the preview You push it out to the live and then you see what it's going to look like in life And one of the things that we felt was a big win for our editors and their contentment with their job Was to be able to like make a change and it's live Granted there's all sorts of cash bushing busting issues that that we deal with there, too So we really did want to keep the enterprise content library somewhat static We pretty much set up a principle for that that you know, we haven't followed, you know, perfectly But we basically did we didn't want any manual process on that Drupal instance We have scripts that run on it that Will take unique IDs between two content types and create a node reference between them For example on an automated basis and we have of course the continual import of new content and so forth but we really wanted to keep it kind of as a repository and You know at some point we may very well just take it out of our system if we don't need it anymore We didn't want to make it part of our sort of future workflow Although we've found it useful for all sorts of other things in the meantime I might also attack on an answer to to what I was talking about earlier As mark was mentioning our content appears in several different places It's not like there's a recipe on one page and one page only that recipe can exist in a content gallery and You know and on a tout so our URL the RSEO strategy It also employs not just the node ID in the title But also where on the site and in what context is this recipe appearing so that's some of what we were doing there Thanks for a great presentation With regards to the more elaborating on SEO and related to the Russian doll hierarchy And I mean you talked about how you switched and the SEO implications there But after that and in this I presume new hierarchy that you have on the news site You can elaborate a little bit further for example I noticed it seems like you've grouped all the recipe content in one sub directory Any any other you know things like that that you can mention I'm not quite sure I understand the question Do you understand it mark? Yeah. Okay. All right. I'll let you I'll let you read. I hope so So I think correct me if I'm wrong with the question is related to SEO and the structure of our content And how we handle that when we display this right, so Well first off, I should say that none of us up here are SEO experts. Luckily we have people that we work with that are so but maybe hopefully this is a good enough answer for now, but We kind of are a view from the beginning was that we wanted to have the flicker model so, you know on Flickr you have a photo and it has its own individual page But you could also have that same photo in a gallery in a set and so they're dealing with some of these same ideas thematically that we are so We wanted to have in our URLs a similar fashion So when you go to a recipe you see an ID slash title when you go to that same recipe But in a gallery you see recipes slash or an ID slash title of the recipe and then we use the pearl module persistent URL to then add on It would be at photo at content gallery and the NID entitled the photo of the content gallery So we've got all those elements in the URL. So hopefully that helps us with SEO but we didn't want to have we wanted to incorporate all those elements and bring it together So it's not this totally different experience. It is the recipe. It's just being displayed in a different way And so that's always pointing to if you're viewing a recipe in a gallery the canonical tag on that page is actually pointing to the standalone recipe page I have a couple questions in regards to the workflow You were talking about I've had a couple projects where I've used workflow quite a bit and I'm curious to see if I'm guessing you did in course of this project if you evaluate a workflow versus workbench and the reasons for choosing workflow and Second of all you had made one of your points was to avoid using actions now usually I found it very easy to just write custom actions to plug into workflow transition states And so I'm curious to see why and what you used instead of actions Okay, well the question was why we chose workflow instead of workbench When we began this project two years ago workbench was not out. So we were we had no choice and I believe that workbench currently is only on Drupal 7 correct Yeah, so it's not a Drupal 6 module. We you know, we were we were forced into using workflow. You know, we're not sorry It's a great great set of modules As far as the the choice to not use actions That was really more related to the fact that because of our deployment model and because we wanted everything To be automated and to reduce the necessity to go into user interfaces and change things around Features was just being introduced when we started This development work we made a decision that we were just going to do everything in code Put as much in code as possible. We do actually call You know the the do action function at times And we we do engage with the action module But we just decided not to use the action UI interface and to set up actions to deal with our workflow transitions We just encapsulate everything in code and that just allowed us to do a little bit more with the workflow hooks and the transitions between states


Okay, all right Welcome everybody. Thanks for interest. Wow pretty crowded This is Kim Jill. I'm under clapper. We are 50% of the engineering community team at the wikimedia foundation and The talk is called wikimedia adopts fabricator deprecate seven infrastructure tools So this is basically what we've been working on for the last maybe nine or 12 months I I looked at the talk description again. I realized there's a few aspects that you might be interested in There's there's a part that's the social process decision-making There's the migration process itself And the functionality of this tool that we're using now fabricator itself So you might be here for different reasons I tried to cover everything a bit and you might all be disappointed in the end because it wasn't in depth enough So first of all wikimedia It's a project and community behind wikipedia Commons and a few other ones. There's volunteers. There's stuff of the wikimedia foundation. We tour stuff Development engineering teams They all have the different needs workflows tools or heads depends Of course also projects like core extensions So what I want to say is there's quite some diversity in teams and attitudes and workflows and tools But I think we can say that free and open source software and free knowledge are kind of like shared core values So to to explain where we were about one year ago The main tool was bugzilla instance You see bug report here with lots of interesting ui fields. It's just the top who has who has used bugzilla before Hands up I see Next one is rt Hands up Okay, rt was mostly used by by the operations team in wikimedia foundation This is mingle not very popular. I see well at least a few Yeah, it's yeah, okay wikimedia stuff So you so you see the card view here It's this is not a task or a ticket like the two previous ones This is a function that for example bugzilla doesn't have and you see the columns and you can move the cards and Um, we also had a I think this is trello. Yeah a trello instance trello anyone. Well, that's more popular. Okay, cool And for code review we still have a garret instance Garret anybody well also quite a lot. Okay And uh, we have and had all these tools and uh in between these tools we had some interactions for example a hook to create Comments from garrets in bugzilla tickets And as some of our teams, uh, like to use mingle and trello for the project management We also had self written custom scripts like called mingle and bugello to sync a bit between bugzilla and mingle and bugzilla and trello and all these funny things You might see the problems with the setup Collaboration between staff and volunteers and also in between teams Communication openness transparency accountability Um, for some stuff you needed a login even like like mingle or trello was harder to reach rt. Some people said it was a black hole Duplication divide we had to maintain these scripts Things broke sometimes for example once we upgraded bugzilla then the garret notifications didn't work anymore for version 4.4.5 or something um programming languages media wiki the the main thing is php, but um That was pearl and java and uh, we weren't it wasn't that easy for for people in the community to even do what they want Uh, then onboarding new contributors Learning curve is pretty steep the more tools you have and people nowadays are a bit used to github um, yeah and bugzilla simply Isn't for project management. So everything all together slowed us down So the idea was to find something better or uh, at least discuss if we can find something better with clearly developers as the main audience and not for example Editors or or readers of wikipedia or something um, the the problem as usual with These things is if you want to come up with a new standard, it's uh in your community You might end up with just one more tool. So that was something we wanted to avoid um, so what we did uh in i think around May march or april last year was we invited teams and developers to describe their needs On the page that was december january. That was december january. Okay All right Cool. Uh, thank you. Okay. Um, so so we asked a few simple questions. Uh, we had 24 replies Uh, by the way, um All the stuff you see here is of course on media wiki.org. So, uh, you can look it up 24 replies but from 24 types of profile that you think that you know, we have more than 500 people But really like we we we went into depth into different roles in the community in our organization, etc um, so so we collected these that feedback, uh, then Uh, we consolidated it into must versus would like to do this or that in in the tool I'm using for my project Split that into several categories. So we we got a basic idea basically what what are the needs? what what do the people want and um Then we asked people to propose tools that could be options because Developers and in general people have opinions. So Normally, they also have ideas, uh, which projects they like And I have to speak like this. Yeah, I'm sorry. We only have one microphone. We are really good colleagues So no and we did that because uh as if you if you enter into A discussion like this in your community probably the first thing you will get is oh, I like this tool I like this tool and you're discussing basically your preferred tools and you're not discussing what you really need to discuss Which is what you want to do. What's the current problem, etc, etc. So that was the reason for the survey um, and after that Yeah, we had several tool options at the beginning. Um, we Encouraged people to discuss to investigate the functionality too And then we were after decreasing the proposals the items in that list that we considered We encouraged people to set up test sensors. We have wikimedia labs Where we can run or install any kind of stuff um And and the feedback quickly Turned into this one question like shall we move to fabricator? We we had looked at seven or eight other tools sometimes also combinations Uh, shall we only use that part or shall we use this part together with that from from some applications? But Really we went we went two or three weeks. I guess it was like fabricator seems to be Best fit for the needs. Um And then together broader community support and opinion Because yeah, we just had a few dozens of people discussing this We had a three-week request for comments, uh, which is also a wikimedia specific thing where we Asked the community put up banners on pages Communicating widely basically And that rfc after three weeks, uh, it seemed that there was a general support for moving From our infrastructure tools, um to fabricator So that was as I said, there was also another proposal Proposal just use a part of fabricator and keep some other stuff But generally everything so and fabricator is according to wikipedia if you can trust that suite of web based software development collaboration tools um originally internally at facebook it's in php and it's under apache two And uh, I'll have more screenshots in there. Uh, and if if you're Can't wait fabricator dot wikimedia dot org, but it's it's a software forge So it has several applications well integrated into each other And you see there's there's a few core applications for example for for tasks and bug reports where something called manifest You have you have code review, uh notifications At some point Continuous integration Yeah stuff But the thing is it's well integrated. So it things linked to each other Um, so as we had this rc, uh, let's move to fabricator. Uh, how to approach that. Um, we Created a team basically it was cross team, but you needed you needed, uh, some expertise from some areas So, um from from the release engineering, uh core team We have somebody with php skills and and he also had maintained a fabricator instance before Lower level stuff operations. Um The migration script itself somebody had to write it. There was nothing So that was also done by uh chase who's who's in operations. Uh communication Wide uh in lots of places with with lots of stakeholders With the community for example with with the upstream project because hey, we're interested and we'll probably come up with some needs Yeah, the debug management aspect itself and, um, trying to find all these other small bits that might fall through the cracks Because you probably always forget something Because nobody has done something like that before with with that size and there was no migration script anywhere And of course, um product management release management that that you have contact partners there, um That you can discuss the needs, um And what what I consider is especially important the relationship with upstream. Um, you must understand their model their attitudes Uh discuss what's feasible and for fabricator upstream. I I really have to say, um, they're really clear on priorities They're They answer unbelievably quickly, but they the answer might not what you want to be what you want to hear They're pretty clear with one fix this is not in our plans Our priorities priorities currently are different if you want to challenge them. You need really good arguments But if you want to hack this up yourself custom here's the explanation and Really well technically written the eight or nine things you need to consider So upstream is very clear on priorities And very helpful and very responsive Something amazing about upstream is that you see fabricator and all they have and this is basically three maintainers and It's amazing. I just have to say it's uh You have to you have to see to believe and three maintainers that they develop all that stuff But they also give you a great support. So even when they tell you know They tell you know in a way and with an explanation. I say, well, okay fair enough So it was an amazing it's an amazing experience to work with them Um, so the basic steps we performed was um, we had this test instance instance still, um, but uh Our first aim was set up the production instance Then also Yeah, after this number one, uh use that production instance itself for planning. It's like dog fooding Um, but that also means that you already create tickets in there and these tickets have a number So, uh, then, uh, we set up another test instance with Buxilla data migrated so people could see like what does this script do how would it look like What is all the data you dropped? You can have wonderful bike chat discussions about this like which priority values do we want? So you also need to manage that part somehow And then the actual migration of the Buxilla data We did that in late november over four or five days And we also migrated rt in december, but that was not too complicated But just took eight hours or something Um and way less box only one thing we didn't think about before with rt with Buxilla. We had a few more things that we didn't expect it Um, we identified the task for each of these steps You have this list, uh, we have a list on wiki for for the migration itself step by step who's doing what But uh, we also had this on fabricator, of course because we've been using that tool for planning this actually and uh communicating we asked Weeks before we migrated Buxilla our users, please please register already in fabricator Because we somehow needed to link the accounts and the data so We can still do this with people who who haven't Registered yet, but the more we could already handle in these four or five days of of users the better Um, yeah, this is pretty close to how how fabricator looks I mean this is a test instance So some colors are different and and some some decisions how how to tag Things we've changed in the meantime and the ui might have changed a very little bit, but um This was the test instance. We we set up. We asked the community for two or three weeks To play with it. Uh, we got 45 buck reports 25 uh, we fixed so that was really helpful. I recommend that um Yeah, and then this uh weekend came where we Migrated this was the wiki page. Uh, normal people got uh redirected to while a few people still had access Um So these are just the first steps to the complete list is on the wiki, but switching back to that to read only Was an interesting hack Yes, please interrupt me so here's the Before you was the bug master now, this is the fab master. Uh, can I ask you for some numbers because I don't I don't The volume of the whole thing is what counts. I mean, so just uh, how many monthly users we have more or less Yeah, you've got a good point. Um With with bugzilla we uh had about We had 20 000 user accounts. Um We had 75 000 tickets. Uh, we had on average 500 users per month logging in I mean it's it's been running for 10 years. Um And How many bugs created on a daily or weekly basis? More or less On a weekly basis. I think in bugzilla it was like 250 would be my guess roughly This has increased uh, since since we're using fabricator. So, um All these two stress all these two stress that we are talking about a big machine Used by a huge amount of people very different people and basically we are telling them So we're going to change your car. Uh, by the way, we're going to be offline. Maybe four days five. I don't know Uh, it might even might even be that we have to go back because nobody has done this before So this this was the real context that we had so don't don't only look at the technical work Look at the social situation. Um, where you can lose your head easily just for I don't know if you have ever attempted to change the homepage of your project And the discussion is my ring only the homepage So now saying, you know, I think this is an important point that you need to take into account We get more tickets per week now simply because we've invited way more teams. Uh, also non-engineering teams, uh, into fabricator Um, it's not about bugs anymore. It's not about bugs. It's tasks now. So it's it's a wider field and, um, so so we we've been running fabricator with with uh, bugzilla and our team Well, we grade it now for, uh After that for full two full months and so far the numbers are too early, but 620 670 active users per month This january was 670. So so far it's it's increasing compared to uh, bugzilla That felt like slowly going down because less and less teams accepted it because you couldn't really do project management Um, yeah, so so all these funny steps, uh work around known bugs Um fetching the tickets uh from bugzilla uh took like five hours creating tasks Commons attachments and fabricator took 25 hours Uh, we had a separate database like not directly uh, also, um creating tasks, uh, for example one wonderful. I think slightly bike share tickets was uh As as we already had tickets in their production instance, we couldn't map like Bugzilla ticket number one will be fabricator ticket number one Manifest the the bug task tool in fabricator t1 But so we we try to at least have have it linear, but uh, that wasn't too great for performance, but uh Chase from operations who wrote that script uh somehow managed so, um May I ask a question? So how many of you are in a project where bug number one is something emotional and significant? No raise your hands if you are Well, our bug number one was Fix all the docs so fix the I don't know broken documentation and it's talking back with And we had really a lot of discussion because we are going to lose our bug number one But apparently we just We ended up with the rule a bugzilla number plus plus 2000 Is the fabricator number and we also set up redirects, uh for bugzilla IDs. Um, and um Of course that required to change the bugzilla url from bugzilla wikimedia.org to old bugzilla wikimedia.org And then uh, yeah claiming accounts as we asked people to register before we migrate bugzilla We had a script for that For the most active bugzilla user this took like about 15 minutes um Here you see before and after so this is our bugzilla import script Um, and on that side on on the uh, right. Yeah, right You see, um The the actual accounts after they've they've been connected to the comments and attachments and everything Yeah, okay, this is slightly edited here. So it's on the same level Um, so yeah, we we uh on that weekend we had to test a lot of stuff like access restrictions We we have a common a custom extension to For for access restrictions to certain tickets that might be security vulnerabilities or so That's not an upstream. Um, we also have a custom extension For dealing with sprints now You have lots of other small bits to do around in your environment on wiki templates I think I I edited dozens of of pages of documentation that link to bugzilla that now need to link to fabricator And you run into problems in bugs. You never ever expected to happen. Um, which can be, um Interesting. So we didn't think about moving bugzilla like, uh, yeah, we need a new certificate for that subdomain. Well Um, you had something like private comments in bugzilla and our script when, uh um Putting the the account data together or claiming it again Uh, it's overwrote that flag made invisible again, which Uh, wasn't too good. Um, well, we fixed I really got to say we uh, we are mostly chase and mukunda As as developers with the help of daniel another developer in the foundation and also with the help of upstream upstream because, um Even from upstream he was available on a on a sunday morning for him to, uh, fix our issue with the search indexer performance uh, it was an upstream issue, um And yeah, so, uh, I I blocked about details and you had something. Yes So this was for people. We had never met face-to-face Like, you know always online And this was one of those weekends where you end up being really good friends, you know one of those So, yeah, and we finally met last week in the same place of four of us. That was nice as well true Yeah, it's people you deal with basically every day you chat at least for an hour and yeah um, so All in all it went surprisingly well I was afraid of two things after the bugzilla migration because that was really the biggest Tool we had and the most popular one when it comes to numbers of users and opinions um, so I was afraid of two things, um One is after migrating bugzilla that everybody will come up with I have this great idea I can I do this in fabricator and the second one was Maybe some vocal people somewhere whatever staff community That might there might be some backlash. Oh, I didn't like I don't like this Why did you do this or that and and the second one? Didn't really happen which made me happy because then you can still get some sleep at night And if you only need to deal with the first one um First of all compared Okay compared to bugzilla we have a unified login So my naive idea so far is that's maybe also why we have so far a higher number of of users compared to bugzilla and other stuff um That that was together with uh exposing email addresses the two main complaints are ever overheard when it comes to bugzilla Uh, when you compare front pages, this is the latest upstream development version of bugzilla And this is the front page of fabricator where you have the tools here on the side Um, and some feeds over there with with latest stuff. I I have notifications up here telling me about about stuff. I'm watching Um, and if I click here on manifest, uh, which is where I spend most of the time, um You you you get to uh the default page will be uh your queries that you have saved And I have the the tickets that are open and assigned to myself on the top So that's what displayed here and uh, I basically go there and see in the morning. Okay. What to work on And you can have your saved queries for other stuff as usual um Yeah, this looks complicated But but the ticket status workflow has also has gotten easier. I think because the statuses are really just like the brown orange ones And these are more more like actions nowadays. This this will go away once we mitigate code review to to fabricator We haven't done that yet but all in all there's just the status and fabricator not the status and resolution anymore and um, I think we simplified it a bit by by having these four statuses over there, which are all closed Uh, and a bit easier than the workflow we had before And maybe the most important thing or criticism, uh There's project management. You you have work boards. You have columns. You have cards Uh, they show you the assignee here. They don't oh, yeah, they they do here. Okay The color refers to the priority, but you can Backlog needs discussion ready to go needs design Doing this implemented needs review whatever you can come up with your your custom Uh for each project with your custom uh columns So for example when you have a developer summit like last week, you can have the idea to Assign the rooms by using columns, which who had that idea? It was interesting. I liked it So you can basically plan quite some things Um, thanks to wikimedia germany. We have uh an extension custom burn-on charts for sprint projects Which many teams wanted we had a scrumbugs extension before for baxilla, but that was also like not really maintained One interesting change I like but but it's challenging is there's a flat namespace. So you so you have only projects you can use different colors and different tags for things Sprints target milestones teams Like keywords with a cross project basically like accessibility or performance actual Code projects or code repositories. Um, but it's a flat namespace So in baxilla you had like tags keywords status whiteboard product component and all different fields and um I I prefer it and here you see also the column that it's in on that work board But you have to have some idea at least how to organize that first You shouldn't just start So where we are right now, um We're still like after increasing the acceptance across teams. We're pretty happy pretty fine so far But of course, there's always teams that have some specific needs still that we haven't covered yet that they yell like Oh, yeah, I want to move everything now to fabricator. So they're a bit reluctant um We have a team practices mailing list to ensure effective and consistent workflows because like oh, okay I have this work board now which columns so that creates and what works for my team uh maintenance We are pretty selective with local patches because upstream moves so fast that it's Last time we we pulled a new version from upstream. We needed the second try because uh, yes We were we still haven't totally sorted out the testing yet for that Upstream doesn't have versions. Yeah, upstream doesn't have versions. That's true. So you just pull and see what you get But normally it's good. So it was more on our side the problem. It was on our side So Just to give you an idea Within a month we upgrade a month after and there were 14 Tasks that have been reported by our users that they had fixed 14 in a month now go back to bugzilla and your experience and let me know if there's anything similar to that Yeah, I agree I'm continuing the immigration from from mingle and trello for trello. We don't have a script yet I think our colleague s will work a bit on that from mingle some teams have already migrated. There is some script um And migrating the the code repository browsing currently we use a git blit for that moving that to fabricator and uh Begrading code review and continuous integration. That will I think be the fun part So this is this is a screenshot from upstream fabricator. So, um Code review. Yeah, you see reviewers. You can require lint unit tests Test plan, uh, it's uh, where's the is here a link? No, not yet To the well to the ticket. It's here at least this is this is the bug report the task number Um, and and to commit as this is merged And the lower part of this, uh Yeah, this is how it looks like Well try it. I mean it's upstream. So you'll find it Um, if you're here wondering like oh, is this something for our project? Um Do understand uh the upstream's understanding of what they do and what they reject. They're very clear on this Want to add something? Okay um upstream They in one ticket recently they explicitly said our api is considered unstable for the next two or three years So if you do customizations yourself, you have maintenance costs. You need somebody who's willing Uh to maintain your stuff and they won't announce their api changes either So you will realize once you pull but it's better if you realize a bit before If it has a public or rest api If the rest api is also unstable, um, I would assume so I'm I'm not sure. Uh, I haven't checked that they don't have an api promise at this point And they explicitly they're promising you that they will break But it's software. We like it, right? Um, they have a cla in place and um, some people Might not like that, but I I'd say like read it and decide on your own um Noven heats of your projects and developers Because you can't just push something down there if you don't have acceptance And uh involve your community and and the stakeholders like like project management Release management in your discussions Communicate early often wide and transparent I sometimes wondered like if we spam people too much when we had Two irc office hours per week where people could ask questions and put these banners up on media wiki orc And put this banner on bachzilla and this mailing list and better also this mailing list and um, but I think it worked well You will never be surprised you do all this and still people come three months later saying you didn't say anything so It's the internet. You didn't personally ask me for permission to change this website. Yes Um, so yeah, that's it. Um, you can find a lot of uh documentation help that we wrote over these months on media wiki orc slash wiki slash fabricator the sub pages like How to get the code uh our immigration code it has bugs, uh, but I've written that down now There's the project management page, uh, where we discuss workflows how to create projects. Um, there's also, um, how to set up a test instance How to pull our changes that we have on our production system and uh Yeah, if Send me an email or if it scales we'll see Or ask questions now Thank you Yeah, I'll I'll detach this and uh you Um, so so the question was uh, how the migration script was written if it was just like In one or if it's in certain steps that you could basically restart again uh, it was So, um, we we separately pulled like tickets and comments from bachzilla And then separately attachments because we had to work around an upstream bug in the xml rpc api Um Putting it into the database then again a very different process like putting that stuff from the database into fabricator a totally different script actually Um, we we had to restart Realizing after six hours Completely restart that our attachments were still Scrambled because of some encoding issue that was the upstream bug. We thought we had dealt with it We hadn't so at that point we had to completely restart at last six hours But our database engineers somehow made some things faster. So in the end we were pretty fine and we had enough buffer So yes, there was this this, uh, the man for something more incremental that less disruptive But really I mean you're dealing with 73 000 bucks is your whole history And you say guys just take a week and enjoy this great weather Come back on monday, uh, you know, that was that was what we did So one two We still use Jenkins. Yes, that was the question We still have Garrett for code review and we have Zool and Jenkins still but we'd like to kill that in the next year The next year is we are in February already. So it's this year actually I mean, sorry 12 months Uh, if we did work to integrate media wiki and fabricator Um, not much yet, but but I'm not I'm not sure. Are you after something specific? So so what we had for example were on wiki templates that uh on on certain wikipedia's and other uh sides People discuss things sometimes these things are bugs and our tasks So we have a template to link to that task for example But what we do not have in place yet, for example is automatically Pulling the the task status and displaying that in media wiki We need a bit more setup on laps with Some jason format that was special Um, if you can pull media wiki into fabricator, um, what's the use case? No, I wonder if you have just something specific Uh, it uh fabricator has a wiki. We just disabled it So, uh, it has a lot of things integrated. Yeah Yeah, so basically uh, and it's an important question actually so we decided Uh wikis are good for documentation Uh fabricator is good for project management It should know move to fabricator. I'm not going on. That's something a community should discuss So, um What I recommend, um Involving him or without involving him As I'm also active in gnom, um I don't know yet. I want to run this a few more months. Uh in in wiki media. Um, see how how things evolve How how happy people are and You need uh, both like Yeah, manpower to to maintain these things. Uh, that's something to sort out first Let us summarize this comment. Um, so yeah that that enlightenment uses fabricator and When uh, encode review when patches are a bit more complicated or longer cover more files that More patches together that it gets complicated and also with the command line tool arc, uh, probably arc, right? Yeah arc, uh, to use that part Yeah, um, I'll cut you because we still have some questions only 10 minutes left So, uh, we decided to have everybody registered Uh, I we have heated discussions and anyway, we want to give feed I mean, we know for many reasons we decided it's registered you have the policies not to do so And it might be that you find some surprises because the main use case for fabricators also with registered users But technically you can uh, but you know No There was another question down there and one So the question is if we're a bit afraid that there's only three people maintaining this upstream Um, I think the situation is is a bit different compared compared to bugzilla because so so I I've been following upstream bugzilla for quite a few years and um it's it's mostly um There's a few other companies, but it's mostly mozilla, um working on it and uh, they have their own special needs Their custom extensions some are upstream some are not Um, but it's it's not the main aspect that that mozilla focuses on And bugzilla is sometimes some people in the bugzilla community see themselves as a separate entity While we're fabricated It's a company behind it. Um, they probably want to succeed. Um, so I'm pretty positive that They're interested in Yeah, I mean, I think there's I think that's a another good question You're going to bet your your house on that. So I personally took two factors one factor is Really go to the repository see the speed of development there. So yeah fabricate on my day one day But like I tell you it's not now It's really fast and also like there's three main maintainers, but there's also a matter of adoption They are getting a lot of adoption from different types of projects. You can see how A ring of more experience contributors are getting more and more involved and you have people from other companies or other As of now, it's more about companies I think in the open source committee is not that well known, but I have no doubt that more people will just dive in And then well, there's another factor which which is subjective is who are these three people these three people are Five minutes these three people are Former facebook employees. I guess they had their options when they facebook goes public I don't think these guys have they love what they are doing. They don't have any any I'm I'm sure they have don't have any pressure to pay the rent or something So, yeah, they they could have stayed in facebook and have a safe life. They just said let's create this company Let's bring fabricator to everybody. So they really love what they are doing. So I think I I don't know. I feel safe at least in with this option One more question Which other options we had to choose from Of there were several things. I I really recommend the Review page, but there were several things like I think rat mine was brought up a fulcrum Oh, I don't remember. Yeah, but but but we at least had like track nine different ones That were brought up and combinations between them, of course The main point was that it was The alternatives are not very exciting So they solve problems that we had with vaxila, but they come with other problems that you quite know I mean like track for instance. Yes, it's a different type of beast gives me integration some points But we also know that beast and so there was really nothing It's unfair if I say nothing exciting That's not true But simply not not enough exciting to leave the place where we were And that was really an option where I was like, okay, should we go for this? And there was a sense of excitement of the opportunities that that this type of game would bring Okay, well, thank you very much