Lawmakers have been urged to the federal government to redeem the 10-billionaire promise made to the people of Benway to rebuild the destroyed communities. The Senate made this resolution after the adoption of a motion at plenary on the need to relocate displaced people of Gwe West, Makodi and Guma local governments to their ancestral homes and avert an imminent humanitarian crisis in the state. The motion was sponsored by Senator Titus Tatenge representing Benway. Tatenge, in his debate, recalled that farmers in Gwe West, Makodi and Guma local government areas of Benway state have been sacked and displaced by bandits disguising themselves as headers for years. Of the people at the camps, potentially exposes them to humanitarian crises of outbreak of disease and other health challenges. The promise that is unfortunately kept in breach. The Senate is therefore invited to resolve that one, to direct the chief of defense staff to immediately facilitate and supervise the quick return of all the displaced persons to their ancestral homes and as well provide a sustainable security corridor to all flash points within the affected communities. Moved by my brother, Dr. Sam, and in addition to that, Mr. President, this motion is very sensitive and is expected to be given very special attention. Why stating that Catherine Ford, the huge population, was beyond the financial strength of the state government? He further expressed worry that between January and October 2022, 560 children living in IDPs' camps in the state were malnourished. The Senate therefore urged the chief of defense staff to facilitate and supervise the immediate return of all displaced persons to their ancestral homes and provide sustainable security. Prayers, prayer one, direct the chief of defense staff to immediately facilitate and supervise the quick return of all displaced persons to their ancestral homes as well provide a sustainable security corridor to all flash points within the affected communities. This week, colleagues, prayer one has been amended and seconded. Those who are in support of prayer one as amended say aye. Hello, hope you enjoyed the news, please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


You know, if you will think more in terms of, how can I add value to people? Rather than how can I get a job?  And down that road you will come to an intersection between what lights you up and humanity's need for what lights you up.  And there you will hear the voice of the universe calling you towards the reason why you are on the planet. And we are all waiting for you there.


Let's go Being agile we're very timely and We're back. All right Hi, everybody So my name is Adam Ash That's James Kutz. That's Chris Herring. We're from NBC Universal and we'll talk a little bit about what we each do but The topic that we're speaking about today is really on What we've learned over the last Year and a half two years of trying to move our organization Onto of the Drupal platform essentialized Drupal platform for everybody Before we started doing this Different parts of our company and we'll talk about that a little bit. We're each on their own either platform or instance of Drupal or custom platform Yeah, and the truth of the matter is they were on lots of different That's true and as an organization the decision was made to really make a push for going to one centralized platform that we could take advantage of as an organization and Being an extremely large organization, which we'll talk about momentarily We've accounted many many challenges that we have come up with ways to address starting with the beginning of our projects and What we very often see and I think that a lot of folks have this same experience is when we start Projects and we had started projects in the past our partners and the people are going to work with right away feel like their thing is the most unique and interesting thing in the world and One centralized platform couldn't really address their needs that Most times when we start any project with any client or any partner right away They were thinking that that is something that isn't possible a lot of folks come into our first initial meetings with a picture or they go out to get a design before Even starting the project and they hand it to us and say here just build that how hard could it be right and Then of course we all get the question is is it done? Pretty much from the initial meeting right okay now that you know we want a website is it done and so And so these are kind of the things that we have learned to address on a very high level and we'll go into a little more detail so who are we and What do we do I think most people in the room have heard of NBC Universal But I think this slide illustrates our challenge pretty pretty succinctly, which is unlike you know One of these brands building a platform or content management system for one site We have the unique challenge of building not only a platform But a process that can scale for all of these sites and we're not talking about We can talk about this in a little bit more detail later But I don't necessarily mean just like a hosting challenge or performance challenge I mean the challenge of sharing modules or or function in my functionality between sites That's something that's very difficult to do in a large organization like ours because each of these brands Think that they are unique in one way or another and they are of course And we love them and we love their uniqueness, but they're always listening But I think you know what we have learned over the years and not just the last two years But we the three of us in our entire team have many many Dozens of decades of years collectively of experience building websites is that if you strip away the chrome and the design The functionality is very much the same across a lot of sites on the internet and almost all of our sites within the company So this goes into a little bit about what each of us do and how we actually address this problem. So Starting up a prop the project is really a critical part And so what we found and kind of the theme for today is really we the way we've addressed the most critical places to make our project successful and This kind of leads into a little bit about what each of us do so in the organization. I'm responsible for the process we follow Chris is responsible for the platform we build on and James is responsible for the implementations on the platform executing this so We've addressed each of these things within our process, but each of us really own a piece of this So when we're talking about the process we're talking about the roles and responsibilities of the people that are on the teams we put together and The control mechanisms we put in place for actually following our process We're talking about building a foundational core platform and we call it the core team and the core platform And we actually call it publisher seven. It's our version of Drupal seven copyright Chris Yes, and and James is responsible for the execution of this so James really is thought feeling the brunt of our partners Needs and desires all of the time because he's really at the forefront of actually making these things happen for them That's right, but actually with a clear process a good platform. My job essentially is pretty easy Which he'll tell you after the he's lying So the first thing we did was look at actually what our core platform will be and Chris is responsible for that platform Yeah, so the team that I manage for the company is responsible for building this product, which we call publisher core And the thing that's interesting that you're gonna hear about today is we use the same process that we're talking about today Not only just for building the sites that we mentioned earlier for all of those brands But we use that same process for building this product and the products The products built on Drupal 7 obviously, but what we do is we focus on core functionality that is common across the company common things like Editorial workflow that's very common for scheduling previewing publishing and syndicating content within the company or externally common functionality integration with third parties for analytics and content distribution and Video management for example, so we want to build those common areas of functionality in the form of typically Drupal modules by the way And make them available for all brands So when publisher is installed for the first time and made available for a brand on our hosting platform All of that functionality It just needs to be configured and usually it's just a simple configuration screen where you're putting in your keys That enable that functionality for your brand But that's the job of the platform and of this team So as you can see when we start doing a project already out of the box We have some standardizations that we're starting to work with So the first thing we did when we were applying this as Chris alluded to was we had to come up with a process that works within the organization but then works across our partner platform Implementations as well. And so we started actually applying agile methodologies into the organization a few years ago And what we've done over time we started basically with kind of a scrum framework and then over time We actually applied more and more lean and con button other agile methodologies that help us actually manage projects Within the organization. It's a complex organization. One methodology doesn't necessarily Scale across everything we need to do. So what we've done is we've created an agile framework an agile process that we apply To doing this so on the bottom row here is our agile project SDLC or life cycle Basically, those are the phases and we're not going to go into those in detail today But just to get an idea that we go through multiple phases that address specifically how to successfully start a project through How do we successfully deliver and measure what it is that we've delivered? We'll talk about some of those things today. Those are the critical things to actually what we're doing today but at a high level what we're really always addressing is what you see at the top which is Critically doing a great job of defining what we're doing Delivering the value and the benefit of what we're going to do delivering Benefit and value and then measuring benefit and value and so throughout the life cycle of our Projects we're always looking to do one of those things and those are the three things We're basically focusing on to make sure that we are successful each time the goal of all of our processes really to Reduce the risk of failure. That doesn't mean we won't but it means where we've learned at least in our organization how to try to address that So Basically when we're getting into the defining phase You're gonna see a theme here where we basically start at a really high level and then we start breaking it down So one of the first things we like to do when we're starting to work with our black brands is really start to identify first of all What are they gonna leverage from our core publisher seven platform? What's gonna be custom work and in some cases what are we gonna develop as part of the project? That's gonna go back and become part of our core product because other brands are interested in it Or we think it might scale out to those brands. So these are some things that we're gonna identify pretty much You know right in the beginning of the project We're also gonna look at on both sides once we sort of define Whether something is going into the backlog of our core team or whether it's going into our implementation team at both ends We want to start to get an idea of you know, how long this thing is gonna take What is it? How big is it? How complex is it and we want to start getting high-level estimates the nice thing about Working in a common platform and having some experiences with some similar brands is that you know a lot of this is Repeatable so and we're also we'll get into this a little bit But we're also working with teams and those teams have a little bit of history But even if we're spinning up a team we can start to talk about this At a high level and get a feel for you know What is the complexity just given our own experiences and the experiences of other teams within our group? Yeah, I think the important part here just to add is that Just because we're doing an agile Development process doesn't mean that we don't estimate and try to project how long a project is going to take I think a lot of our brands think agile. I don't have to plan and it's really quite the opposite We do we do more planning than we ever thought we were going to absolutely Yeah, and that's one of the critical issues that we address all the time one of the key misconceptions of doing agile first of all but also that Good project management is good project management regardless of the methodology you're following And it's always really important to understand that Any customer any partner we have really what's important to them is when are they going to get what they're going to be getting number one but number two I think it's a misconception that agile doesn't work well with deadlines and We work in an industry where you could imagine has very very strict Unmovable deadlines when a TV show is going to premiere on the network It's they're not going to wait a day because we don't have the website up when a movie is coming out They're not going to delay the premiere because something's not right So the Olympics and we can't move the Olympics believe me So, you know, those are the kind of things that we address all the time that yes So what we've actually done and we'll talk about that throughout is managing by scope But also determining what the value of things is So I talked about our approach where we start really high level and we start breaking it down One of the first things we want to do when we're talking with the brand is get their vision So in the past we might and many of you might experience this, you know, our brands come to us And they say, you know, build me a website and that's sort of their vision I want a website at the end of the day Well, there's a lot more to that and so we'll run an exercise We'll actually dig into this is actually what you're seeing here is actually the outcome of one of our exercises Which is we actually want them as a brand, you know Sort of the stakeholders to get together and and agree upon what the vision is for the project You know and that vision will carry out throughout the project and really aligns everybody At the beginning of the project and through the end and helps us really determine the priority of what we're doing Yeah, and I've seen the teams just say well that doesn't align with the vision They actually remember it, you know, developers scrums in four or five sprints in they actually remember that, you know A new feature request doesn't align with the original vision. So that's healthy Absolutely So we also Then take, you know that after that vision statement will gather high-level feature sets and this looks like It can look like a variety of things depending on who we're working with This is just an example of And we're giving you a little bit of preview of how we actually then further break that out But sometimes it's kind of a one-liner with a little bit of a description Sometimes we have a little bit more context there, but the idea is let's get the whole feature set of the project It may not be every single thing that we're going to do, but let's let's bound this thing Let's figure out what it is and what it isn't so that we can start having the conversation about scope and you know Given that we work with deadlines will often start with the date and then get into the project And so the really the best way then to manage that is either through Resources that's you know, that's one option that we don't have. Yeah and Don't always have and the other way is through scope and it's not eliminating large chunks of scope It's eliminating scope of of pieces along the way And a great example of that I just like this example because it happens quite often is what one of the first requirements most of our partners asked for is a photo gallery and one of the first First of course we got when we're doing and by the way, this is most of these Images are coming from our actual one of our actual projects NBC.com if there's a redesign of NBC.com One of the first things they said before we really got started with them was we need a photo gallery And they thought that was the requirement They had they were perfectly happy and satisfied with we gave you a requirement just build it and they had a comp They did have an image they had a picture of it But but what they've learned really soon thereafter when we said that's not really a requirement What do you mean? You know what what do we actually mean by that? Well, how many images go into the photo gallery? What sizes are they where do they come from what happens when we don't have a license for something what happens when? Something's not available any longer what happens when you click on it You can imagine all of the things that can happen when you have photos in the gallery and what can come up they were actually happy that we actually did that because Previously people would take that I want a photo gallery and build them something that more likely than I didn't satisfy their need But someone took the assumption though Well, I think this is what you mean and we actually say wait we don't want to think what you mean We want to know what you mean even if it's not perfect We can interact with it and then we could talk about changing it So that's a good example of a high-level feature might start out with photo gallery, but then we'll decompose very quickly into more detail Once again, what's critical for us is that we are estimating and planning all throughout the lifecycle of this project Starting from the very beginning We're starting with very high-level estimates what we call t-shirt sizing so small medium and large We have a very good idea based on some team performance and velocity and the types of work We're doing so remember where once we're on a centralized platform We're going to be doing similar types of work So it makes it a little easier for us to then project what certain other things will be like So NBC comm took X to do this thing another partner will take a similar amount of time to do a similar thing So we can project in a high level out early and then as we move into the project We're decomposing into more granular Estimates so we go from t-shirt sizing to Fibonacci sequence story point sizing to get our velocities and then when we do the work We get into tasks And so we're at the actual level of effort when we're working during iteration. We're doing ours So we get go along the whole way doing that And the way this sort of translates to the team is that we're doing what I like to call just in time planning So at a high level we're really true our goal there is really to try and get an understanding of how big this project is You know largely is it going to fit in the time frame that we have how many resources? Do we need where do we need to look to sort of leverage more of what another team did etc? When we get into story sizing now we actually have a consumable Digestible requirement that we're actually going to share with the team and the idea there is to get them to Ask questions and talk about it when we get into the task level Everybody's clear on what we're doing and we're really just sort of laying out a plan for our iteration So it's just in time planning. It's not doing that task level at the beginning of the project It's doing it just when we're ready to start executing on that. Yeah, and at the task level I like that the team agrees to the commitment. That's right. That's right not any one person So what example of this is really it's I know this hard to see but I'll show you another slide in a moment But this is once again, we do these exercises with our teams to actually elaborate and get this information So earlier saw one of the vision and we get often get very long vision statements that Basically have a lot more information in them. That's part of the one we saw this although You really can't see it is actually are a very early in a project planning Kind of elaborating on how long some of these big sections will take from the t-shirt size point of view Basically what this is is then put into something like this. So we've translated it into This yeah, so you know this again still a little bit high level This is not a Microsoft project plan that has every task written out, but this is enough This is enough for us to sort of set a date to understand, you know the how much time and This ultimately leads into what I call sort of giving each feature set an allowance You know if we're restricted by a date and resources We're going to control scope right so we want to really Focus on what the priority is of something and sort of give more and allow more of an allowance to that feature set And then things that are lower priority, you know, it doesn't mean we're not going to get them done But we're just not going to spend a lot of time on and this is sort of the beginning of Starting to you know put that together very high levels still We also have to remember that we have executive level sponsorship and support Very often regardless of the methodology we're following people want to have an idea that they're comfortable that we're going to hit dates That we have things laid out that things are planned for once again. We're still agile We're being very flexible about what we're doing when we're doing it managing by scope, but that doesn't preclude us from needing to actually Plan things out a little more elaborately than you often see in some agile projects And again, you know a lot of the things that we're doing because we're using a common platform Some of these things are repeatable. So we can in some sense Start putting this together and really getting giving the the brand kind of a good idea of what the the project's gonna look like So we're really as we move forward from those high level. I just want a photo gallery We move into more detailed requirements Detailed stories that actually start talking about not only what we're doing, but the acceptance criteria for delivering them And we just came from a session talking about Excuse me BDD behavior-driven development, which we very much support and our acceptance criteria Although we aren't doing a formal BDD yet. We write our acceptance criteria so that we can actually number one and very Well defined when we're done with delivering something, but also we start with those as a platform for our testing So it gets into a lot of decomposition as we go along with the more detailed stories At this point in our project. We're also getting wireframes and the difference between a wireframe and a Comp is that this is not a definite thing that we're going to deliver We don't want to ever have our partners confuse a finished looking picture with what they're actually going to get online I mean we haven't we've just started doing In browser design and I think that's really a great step forward to what we're doing when we're working in a certain platform We believe that it's best to move towards showing people what things are going to look on that platform as opposed to a piece of paper Number one, it's interactive. But number two What's on paper doesn't always translate very well into what people are going to get at the end And we don't want to them to have a poor poorly conceived misconception of what they're going to actually get So our rule is that the requirement is what comes first The wireframe is actually a visualization of the words of the requirement We know because we work with teams actually all over the world And we work with teams that speak many languages that a picture is very important for us to be able to translate our words correctly So it's critical that not that we have them But we don't want to confuse the picture with the written requirement We also want to make sure that it's easily updated because depending on whether we have a ux resource on the team or not And often we don't we might have to update a requirement We want to be able to update the picture as quickly because that will be the default that people work from So, uh, again this theme of breaking things down. So Once we get into actually writing stories now, we can actually start to lay these stories out Maybe even epics, which are sort of high level themes Across iterations and what this again helps enforce is How does this how does this go? How does this Align to our timeline The other thing that we like to do is in this actual example We're actually using two teams to execute on the project But we also have things prioritized for our core team And we also have things that our support team is is executing on it as well So again, yes, we're using an agile process But you know, there is a lot of planning that needs to go on when you're working across multiple teams And really the key is to define this stuff start breaking it down in such a way and again that just in time planning Which is getting at the level of detail that you need at the right time So just to reiterate this goes from this which is a very high level plan to this which is a more detailed plan I think that we've all seen and since we're doing this for a while now And we're doing but we're doing with a lot of folks for the first time That they're a little bit surprised with the amount of work that has to go into just setting this up Just getting good requirements just really getting to the level of planning that they really want that's accurate It's very easy to give someone a plan that looks like this And not really map to it not actually deliver to it It's really not that easy to deliver to this and we people expect that we will So that's the big one of the big challenges for us Developments developing something is expensive Working out some of these planning issues working out better requirements is a lot less expensive than trying to figure out What we want what through building it so that's one of the things and the other thing I just wanted to add is that my team's publisher roadmap is influenced by the various implementation planning that happens Right, so if I see common threads where three or four brands need a certain type of functionality I'll prioritize that for for our team so that we can deliver that and build it once and then make it available For the implementations and then then it's less work on the implementation team as well So there's this constant realignment of roadmaps that's happening Continuously and you know one thing we didn't really mention But even though we do sort of plan this out the plans change Right, and so we need to be flexible and understand that but having a plan and then changing it now You know what the impact is right? So if you didn't have a plan and you were just sort of iterating through this and hoping you would hit the date You really wouldn't understand the impact of a change. We certainly welcome changes into our process But at the same time we have to understand what the impact is and it's really our job to really Educate the brands about what those what those impacts are and in many cases it's multiple brands that are affected by that So it's a it's a big challenge, but Through collaboration communication some of the groups that we've set up Across the organization. We're able to communicate that And you know to some level some degree get a little bit of buy-in on that Right. So we're going from the defining process, which is the beginning of our project life cycle Where we're setting up making sure we have really good requirements making sure we decompose to have enough information to start measuring And be able to deliver and now we're going into our delivery phase Which encompasses our setting up our environments making sure our teams are fully in place Doing our release planning as we were just doing and now we're really doing development And once again, we're always thinking about delivering benefit and value And managing by scope To make sure that we fit in the timelines that we're looking for I think one of the you know besides requirements I think one of the most important things that we are doing is really setting up the right team structure And this is hopefully It's a very sort of complicated diagram, but it really explains sort of how we How we have teams different teams doing different activities And again, this is across a large organization. So in some cases, we're working with multiple teams are the output of One particular team's work. So it's really important that we have teams of people and not individuals So, you know, first we have sort of a large team of people who are working on requirements. We have You know stakeholders. We have designers. We have Requirements engineers, which is kind of our our a little bit more than a business analyst All of these people are contributing with our product owner and with our brands to produce requirements. We then have And this is typically for a large project we then have Tech leads sometimes it's usually multiple tech leads on a project So that they can help sort of set the approach, right as all of you know with Drupal There's a million ways to do one thing We're trying to create not only Use a standard platform use a standard process, but really down into how we're implementing the approach that we're taking on certain things We want that to be somewhat consistent and we do that with Our tech leads and our tech leads talking to one another and working closely together and then that can sort of That approach really spans out to the teams So once again, it's it looks complex and maybe a little complex, but Uh, it is complex a little bit. It's not so simple just to say we have a product. I think it's complex because it is complex Well, it is I guess I guess you could say it but once again people don't realize sometimes and this illustrates it There is some complexity going on here. We're being very diligent to manage it in a way that it's Is opaque to our end users. We don't want people to have to worry about this We manage this so that once again our our partners and our clients and our end users They don't have to deal with this so we do Yeah, and so we can answer those original questions like when will it be done? Right, we can accurately answer and we've organized our teams and james alluded to this in a way that we've Actually created a position for people just to help get requirements done Because we found that one of our gaps in our organization and frankly I found on many projects through the years Is that we often work faster than the amount of consumable requirements we have ready to go And so we created a group of people that just helped build the backlog so that we have consumable requirements For the teams all of the time. It's it's something that they just that they only focus on that So this is an actual sample of one of the iteration our iterations are two weeks This is really our day to day schedule and it's consistent. So every every day everybody knows What meetings are happening? What where we are in the process? What everybody should be focusing on and the interesting thing is that we actually in that two week process We're actually starting at the end of the iteration We're starting to plan for the next iteration and that really is Can be challenging And it actually it turns out that developers and testers actually like sort of a consistent schedule Imagine that so by having a rigorous schedule and really sort of moving Have it in earth to get everything in by the that time and keeping the meetings is really important to us because that way We create consistency And really all of unpredictability and all of this is really around that predictability You know, we find that once we start delivering while we still have that end date Sometimes we have brands that have create sort of artificial deadlines within that project And we find that once we start delivering every two weeks and there is transparency and they really know to expect Oh at the demo, I'm going to see These 20 things that the team is working on and I know I'm going to see exactly that A lot of those other deadlines sort of vanish and we start having different conversations And then we can start talking about, you know versus are you going to meet that date? When am I going to get it? We can start relaxing a little bit and start talking about What's the best thing that we can be doing and really start moving on around and you know, that's that's really You know, I think all of us here in the One of the presentations we saw this morning somebody mentioned that You know, we don't just do what we do just to sometimes we do it for a paycheck or Keep ourselves occupied to keep ourselves out of trouble during the day, right? But I think all of us are here focused on quality, right? And so if we can build quality into our process And really allow for You know changes and and the sort of the best possible Approach to things to come forward, you know, then we're doing our job Yeah, the only thing I would add to this slide is that I think predictability breeds trust with our brands And that's something we've struggled with for a number of years is getting the brands to just really Trust that when we say something's going to be delivered every two weeks It actually is and by having that predictability it actually helps with that trust factor So continuing we're decomposing further Into our yeah, so this is just another look at You know that last step we said so we have our you know our high level Sizes we get into stories now We're down at the task level and you know task level is really we have it broken down understood And we're really just trying to map out How our how our teams are going to execute across that two weeks print two weeks is not a lot of time Especially when we're talking about developing and testing In that time frame and being completely Production ready doesn't mean we're always going to production, but it does mean production ready So that means we're breaking things down. Maybe smaller than some some some of you are used to that's okay Maybe we're You know having more people involved in developing one feature, right? That's not so bad Maybe we're trying to build, you know better test cases early on and leverage test cases that other teams have used And also other functionality that other teams have used these are all good things and you know, that's really we really drive it through This what I call aggressive iteration schedule And of course this is also where we're applying during our delivery phase our visual design We haven't forgotten them But when wireframes were enough to start in the beginning to really get started and working and make sure we're delivering The overall framework of what we're building We have in parallel the designers working on the actual design. We're going to apply So we haven't forgotten it We just want to make sure that that the design isn't driving the features that's not the requirement Right and you know, we we can build without this. We don't need this right we can start building Back end functionality even front end sort of how we're going to use that right? This is all theming and styling right at this level and we can do that later in the project and that's sort of where You know that that's that's great because now our designers can really see what it sort of looks like what that functionality looks like And what our interactions are going to be and start doing that work Once they sort of deliver us pictures and Yeah Also, well, this you know, you can start you want to talk talk about this one Start The most one of the other most important things although There's all pretty important and one of the things that we we really feel are really important the metrics We're getting out of this so from the process perspective. I'm looking for certain things out of these I'm looking to make sure that we're delivering consistent velocities that we are addressing the numbers that we're seeing And that I being responsible for process make sure that I'm adjusting the process where I see anomalies in the metrics Or some of the metrics tell me something about where the process might not be working very well Each of us and James and Chris will elaborate use these same metrics for different things So we're actually, you know creating a lot of numbers, but we're actually getting something out of it So, yeah, and you know, I would say, you know, we definitely use like a team's velocity to understand You know predictability, right if we know a team can consistently deliver x amount of points per iteration That helps us with our our obviously even our high level plan, but also our detailed planning And then defects as well helps us really sort of control the the quality that that's happening in the iteration But you know, the other thing I want to mention on this is that this is kind of all that we track You know, we don't get into a lot of really fancy Spreadsheets and you know diagrams and whatever Keep it simple is sort of our motto, you know, just enough These are the metrics we need now. These are the metrics we need today. And so that's really what we're tracking I totally should put in that keep it simple slide. Yeah. Well, just make sure you spell it So ultimately we're delivering what we said we were going to deliver at the end of the project Which actually we're not quite at the end of nbc.com But this is pretty close to what you know something close to what it might look like in the future But the fact is delivered doesn't mean done And so this is part of the process that we want to make sure so, you know, once again We're in a large company. We're delivering value not just a product at the end So once we've delivered this the next part of the process actually begins Which is measuring our benefit and support and we often forget that that even though the development phase might be over And the team might be rolling off of this The business is still obviously very much involved. We're now delivering something Once this goes in front of our audience once this people start using this We're going to have to start measuring whether or not we're achieving the goals we set for this So this goes back to the vision our vision Usually includes some of the goals we've set for this this redesign or this project And so basically what we're saying is once we move to a new platform We want people to do this more or do this less or go hear more or experience this Now we get to start to measure that so that's really important for us We're measuring the benefits. So once again the business is looking to see if they're getting out of this What they expected once again if we're already in production if we're delivering every two weeks We can track this all of the time tracking functionality is something inherent in support making sure things are working We start gathering feedback again because now people are using it so we're getting something out of it and Creating user stories. So we're back to The backlog creation and each one of the Projects that we work on for each one of the brands have their own backlog My core team has a backlog as well and they're all influenced by each other And the point is and the reason why we had that slide titled not done Is that at the end of the project we want to re-groom our backlog and take another look at it And sort of start again and make sure that we're in a good place to deliver more value to each of the brands that are using the product And more about value to those brands themselves, you know, it's interesting because you know I'm in charge of the projects and actually implementing them But that's really sort of the beginning right at the end of a project is really the beginning, right? So now we have a community, right? We have a community of Of internal developers, you know, some of the brands have their own smaller teams that they use We have an internal community of end users and now we have a community of You know of visitors out there that are all using the same platform. So really at the end of the day This is the beginning and now we're really all part of a larger group Which is exciting Yeah, so to summarize I think what's what's the message we're trying to get across to you today is really that this process Although it is complex. It's really designed to Deliver on these five areas, which is a scalable platform in in the form of our publisher product That delivers common functionality to all of our brands So that these things are built once and reused and that's the point of the second bullet Which is reusable parts that come out of the publisher core team as well as functionality that's built during an implementation that could be used for For an input an implementation that's built for brand a could be used for brand b, right? So on mbc.com if they're building something that's Oxygen could use we want to be able to reuse that defining your requirements Obviously, I think we all know how important that is we put a particular emphasis on requirements in our organization Mostly because we've been doing this a long time and we know that without requirements We really can't build anything and we can't project with any certainty when something is going to be done Which is all that you know our brand leaders ultimately care about The relationships and collaboration obviously we're here at Drupal con and we've been coming for several years We want to participate not only and collaborate not only internally within within each team But between teams and with our brand leaders as well as with editorial personnel throughout the company That's a lot of collaboration and a lot of communication that has to happen But one of the things that I've really become fond of is the fact that talk is a lot cheaper than development. So We we put a heavy emphasis on that as well And then the metrics that adam mentioned just help us refine and improve our process And make sure it's sort of like a gut check to make sure that we are delivering value to each one of our brands And we are delivering value to the company if we build something once we want to measure the effectiveness of that Functionality by seeing how many times it's used how long it took to build and what that value really is at the end of the day So I think two things are important. Well one most important thing is when we started this I don't think uh, which was a while ago a couple of years ago. We started implementing this Not everyone was buying into it. Not everyone was sure this was going to work and one of the gratifying parts of this is now that we've been doing it for a while Especially with new folks that have never worked this way before we're getting a lot of satisfied customers They're telling us that they appreciate the way we're doing things that it's a little more painful in the beginning Usually we're getting a little pushed back at the beginning But um We're really pushing them a little bit more than they may have been pushed in the past They're getting good results out of it and really ultimately it's getting the result We're looking for you know a lot of this revolves around estimating and planning and really having a good process In place to do that. So actually just to mention that james and I tomorrow total plug Tomorrow james and I are holding a buff Just to talk about estimating and planning Uh, we could talk about what we're doing But we'd also like to hear what other people do because that's really one of the biggest challenges that we have is really Refining and getting better estimating and planning. So please come to that And uh That's it for today. We'll take questions So yeah, go ahead. Go ahead. So uh, we we try to use Simple tools we're using rally mostly for managing our iterations and managing our requirements We're trying to Use it to We also use it for our metrics as well We're trying to use it for some of our sort of pre-planning But it doesn't really do a good job. So we're really using homemade custom spreadsheets at this point They're just aren't tools out there. But uh, you know once we develop them I think rally is a little bit better on the product side for release planning, but not on the implementation side Yeah, can everyone here because we could I'll I'll repeat the question. You know actually that mic we could pass around Yeah, the question was how big is the core team and how long did it take to release the the initial version Um, the core team now is about 20 people and that's all in that's requirements dev qa product people project management as well And it took really before the the product was ready for the first site to build on top of It took close to a year And that was uh There was a little bit of faith that this company certainly gave us to build that core product But by using this process that that we've been talking about today to build the product itself Um, there was a lot of trust that was gained through predictability. We didn't start that large either It was a little bit smaller. It was probably about 10 or 12 in the beginning Um, but then because we had some good metrics We were able to project that we probably needed a little bit more help in order to Leverage it for all brands, which was the goal all along other questions sure Okay Oh, please We'll get to you next Excellent question So the question was, you know, if we're measuring velocity and we have so much pressure from the brand leaders, how do we Handle that communication and that pressure You know to me, I feel like we handle it through transparency. We don't push back We let them make those decisions. We're just very transparent about What the velocity is of the team what we think they can deliver in a certain period of time And let those brand leaders make those decisions themselves We don't want to force that force their hand in any way. I think that's the best method quite frankly You know when it works well The the brands our partners make those decisions so we have before they even come up actually So we had on this very project mbc.com and we started with the scope of the project We knew that there that there were things that we knew they were crazy They were just too big for us to have well, I don't know if we thought they were crazy But we knew that they probably bigger than what we can handle even at the beginning We had an idea But the partner to their credit said no, we want to keep them in we we understand we hear you we may not believe you right Which is fine. They're really running their product. They're the business But over time when we were delivering consistently they saw that they were getting good quality They saw that we were legitimately working as hard as we could we were delivering maximum amount of stuff They started to realize it for themselves that we weren't going to get all of these things And that it was true right and what we were saying and they literally took out Everything we had said probably should have been taken out in the beginning But they at that point realized it for themselves. It's very helpful to us when they do that They also knew that we could add some another team which we to add capacity. That's always the possibility But the fact is you're always constrained by something And so the constraint is either the amount of things we can do or add more people to do it Which adds to complexity which doesn't mean you're going to get everything still so Ultimately, we're just being very transparent about what's reality is is is the killer here because Traditionally people are are led to believe they're going to get everything or they're been yes to death Oh, yeah, no problem. We'll get it for you. Oh, no, that's not going to be a problem when they don't It's quite a disappointment and rightly so but we're saying look We're going to give you everything we can give you and we want to make sure we do the important things first Which is by the way also a different way of doing things people aren't used to having to make choices Even though they they should be right So there's always a finite amount of time or capacity or something yet in the past people have just for some reason felt Well, we're just going to get everything at some point. We're not going to get everything at some point And so we're being realistic about that up front It's it's uncomfortable in the beginning for some of these folks that haven't worked with us in the past or haven't worked this way ever But ultimately over time. I feel like that uncomfortability goes away Yeah, it's also, you know about breaking things down even further nine times at a ten something that is big The team will come back and say well, that's pretty complex as we break it down We can have additional discussions and that goes really a long way So if things seem too big where they want to get more work in a lot of times We break things down a little bit further and we find out that they're a little bit more digestible too The the velocity is the Is the result of the team? So it's not something that we control. It's a metric that tells us About predictability right and so we can actually if if you know in that slide You'll see sort of some dips and things like that. I can tell you you know almost You know there's there's reasons behind everything the 131 we had some vacations Dennis, I think you were out on vacation I'm 27 and we were still had people on vacation So there's reasons why the velocity dips but predicting that throughout your project Gives you the ability to have that conversation about prioritization scope And you know, uh that that goes a long way Yeah, that number is just a reflection of the point sizes that they were able to deliver not not hours Yeah, and then we have conversations like what would a second team how would that help us? What if we move these feature sets into core? What if we leverage something that somebody else used, you know to start really these are all conversations that we can have rather than Hey, can't you just have the team work faster? You know I think I heard that one the other day or yeah more hours there.  Yeah Now we need the picture of me on the four phones. Yeah Any anything and everything I think is the answer. I mean Primarily we do a lot of meetings, right? But for a lot of the planning meetings for these projects They have to be people have to either be in person or online So WebEx and teleconference calls we do a lot of that for some of our larger projects where We're fortunate to be able to use things like video conferencing. We do that as well So anything we can it's a we really want to foster that collaboration as much as possible And we're pretty religious base camp users. Yeah, and IRC as well. Yep In your point of view Which are the like the skills that you were looking at project manager? If you were hiring someone new for your organization that fits You know your your methodology and and what really matters for you because what project manager might have so many skills Yeah, that's a good question. What three are the the ones that really matter I think first of all, it's somebody who's done this before so done some level of agile Hopefully it's somebody who's sort of done it Obviously successfully but really with kind of a team that was co-located because we're not co-located We have teams all around the world But having that as a reference point for a project manager And what they're driving towards is really important having that vision of like You know, this is what the team should look like even though. I may never get there This is how we should be operating and constantly driving so experience with With agile And but flexible enough to realize that we actually stopped calling our process agile because everybody has their own idea of agile So we just call it the process the process, but it's you know, it's based on sort of scrum And other agile principles. I think also we look for technical project managers So somebody who we don't need to really dive into code But somebody who really understands when a developer says I have this block I need it removed Like you need to understand the severity of that, right? And you need to be able to talk to other teams in some cases and go to you know Talk to the architects on other teams and say, hey, listen, you know, we've got this problem, you know So you need you need those skills and really somebody who is You know incredibly good at communication That's really those those would I would say be the top three upfront yes well we Our philosophy is it's still based on the principles of agile if you if you've done scrum, you'll recognize what we're doing It's not a completely different thing. We use Much of the language of scrum Lean and kanban a lot of basically it's scrum lean, you know very lean thinking in kanban So it's it's not like there's no lexicon for what we're doing But in reality what we felt and james and I have been doing actual agile implementations for a while is that We we wanted to make a process that fit in the organization So our organization This process would work pretty much everywhere, but it might be tweaked again for a different type of organization So what we've done is fit things to work Agile within an organization that isn't necessarily all agile and nor really understands agility at all levels So just now we're starting to interface with upper management and understanding Maybe we can actually help them project and do budgeting with some of the numbers We're having because we're getting that consistent things where most years and most places will just you know Do their usually yearly guests for the next year's budget So we're we're just applying things but what we found over the years just from our agile experience And I've been doing agile before scrum right so before even scrum was scrum that One methodology doesn't necessarily scale or fit all of our needs And so we don't want to be not agile, but we want to find what's maybe a best of methodology that fits in What I like to tell people is I know when it's not agile Right and that's what we really want to do and part of what we're responsible for also is training people on the So we actually do a three-day training course at the beginning that so everyone's Starting from the same place and the reality is that scrum and agile don't do a good job Describing how you work with global teams. They just say you don't do it, right? But the reality is we do And I think we're successful in that and so we I wouldn't say that we've changed it We really take that and it's it's more of a broader Sort of process that goes into more detail into things like this is how you write a requirement This is the type of meetings that you have. This is how you work with global teams, etc This is how you work across two teams. These are things that really the agile community hasn't Defined and things that people are struggling with and really if you think about scrum and most people are familiar with scrum Depending on who you speak with and some of the founders of scrum and they've changed their point of view over time as well They some of them say well, we didn't say these things But we didn't mean that you ignore them like architecture, for instance And some people say if it's not in the book scrum doesn't do it project management isn't part of scrum But project management is necessary to in our organization to run a successful project So what is how do we make that an agile thing? Requirements have a standard form which we've learned from scrum as a user which we use But we we extrapolate we actually take the benefit statement and make that a separate line So that we can actually force people to think through what we're going to measure later So it's not mixed in with something it's separate. So we've just tweaked it a little bit to kind of highlight What we think is important in the process Do you want to try that one? I don't know if I So I I think that what we're trying so we're just starting basically to implement this idea of business value now So we haven't been doing it for a very long time. We're just at the point So what's great about our process is it's evolved since we've been there and the focus on business value is actually something That's organically happening within the organization So we're actually now starting to focus more about what it really means to have business value What it is we're going to measure so we actually use the semantics of a benefit is something That you can actually measure a trend or see so if I have I'll start with the value a value is a definitive number So if I have the goal of increasing x by 10 percent or views or advertising revenue Decreasing something by a number 10 million 100 million. That's a value that we can measure So every iteration we can measure to see if what we're doing is actually working within the organization A benefit is is less measurable. It's more like I want people to watch more videos I want people to go to this page more So the first thing we're doing is introducing the difference between the value and a benefit to the organization And now we're actually just starting to look at how our partners are Measuring things so they can actually take advantage of this So these are numbers that we're just starting to introduce because not that they weren't important before but people weren't using them And so this is something that's actually new in our organization Especially since we're able now people are buying into the process So they're buying into some of the results of the process And I think the value is a little bit easier to Measure on the product side to a certain degree because you could say, you know Did this save the editor time in publishing this type of content or did it, you know Was it reused on a number of different sites without any added? Development time right so that's a value that can be measured But we don't have a lot of control over the performance of a website in terms of monetary performance or traffic Right, that's all a very complex thing that we don't control But we can start to measure against that and we want our our partners who do control it or have more control over it To really start thinking about it So when we say something's not done from our organizational point of view once our thing is delivered We are kind of done But we want them to know that there's something more that these metrics are important So we're building that into the culture of the company more than we so it's a great question I hope next year I have an actual definitive Answer with results for you And a formula I think we got time for one more sure So at the at the task level where it's really at this point we have the stories the requirements broken down And it's really, you know, how much time am I going to spend on that? And yeah, it's it's a little bit of a group effort in some cases We'll have sort of a lead Work with some of the other people that are more junior to sort of, you know Share their experiences and say yeah, I think this will probably take you this amount of time but the But the important thing is that they're having a conversation about it and you know, you can't just sit down and say Oh, I'm going to build this module I think I need this task and that task and it's really about the conversation that you're having Over and over and over again first with your product and requirements team And then with your peers on the on the development side to really understand what that is and at the end What we get is that on that first day of the iteration people are sitting down and writing code Instead of like sort of, you know, spending a lot of time thinking about and dreaming about what this module could be Because we've already done that we've done that ahead of time So yeah, and the whole team has to agree on that on that number right that size including the testers who have to test it So So that's actually interesting that so the leads are kind of uh, you know, some of them certainly participate more in the iteration than others Um, it really depends on the team size Some of them are literally coding and in the team and some of them are more mentors dealing with upgrades to core Um dealing with um, you know, planning and architecting and figure out what modules Um, we're going to use um, and so for their time. We're not really interested. We don't really track that necessarily in the iteration when we're tracking our tasks in the iteration What we're really trying to figure out is how are we going to get this work done? Given, you know, all the tasks that we have to complete and what's our path to do that including testing So it really helps us do more like plan out two weeks. So and then the leads we really sort of uh, Just overwork them to death, you know Well, thank you everybody


Okay, hello everyone, and welcome to the session. We are very excited to be here talking about our Swift implementation, so thank you very much for coming. My name is Max Katch, I'm a Senior Infrastructure Engineer and a member of the Cloud Builders team at Mercado Libre. Together with Maximiliano, we'll show you how Mercado Libre is freeing itself from expensive NFS storage appliances by simply using OpenStack Swift on top of cheap commodity hardware. If you don't know Swift yet, it is a highly scalable, fault-tolerant and distributed object storage solution. It has a flexible and intuitive RESTful API that allows you to store large amounts of data. In this session, we will walk you through the different stages of our architecture. I will start talking about what we had at the beginning, what problems we needed to solve, our move to the cloud, and then Maximiliano will talk more in-depth about our Swift implementation, how we improve the solution, and also he will share some tips and experiences about what we did. But I'm sure some of you never heard about MercadoLibre.com, so let me give you a short introduction about the company and what challenges we have today. MercadoLibre is the leader in commerce platform in Latin America. It is the eighth largest online retailer in the world, and it is one of the fastest-growing companies of 2012. It has presence over 14 countries, including the biggest in the region, Brazil, Argentina and Mexico, and we have more than 81 million registered users. Now that we are an open platform, our APIs receive more than 2.5 million requests per minute, and we have around 4 gigabytes of bandwidth per second. This number increases on holiday season to 4 million requests per minute, and around 6 gigabytes of bandwidth per second. We have more than 1,600 employees, including an IT staff of more than 380 people between developers and engineers. But to get here, we had to walk a long way and change a lot of our infrastructure. So let me tell you how we started. We felt like being cavemen in the Stone Age, because MercadoLibre had a monolithic application, a product, that it was running on physical servers, and we were storing everything in a single NFS volume. As the site grew, we implemented a flat-send virtualization, but we kept storing more and more data on the same NFS storage appliances. As you know, this is hard to scale up, and when you have problems, it's difficult to manage the maintenance. So let me give you some examples of why I wanted to change. Because we had ISKSI block volumes, and we needed to manually create them, and they were directly attached to physical and virtual servers. This was a hard to troubleshoot, and we wanted to change that. We also had NFS volumes that were mounted on top of other mount points, and this was confusing us, and we had also problems because we had even late data loss. We have also performance issues when purging files, because we had millions storing to the same directory, this was out of control, so this seriously affected other applications stored on the same filer. So as you can see, every single event happening on the storage side was affecting our platform, even we had peaks and even downtime because of that. So obviously the wheel wasn't rolling for us. We had more by this time, we had more than 50 terabytes of images and side data stored on the same place. We had also just one storage cluster per data center taking all the bullets and receiving more than 1 million IOPS per second. We were very slow in terms of deploying new infrastructure and analyzing issues and the way things were, we couldn't really handle a caching layer outage. So the performance wasn't good as the IT department was tagged as a bottleneck in the company, and the overall situation was pretty bad for us. So we knew this couldn't go on any longer because we felt like we were chasing an reachable carot, and there was no point adding more and more hardware into our infrastructure. If we couldn't take out some of the complexity that we had in our architecture and in the way, if we couldn't simplify the way we work in our workflow. So we dream about a solution where we could have no vendor lock ins, where it will be flexible, where it will be possible to scale out just using cheap hardware. So we realized that what we needed was cloud and together with it to move the storage layer to the cloud to provide the storage service solution. Then we will be happy. So we got fed up and we decided it was time to fight back. It was time for a big change of paradigm in the way we work, in the way we buy hardware, in the way we deploy it, in the way we use our infrastructure and our human resources. So that's when we decided to take a leap of faith and move to the clouds. And I can tell you that the flight had some turbulence, but in the end we succeeded. And let me tell you the biggest changes we had in our architecture. First of all, we changed from a high dev ops model to a no ops model. What this means is that now, instead of having to interact with our internal users, we are now free and we can focus on what really matters to us, which is developing and researching new products and features for our cloud. Second, we migrated our flat and send manual virtualization and we deploy a full OpenStack compute cloud. Now our cloud users are able to provision their own demands with simple HTTP calls so they can get all their resources they need freeing us and we are not the bottleneck anymore. Third, we stopped making handcrafted virtual machines using the common line or scripts. We started to fully automate the installation, configuration and deployment of these virtual machines using Chef. This freed us also some valuable and precious time because you have a lot of recipes and cookbooks that you can download and just use. Fourth, we changed from stateful to stateless and this was a really important point because we needed to work hard to change our developer's mind and they needed to understand that resources in the cloud will eventually fail. And with that in mind, they can now create applications and services that they are full tolerant by design. So they are cloud ready. And fifth, we stopped what we used to do before that. We created manually, we provisioned the volumes, storage volumes. And now we have storage as a service solution. Now also our cloud users are able to create block volumes for themselves. They can take snapshots if they want. They can destroy them, attach them, detach them from instances. And what's more important, we are freeing ourselves from the NFS volumes. And now they are using Swift as a back end to store all their files. So as you can see, what we did was to change our crappy and flat infrastructure and we transform it into a full infrastructure as a service. And also with this big change, we managed to upgrade our monolithic application and we transform it also into a full open and distributed platform.  Now we are not the caveman in the stone age, so we reached the clouds. And let me tell you why we succeeded in this. It was because we chose OpenStack and we chose also Swift. First of all, we chose them because we love open source. It's now a DNA and we love the philosophy behind it. And I can tell you that MercadoLiber.com is almost entirely running and using open source software. Also, we love that it is written in Python because we can take a look into the OpenStack code, we can hack it, we can debug it, we can change it and customize it to our own needs. And we can also give back to the community, submitting bugs in Launchpad or submitting patches. Also, as you know, open source, you have no vendor lock-in. You can save cost and our managers love that. And also, because OpenStack was built by Rackspace and NASA, they have these big, large infrastructures. So it was built with and conceived with scalability in mind and also it's high available. And as you know, everything in OpenStack is a core component that has a restful API. So you can use it and you can interact with these components. And you can develop your own tools, as we did in MercadoLiber. And so because also we are using it OpenStack since the very least. So we feel very comfortable about it, we know it from the inside out. So last, and finally, our OpenStack grew a lot in the last two years. It got a lot of hype. So it is now a fully mature cloud OS. You can use it and in every new release, you have a lot of very exciting new features. So that's why we stick to OpenStack. And now let me tell you what we are running, what versions of OpenStack we are using in our cloud production in MercadoLiber.com. We are using, for the whole cloud, we are using NOVA, the Essex release. We use it to run more than 7,000 virtual instances that they are running on top of more than 1,000 physical servers, okay? The whole website, I can tell you, is running on this architecture. We have it in production running, it's working very well, and we are very happy. Second, we have Keystone, the Essex release also. We integrated it with our LDAP servers. And that means that any internal users can just take resources from the cloud without any problems and also we use it to authenticate our internal applications, okay? We have also Glance, the Essex release for image provisioning. We mainly use Ubuntu 12.04, which is the long-term support. We also have support from Canonical, so it's very useful for us. And we also have some Red Hat instances images for better compatibility with our Oracle databases. We also have Quantum, we have it, also the Essex release. We mainly use it for better isolation of specific parts of our cloud that we need an extra layer of security on the network side. And last, we have for our Swift cluster implementation, we are using the Folsom release. And now I will invite Maximiliano to tell us more in detail what we have. That's, yeah, okay. So first of all, hello to everyone. My name is Maximiliano Inesio. And I am a cloud builder from macabre.com. And I will start talking about our first open stack Swift deployment. And then we will go through different points that we considered important in the process to get our current open stack Swift deployment. So as you can see in this picture, we installed 24 stretch notes divided in two of our data centers in Virginia. That we come here with four hard disk drives of three data IT each one. 64 chips of RAM and two XACOR processors. We installed the entire Swift package in version 1.4.8. That is the SX release. And we configure all the services, the three main services, account, container, and object services. And also the background services, Replicator, Update, and Auditor to run in all the storage nodes. In addition to that, we create four ring zones. Dividing in both places also. And three, we manage three replicas. So this is to ensure that we will have at least one copy of each object in our cluster, leaving in both places. And we can also see 12 proxy nodes. There are not 12, but we have 12. 12 proxy nodes divided in two groups that expose the HTTP Swift API and also have configure memcache to cache ring metadata. And a lot balancer layer, as you can see, that is composed by big IPFI. We use VIPFI to balance traffic between those proxy groups. And you can notice Kiston there in the middle of the proxy groups. And Kiston is a really important part of our infrastructure. And it's mainly because we have about 100 of departments inside Mercado Libre from our internal users. And we needed the flexibility to the narrow grant, the access to many cloud services and cloud applications, depending on the department and the user roles that they have. So we use Kiston to do that. And we also integrated Kiston with Swift. Well, finally, we connect all that infrastructure with that to a traffic building, a single building. And we use two one gigabit interfaces, two one gigabit interfaces in each one of the storage and proxy nodes in bonding mode, link aggregation. So we went to production like that and all the results was not as respected. And there are many reasons why we could not reach our stated goals. And we listed here, so we can take a look. Slow response times, concurrency issues, big issues with the support from the hardware side, all traffic to the same building, killing Kiston due to the lack of token caching, no more than 60K RPM peaks, and lack in a safe architecture knowledge. So I will walk through those different points and I will try to share you our experience of solving that point. And I will start with Kiston, killing Kiston due to the lack of token caching. As you know, each one of the HTTPs request in Swift needs to be authorized by Kiston by default. And this authorization process cause an increment of traffic in our Kiston cluster. And that impacts directly in the Kiston service. So as a result of that, we got an extra overhead in each one of the HTTP requests in the Swift API. So to solve that, we start to cache in tokens in our Proxies Memcache, okay? And actually this solved that problem because reduce the traffic in our Kiston cluster, but also reduce the overhead in all the private HTTP requests in the Swift API. But we also needed to be prepared to that increment in traffic in our Kiston cluster. So we also added more servers in that cluster. Okay, some configurations, some hardware considerations. Highly good service support and enterprise-grade SATA drives. Well, those are two points that we need to take care using Swift and large-scale cluster of commodity hardware. And that's mainly because all our drives in our Swift cluster are under a heavy concurrent right load around the clock. And this load may cause many broken drives that we need to replace as soon as possible to avoid performance issues. So believe me that we spend a lot of time fighting with our service support to replace that broken drives. So high good service support give us better response time in case of failures for that replacement. And enterprise-grade SATA drives will give us better performance in all our Swift services and also reduce the failure rate of our drives in our cluster. So we think that in this kind of applications, enterprise-grade SATA drives make the difference because give us better performance, more durability and a good balance between cost and benefits. We now use flash drive modules to install the operating system of our storage and proxy nodes. And this comes to solve a problem that we had in the previous OpenStack Swift deployment. And was that we installed the operating system in the same drives that we installed the rest of the Swift services. And when one of those drives fail, the entire server fail, okay? So we now avoid that issue using Sarah Dom's modules to install the operating system. And well, now we are using solid-state drive for ACANN and container services. As you know, ACANN and container services use SQLite file to upgrade data about your ACANN and containers. So with solid-state drives, we improve the concurrency access and the IO performance of those services. So now we use solid-state drive for ACANN and container services and we leave the rest of the enterprise-grade drives, salad drives, to be used by the option service. And on the network side, we upgraded from one gigabit to 10 gigabit network interfaces also to improve throughput and performance in our network networks. Well, I think that this is the most important slide regarding the performance improvement in our current OpenStack Swift deployment. So isolate ACANN and container services from the option services. Well, as I said in the previous slide, ACANN and container services use SQLite files to store data, metadata about your ACANN and containers in your cluster. And on the other hand, the option service stores, retrieve, delete, and update binary files in your file systems with extended attributes. And this last service is the more IO intensive. So when we mix ACANN and container and option services in the same stretch nodes and we configure those services to use the same drives, well, we get performance degradation in all of them. And this is for the impact that the option service has in all our drives. So a good practice to avoid that is to isolate ACANN and container services from the option services, preferably using solid state drive for ACANN and container services, as I said in the previous slide. Okay? So, well, as I said, and I said again, we use solid state drive for ACANN and container and let the rest of the salad rise to be used by the option services. And in addition to that, our users split the amount of containers that they use to improve the concurrent right performance of their applications. And this is because this way they can balance their right operations between different SQLite files. Okay? And this is really important when you have fast clients, that is our case. We have a big amount of fast clients that generate a big amount of concurrent traffic of a small objects in our service. So multiplying the amount of containers is a good advice to improve the performance of their users. And we also multiply the amount of drives in our cluster to have a better balance of high ups. Well, yeah, we also have a better balance of high ups multiplying the amount of drives and also have a better distribution of ring partitions across our ring drives in our three main rings. So now we decrease the size of our drives and increase the amount of drives in our cluster. And on the network side, well, we use two different network bilans to isolate internal from the external traffic. As you know, the background processes, the consistency processes, a replicator, for example, generate a big amount of traffic using our same to ensure the consistency of the data across your cluster. And by the other way, we have a big amount of traffic that comes from our proxies to our search nodes. So we now isolate these two kinds of traffic using two different bilans of 10 gigabits each one. Okay, so it's really important also for us to have a good monitoring of all the pieces of our Swift cluster. So while we use New York to monitor our Swift cluster times and traffic. And well, this is mainly because Mercado Libre has been using New York for a long time to monitor different kind of applications. So here we just took advantage of that and we integrated New York in our Swift proxies. That is really easy by the way. And with New York, we can figure out quickly that we are having some issue in some part of our system cluster, of our Swift cluster, sorry. Or we also can figure out quickly that some user is changing the way in which they use our Swift service. So it's really useful for us. And for that reason, we use New Relic. But you can use also Statsy. Statsy comes integrated with Swift. It's really easy to configure. It's really easy to install. And you can use Statsy to get detailed operational metrics in real time from your different services in Swift. We are testing Statsy and we are planning to implement that in the next months. And we use also Kafka. Kafka is an open source, distributed messaging service. And we use Kafka to centralize all logs from our storage nodes and our proxy nodes in a centralized repository to make future analysis with all that information. So we can make analysis, for example, with Hadoop. And we'll try with it, as you know, try with it is the Swift service that analyze all your drives, all your storage node drives looking for IO errors. So we customize try with it to send us emails as soon as something happened in some one of our drives. So this way we also improve the response time in case of failures. And we have also custom scripts to check the consistency of the ring and configuration files. As you know, we change our ring files many times because we need to add new drives or remove broken drives or change weights. So each time that you make a change in your ring files, you need to make a rebalance, replicate all that files across your cluster. So it's really important to check the consistency to ensure that you have the same files in all your nodes of your cluster to avoid big issues in the Swift service. So we have custom scripts that check the consistency of our ring files across our cluster. We know that there is a Swift Recon service that is really useful to do that. We are testing Swift Recon and we are planning to implement that in the next months. But what if something fails and everything will fail? So, well, hardware failures are really common using Swift and large scale clusters of commodity hardware because of the commodity hardware. And we know that Swift can handle that failures, the most kind of failures. But in some cases, we saw some performance degradation caused by that handling. So we need to know what we need to do in case of failures. If a storage node fail, and we can solve it quickly, we just remove all the drives from that storage nodes from our ring. And we do it gradually by decreasing the drive weight in each one of the drives. If a single drive fail, well, the impact in the performance is obviously lower. But also, if we can solve that issue quickly, we just remove that drive from our ring. And if a proxy service fails, we have good F5 health monitor to detect that broken proxy or that failed proxy quickly and remove that proxy immediately from our F5 proxy pool because that could cause big issues in our safe IPI. So, well, here we have the architecture as a result of all that previous points. And as you can see here, we have now 32 storage nodes we have that are configured with 12 hard disk drives, enterprise-grade SATA drives. So we have three times more drives than the first time. We multiply the amount of drive that we use to have the advance. Our computer also with two solid state drives for a kernel container. So we have the isolation of the service in two layers, a kernel container and the option service layer. We are using SATA domes and, well, we have 64 Gs of RAM and two XACOR processors. That's the same. We divide all storage nodes in the two data centers as the first time and we have the same ring-sones configuration of Rebicus. We have 18 proxy nodes now. We have more proxy nodes to improve throughput that are caching token. So we have a token caching layer now and our caching also ring metadata. We have the same load balancer layer composed by BIPFI and we have a bigger keystone cluster to support the traffic that comes from Swift. And we connect now all that infrastructure with two traffic billions of 10 Gbits each one. So we have the internal traffic and the external traffic bill. Okay, so this architecture looks similar than the first one, but here we applied all the previous points that we saw to deploy a completely different architecture. And well, the results are also really different because we are serving, we are about 70% faster. We are serving 300 K RPM average. We have better service support with a very response time in case of failures and a lower failure rate because of hardware. So we are stronger and we are proactive. Okay, so here we have some lessons now. Harbor biome planning is the key because we have good service support to reduce response times. We have a better hardware. We have solid state drive, we have enterprise-grade drives. We have set the domes modules to install the operating system. We have better network interfaces. Size-matter because we multiply the amount of drives that we use in our Keystone cluster to have a better balance of IOPS. Well, divide and conquer because we isolate some parts of our services to increase the performance. And it's not bound with throughput because we don't have a bandwidth issue by now. So it's all about throughput. And we are human hashing the hashing ring because we need to be sure that we have the same ring files in all our clusters. So we need to check the consistency of all that ring files across our clusters. And hardware we fail and we fail a lot. Well, we need to know what we need to do in case of failures. Okay, so that's all. It's Q and A time. So, well, we have time for a few questions. Yeah, also, sorry. Before that, here are our contact details in case you want to write us or have any questions. Yeah, that's one. Well, that depends of your traffic. We can notice that we need more proxy because our times are low. So, no, we use New Relic to monitor our traffic in our Swift traffic. So we can notice that that we are having some issue in our response times. We need to add more proxies. Okay, but we have proxies today to support a big amount of traffic. Yeah. Thank you so much. How many servers do we have in the Eastern cluster? Oh, yeah, about 20 instances. Yeah. No, it's also for Nova. Yeah, we have it for a lot of services. The core OpenStack components that I mentioned before, we are also using it for some internal applications because we integrated it with our LDAP servers so we can authenticate also internal users and use that. How many main cache? Well, the same amount of proxies because we have main cache in the same server that proxies. So we use proxies and main cache in the same server. Yeah. Actually, main cache comes integrated good with your sixth proxy service. Yeah. With Gluster. No, in the meantime, we are using this. We use XFS and we thought about checking SEF because we also want to integrate it with other parts of our OpenStack deployment. But no, in the meantime, we didn't thought about implementing Gluster. XFS?  In our case, this worked for us because we have fast clients and we know that we didn't reach, we didn't have too much load on the CPU side. We had two extra cores, so it's a big amount. So the bottleneck was on the drives. That's why we choose to put the higher density storage nodes with the 12 drives instead of four as the first implementation. Yep. It's not easy. Yeah. It's not easy, but we use New Relic also to monitor the API. We have also, we are using it to check the server workload. You can check the IO network on the network on the drive. But it's always in your drives, of course. It's usually in our drives, yeah. But you need a really good monitoring system to check which layer you are having issues. It's usually on the, we usually hit it on the hard drive side. We also, sometimes, if you have, that's why we also upgrade it from one gigabit to 10 gigabit to have a better throughput. But it's usually, in our case, it was where we had the problems. No, actually, we are, as he said before, Maximiliano, we are using custom scripts. To replicate all dream files and custom files. To replicate when we rebalance and check the consistency, yeah. We actually are not using it for that. We use it mainly for the installation and configuration of our instances when they are born, launch it. We also want to make this wider, we use Chef, because it grew a lot also with recipes and cookbooks. So we need to implement everything, yeah. Yeah. On the side of disks, yes, 32. Yeah, we have around a quarter to 250 gigabytes, 250 gigabytes, sorry, in terms of size. This is the current implementation we have, but as you know, you can add more hardware, you can scale out and put more hardware because you can distribute the partitions in your ring and then you start moving data and using more space. Yeah, our final size, I can't remember our final size, but give us many petabytes. We have, I don't know. We have a lot because we have, as we told you before, we have fast cloud, internal clients are mainly fast clients. We started to migrate some of the images of what we had in the storage appliances. We started to migrate it to the Swift solution. So we mainly have a lot of small objects between, I would say, between 40K to 100K kilobytes, more or less. We have, we use it also, some of our internal clients use it to store large files, but it's very small, very small traffic. We have a mix, yep. No, actually, as I said, all this moved from the old infrastructure to the new one, so we needed to work and talk with our developers or internal users because they needed to adapt to this new cloud and this new type of resources so they can be able to talk and tweak their app with the APIs. And so for that, not everyone is migrating just as fast as others, but they are trying to do it because we have a lot of benefits because it's distributed, you have fault tolerance, so you have better performance than NFS and you don't have a bottleneck, you have no vendor locking, so that's why it's better and everyone is moving faster or slower, but they are moving. The main use case, well, we have a mix of users, of internal users in our company, so they use Swift for everything. We are planning to move, we are planning to do the pictures, the images of our site to Swift, but we are planning to do that, so it's just a plan. And the other question, factor? Yeah, we have three replicas. We have three replicas, yes. As we explain, we have two data centers where we have this cluster. We have three replicas, so we ensure that we will have at least one copy of each one of the objects. Yeah, in case of a disaster recovery, you have. Yeah, no. No, because they are physically closed, so we have a low latency interconnect between those, but we are planning to move some part of Swift to Atlanta, to another data center, so maybe we will do that. We need to test it, but it's possible. Yeah. The way it is working right now, it's working very well for us, as you saw and as we told you, the first implementation wasn't good and we learned a lot about it, and that's why we changed and we arrived to this solution. And I think it's really coupled to the way your clients work with it. So, because we have fast clients, and the way this layout is working very well for us, I think maybe if our clients change the way they use it, then maybe we will need to adapt to their needs, but because we have fast clients, this is the way this is distributed, it's working very well for us. Sorry, I didn't hear you. Yeah, right now we have internal users, but we don't build them yet. So, we might do that in the future. We implemented this and we are trying to add, as Maximil told you, we are planning to implement stats D that gives you a better metrics of usage about your Swift implementation solution. Yeah, close to silometer is integrated with Swift. I saw that. Yeah, so we might in the future, but right now we are not building anyone, so we just give it for our internal use to see how everyone is using. Every time a new client arrives, we check their usage and their research patterns, but not yet. Anyone else? SSDs? Yeah, we had two SSDs per source node. Yeah. No, no, no. They are not rated together. Rated, if they are rated together? Rated, no, no. No, every, what? Okay, sorry. What you wanna do is to keep the hardware layer as simple as possible because you want to be able to replace, in case of any failure, you want to be able to replace this. So, we use commodity hardware and every storage node has its own SSD drives and enterprise-grade set of drives. So, in case of any failure, you can just take out the machine, the drives, hot-swap them, okay? We don't have more time. Yeah, I think we need to... We can go out, if you want. We can also go outside, okay? Oh, yeah, sorry. So, here it is. We want to thank you and if you have any question or want to contact us, you're welcome. Thank you. Thank you.