Hello, my name is Chris McGrath. I'm here today to talk to you about our app Kero.com, which you have available in nine different languages. I'm going to talk about the lessons we learned doing that and the mistakes that we made so hopefully you can avoid them. And I should point out that it's human languages I'm talking about here, not computer languages. So just in case anybody was confused. So the talk will be in three parts. The first part I'm going to talk about why we did it and how we did it and give some background. In the second part I'm going to talk about the problems that we faced and the mistakes that we made and how we worked around those. And in the last part I'm going to talk about the tools that are available or are not available and working with translators. What I'm not going to cover in very much detail is the content that's already in the Rails guides, the IH&N guides on rubyonrails.com. It's very well written. You can go and read it there and it saves us going through a lot of boring code. So we're going to try and cover everything else around having your site available in a number of different languages. And it's I've got quite a lot to go through so I'd like it if we could leave any questions to the end. So I work for a company called Portal 47. We have two products. One of them is Kero.com, a property portal. That's the one I'm going to be talking about. That's the one that's in nine different languages. And the other one is something we're working at at the minute called Lokalap, which is still very much in development, but it's for managing Rails translations. So Kero is a property portal similar to RightMove or to DAF, but it's targeted at the XPAP market in Spain. It was started in 2003 as a PHP app and it was just in English then. It's now a Rails 2.3 app and it's in eight other languages. In 2007 we added Spanish, French, German, Danish and Dutch. And in 2008 we added Italian, Portuguese and Russian. So just a little bit of background about the site. It's for state agents are the ones that give us the money. They pay our wages and they pay to advertise on the site. The main visitors are people who are looking to buy our rent property in Spain. So the purpose of the site really is to start conversations between these people and the state agents so hopefully further down the line a sale can be made. So why do we have multiple languages? Well, for us because it's advertising it gives us a bigger reach, a bigger market. The more inquiries that we can send, the more conversations that are to start, the more chances of a sale being made eventually. It helps buyers find sellers. If you have a Russian buyer who maybe speaks Spanish but not English and an English seller who speaks, you know, doesn't speak Russian, it's a bit of a Rosetta stone that they could find if the property was good for the Russian that he has a way of finding it and the English seller has a way of finding the Russian buyer. So that's how it helps buyers and sellers. For state agents it's a competitive advantage for them. The people who, if they're trying to get a client, trying to get a property, they can point to the internet and say, look, we advertise in nine different languages and our competitor down the street isn't going to do that. So you should go with us. And we provide tools to help them translate the property descriptions to standard stuff like close to beach or fitted kitchen into any of the languages. So that's the advantage for state agents. The advantage for us is it's a unique selling point as well. None of our competitors do this. So it's something we can use when we're trying to find state agents and it works very well for us. So that's all I'm going to talk about the kind of why's. How is, there's two steps to making your site available in a number of different languages. The first step is internationalization and the second step is localization. I'm just going to define exactly what I mean by those so that we're all on the same page. So internationalization is the process of developing software that supports different languages, character sets or sets of cultural dependencies without modifying the software. So what it's not is programming in a different language. It's not actually doing the translations themselves. It's using a set of tools to repair your source code to be localized which is the next step. So localization is the process of adapting software to support a specific language, character set and cultural dependency. But cultural dependency there are things like date formats, time formats, that kind of thing. And actually language is a bad term because what you're actually going to translate to is locales. And you can see the example here. It's locales are a combination of a region and a language. And you could maybe include say character encodings and things like that but we've been happy enough with UTF-8 so we haven't needed any of that. So that's all the definitions we're going to do. I'm just going to do a very brief, simple example of how you actually internationalize a site just so that when we go through the further problems everybody at least knows the basics. So this is a very simple example. It's using the IATN engine which is the default for Rails. And what you do is you replace the strings in your template here with calls to T which is an alias to the translate method. And in the call to T you put a key. And when that template comes to run, based on the current locale it will look up that key and pull whatever the correct translation is out. So that works very well and if you've got a variable in your string like name here then you pass it in as a hash with the key of the value. The default simple back end that ships with IATN, it uses YAML as the back end store. So you can see there how the translations would look. You can see how the variables are going to work. It's the same syntax as Ruby 1.9 strings. And you can see how it works in Spanish. And it's very, it works pretty well. It's very easy to do. And it works very well for us. What I want to do is a bit of a side note here. I mentioned a main alternative which is GetText. And GetText uses the base language as the key. So it uses the actual string as the key. And it uses printf style, %s variable substitution. So the original library that we used when we internationalised the site first was globalised. And that used the same stuff. That used the base language as the key and %s as the variable substitution mechanism. And we didn't really like that. We ran into problems with long sentences. We're just horrible for keys. And if we wanted to change the base language, the English text, then we had to go and find all the translations for that and fix them as well. And that was a bit of a pain. I'll show you an example later of why we don't like the %s variable substitution format as well. The second. I'm not a big fan of the GetText tools and the mobile profiles and all that because I think they show their kind of CBS heritage. And it's not, I don't like them. Your mileage might be very, you might like them at all. The reason people like GetText over I18N, I've seen a few people comment on this, is really that they think that I18N is a bit too far, another level of indirection too far. So they want to see the text in the templates themselves. So you kind of have to decide yourself what's going to work for you. And I think the actual tools at Ship of Rails now are built on top of I18N, but I'm not actually completely sure about that. I haven't checked it out myself. But it's something to have a look at if you want to. So I'll just do the variable substitution example now. This actually came from the site when we had globalize. So it was %s, %s, %s, and %s, %s.  Or how you translate that? So we much prefer the I18N syntax. You can see what the variable names are. And if you need to rearrange them, which you might well do, I'll show an example of that later, you can do it very easily. If you're going to hire a translator, you're going to have more of a choice, I think, if you're using this style, because it's going to be easier to explain to people what to do. I think with GetText you can do rearranging, but you have to do %$2s, %$1s if you want to change the first and second variables around. So that's why we don't like GetText, but I thought I should really mention it for completeness. So the reasons we do like I18N, one of them is we do like that indirection, we do like the separation of the code from the content. And it makes it easier to reuse templates. We like that when you use dots and keys, it gives you automatic namespaces. So it's namespace, and in the YAML it turns it's basically nested hashes. So we like that because that namespacing and those keys are important later when we talk about context, and a translator being able to pick the right translation. So we also like that we can change the English language, our base language, and it will get, we don't have to go through and find all the other language translations and change those. And we also find it handy to use as a mini CMS, so we'll just have a little tiny key and then we'll have a lot of content that comes out of it. And that's worked very well for us. One idea that I've had that we haven't done is to use it for A-B testing so that you can have a method that returns a key based on a cookie or something in Redis, something like that there, and you can pull out the correct whatever strings you're using for your A-B testing, and you can do that in a bunch of different languages. So that's just something that having that extra layer of interaction gives you. So that's all I'm going to talk about, kind of the high now, and I'm going to move on to problems. So when we did our original internationalization we made a bunch of mistakes and I'm going to cover those now so that you can hopefully not avoid doing them and learn something useful. So this isn't an exhaustive list of problems by any stretch of the imagination but it's the ones I'm going to cover today. So string concatenation, text and images, context which is very important, gender and large amounts of content. So I think these are the main problems that you'll have and probably cover like 80% of your problems. So the first one is string concatenation and it doesn't have to be concatenation, it can be interpolation, anywhere that you're substituting strings into other strings or building them up. And we had this problem all over the site and we had to change a lot of different places to make it work properly. And a good example, a simple example of why this is a problem is pagination. So this is a simple, you know, you've all probably written the top line in that a few times in ERB. You got two variables there, the page number and the total, and if you were kind of being very naive you might think well I'm using page all over the site so I'll translate that separately. I'm using off over the site so I'll translate that separately. You probably wouldn't think that badly but it's an example of the kind of thing that can go wrong if you're adding strings together or interpolating them. So with larger sentences you do have a problem with that. So the reason this particular example is a problem is sentence order. Not all languages have the same sentence order as English. And in Japanese the total will come first so you can see with the bottom example there there's no way you can actually make it make sense in Japanese. So this kind of thing is better done like this. We use a key and you're passing in the variables and then you can see easily there that you can rearrange it however it needs rearranged for the target language. And that works very well and it's easy. So I could go on for the rest of the talk talking about all the different ways that this can mess you up. Instead at the end I'll give a list of links and one of them is to an article called text fragmentation and reuse in user interfaces. And it basically goes through all the problems that you can have if you're naively substituting strings into sentences. And it's well worth the read before you do any internationalisation. So that's lesson one which is the first lesson of internationalisation club is don't concatenate strings. And the second lesson of internationalisation club is also don't concatenate strings. So that was one problem we had. Another problem we had was text and images. This is the second mistake we made. Way back when before we started the internationalisation process we paid an external designer to do a very nice design for us. And this was before modern browsers really so we had to have some text and images. And when we were doing that we when we were doing our internationalisation we suddenly thought oh well that means we need two versions of all these images for each language that we have. And we weren't going to do that. It wasn't going to happen. So we basically had to redesign the entire site. It wasn't just text and images either. It was button backgrounds that were sized to the English text. And you can see here from this is the top nav bar on the side you can see how much more space German takes up. English is actually one of the most compact languages. So in general when you do localisation it's going to take up more space. So it's something that we are fixed with images didn't really handle. So we had to do a whole redesign of the site which was fun you know. So that's one problem with text and images. Another thing to do with design here is the space available. And it's the length of the translations. It's important that your translators whoever is doing your translations can see where the translation is going to be used. And I'll talk about that next when I'm talking about context. If the best word is a long word that's not going to fit they need to be able to see it to know that they can put an abbreviation or a shorter word even if it's not the best translation you know something that's going to fit. So the example here is from the sites from our search drop down. And if you excuse my accent in French it's Salon de Bain. That's never going to fit in there and Chris is our other developer. He does the French translations and he does the design as well. So he knew that STB was what had to go in there. So that's lesson three. It's two lessons for the price of one. It's don't use text in images which is a no brainer these days anyway. Design for different text lengths. Make sure you can handle it. So the next thing I want to talk about is context. And it's it's really the big one. It kind of underpins everything. It's it's something we touched on before. We've been able to see the translation in context to know what space is available and what words you can use there. It's important for the translator to be able to pick the right target language word. And it's a reason for having human translators rather than whacking stuff through Google Translate. A human translator can work out what's meant, you know, whereas putting something through Google Translate doesn't know how much space you have and B if there's a number of different phrases that it sends back to you, you don't really know which one to pick, which one's right. And a simple example of that is the word floor. We use this in a couple of places in the site where we see it in a couple of places in the site. We see it in descriptions of properties. So we see a wooden floor or marble floor, that kind of thing. And we see it in descriptions of apartment locations like first floor, ground floor, top floor. That all works in English. You can use the word floor interchangeably there and it's not a problem. In Spanish, you can see here there's two words that you can use. And the word for the floor of a room is suelo. And the word for the floor of a building is planta. So a translator, if they see floor in their translations that they're doing, they need to know the context to know which one to pick in Spanish. It's a very simple one-word example. You have this problem all over the site and with any kind of phrase the context really is important. Because as well, if a translator knows the context and knows where it is, they can use their knowledge of both languages to make a translation that's of higher quality. So this is one of the reasons why we like the 18N keys. This is a bit of a contrived example but it shows that if you were a translator and you saw the key was room.floor.type and you're translating into Spanish, you're going to be able to go well, that's suelo.  So that's one reason why we like keys. And having a good schema for your key names is going to help your translators figure out the context. Another thing that can help is having a glossary. So if you were doing a flooring supply site, for example, and you're going to be using suelo everywhere, then you do a simple glossary for the translator that says every time you see floor, use the word suelo. And that's a very simple example but if you have a very technical term that you need translated precisely and you need it to be the same all over the site, then again a glossary is something that you can use there and it'll definitely help with context there. It's kind of a no-brainer that if you've got if you're doing ongoing translations, if you use the same translator if they're doing enough work for you, it's good enough quality, keep using the same people and then they'll know what words you use on your site and they'll know what space there is available in your layout and they'll be able to pick what the good words and do good quality translations. So a kind of ideal that I have for the app we're developing is on page translations and this example here is from another app called chufnik.com so it's a web app that runs in a kind of translator mode you can see the words in context you can click on them, you can type in the translation and you press return and in my ideal solution, you know that gets sent off to our app and then magically appears in your reals YAML files or whatever your backend is and you can see how this really solves the context problem you can see stuff on the page where it's used you can see what space is available and it really is an ideal of the kind of interface that we'd like to have I think so this is something that isn't probably immediately obvious when you're talking about internationalization and localization but context really is important if you're going to you know your site, you know what words are used were a translator is coming in generally cold so you need to have some way of helping them find out what the context is so they can do good translations so that's lesson 4 which is make sure that your translators can understand context and find out where stuff's been used so the next problem we had on the site was gender and it's the biggest problem we had after context and because we use English as a base language in English you don't generally have to change the other words in a sentence depending on what now you're using with other languages you do French obviously being the example and a good example of where we have this on the site is property types we have like 10 different property types and we have a number of adjectives that we put together with those and that works okay in English but it doesn't work in French I'll show you an example in a second there's two real kind of solutions to this one is to try and have a key for each combination of sorry, noun and adjective and the other is to try and rearrange the sentence to get around the problem so different keys works best because the translator can pick the exact right language to use but obviously if you've got too much of if you've got too many combinations it's going to be a combinatorial explosion and it's not going to really work so the example I'm going to show you is for I'm not going to show you all 10 property types I'm going to pick three which is apartments, villas and townhouses so this is how you might start off doing it and that works okay in English you can have a luxury villa you can have a luxury townhouse luxury apartment that's all fine, that works but you can't translate that into French because in French you have to use luxeux or luxeuse depending on the type of the property so the way around it is to build up your key dynamically I've just shown a dot key method here you could use table lines and rails to generate your keys and you can see that in the translations that's all going to work and that the French is using luxeux and luxeuse properly and that's great and it's it's manageable for us we have like 10 property types and 8 adjectives that we combine them with and not all of them combine you don't have new land or luxury commercial property so it's just about manageable but when you have more or you have words combined in a sentence it's not really going to work so back to this was the original percent s percent s and percent s comma percent s complex sentence and it's actually a bad sentence anyway to translate because even having the variable names doesn't really help because you're not going to be able to do the gender correctly in different languages and we can see it's a substitution problem as well concatenating things together basically so the actual example where this is from it's from the rss feed title in our site so when you do a search for a specific set of criteria you can get an rss feed of the results and every time you refresh it in the browser if there's any new properties they come up so it's not hugely used part of the site people who know rss use it but it's it's not important but it is something we want to keep around so if we wanted to have a key for every combination in this sentence we've ten property types, there's four payment schemes, we have five location levels which change the text as well and we have adjectives like coastal and inland and I worked it all out and depending on how many of those we wanted to use there's between 40 and 320 combinations and for rss property feed title we're not going to translate it 320 times or even 40 times in 9 different languages on the site it's just not going to happen the way we worked around it was to realise it's a title it doesn't really need to be a sentence as such so we can rearrange it and this avoids the problem and it's actually easier to scan in your feed reader so it's plural property of name, payment scheme title and location name and that worked in this example because we went into a list and when Chris was doing the French translations that's how he works around a lot of the gender problems that we have by just essentially making things less of a sentence so that works okay for us, it's not ideal but it's good enough for us at the moment anyway if we wanted actual French proper French sentences we're going to have to look at another solution for that one thing I came across when I was researching this in GitHub it was called genderise and that had a way of having your translations having your genders in the actual strings so you didn't need 10 or 20 strings or however many combinations you needed so that actually only had it supported masculine and feminine which is fine for French and Spanish but it's not going to work for German because that has three genders it is neuter as well and then Swahili actually has eight genders Cantonese has won thankfully so that's easy enough so that's really the next lesson is to try and avoid complicated sentences in I suppose a general principle anyway of not having big flurry prose on your site but the less there is there are to translate, the easier it's going to be to translate so that's lesson five the next lesson is to talk about is large blocks of content and that's things like emails and terms and conditions and help, that kind of thing the first mistake we made when we did this was we went through our templates and we translated each individual sentence on their own and that ends up being a concatenation problem as well as a context problem so it didn't really work so the second thing we did was the original globalize and Rails now supports having the locale in the template name so if your current locale is spain es niva.es.html.erb it will pick up that template and use it instead of the base language key one so that is okay and it kind of works but the problem we have then is getting the content that's in our files to the translators we use Hamel so we don't really want to have to explain Hamel and keeping indentation right and all that to the actual translators we don't want to have to give them access to our source code and if you're a developer and you sound yourself well I have to copy this template nine times obviously warning bells are going off in your head going well I have to change it in one place then I'm going to have to change it in all those other places so we're not entirely happy with that solution at the minute we'll move towards using keys like a mini CMS where we'll have the content in our actual tool itself we're using to manage translations so one thing it does do is it keeps all these big blocks of content out of your YAML file and you don't really want to be hand editing YAML files especially not if it's all this kind of you're trying to keep a template and code formatted and you want to avoid that so you have to kind of think of the workflow of how you're going to get the large blocks of content backwards and forwards between you and your translators so that's lesson six and that's really the lessons that I want to talk about the there's other ones that you can come and talk to me after I can tell you lots of war stories about doing it next I want to talk for a while about the tools this is the third part of the talk so i18n doesn't have anything built in unless you can't hand editing YAML files as being built in and we've always ended up building our own solutions to do that to handle our translations and I'm going to explain kind of what the problems that we want to solve are to show you why that we've we've always ended up with our own so the main thing that we do is when we launch a feature we launch it in English first and then it so I'm useful back so for a while all the other sites have the Spanish site has some English in it until we do the translations and approve them and when we were on Rails 2 and Globalize 1 we built something on top of that which was we put the translations at the bottom of the page by Monkey Patch and Globalize and we ran a dev mode listener against our production database and they could type in the translations and then when we were happy with them we just restart the production listeners and the site would be all the English would go and it would be in Spanish or whatever language so that worked okay for us it wasn't perfect and we had a list of things that we'd like to do better but it was good enough that the effort required to do all that extra work wasn't really worth it so we still had a list of problems but it was the all lower priority than other stuff we wanted to do on the site so when we upgraded to Rails 2.3 and to the IH and NGM we had to throw all that away and it wasn't a problem to start with because we wrote another tool to one off throw away thing to port our translations from Globalize's database, pull them out of the database and put them in YAML files and put the keys into our source code and that worked okay it wasn't very pretty but it worked and we had our old site back as it were and all the translations in YAML so that was fine but there was nothing really for us to to work with new translations for new features we kind of looked around to see what was available at the time the first thing we looked at was an open source thing called TOE which is 37 signals used that you still have to hand edit your English YAML files in the base language so that didn't really work for us and there's a few older ones that I've noticed over the years but they don't seem to be maintained if you can have a look around GitHub for some of these things so we didn't really see a solution in open source so we looked around for other basic stuff for the service style solutions and there's a few different ones my Gango, WordChuck 99 translations, WebTranslate it the one where we're building ourselves some of those we'd seen before we made the decision to build our own and some of them we'd seen after I'm not going to do a feature comparison or anything now but most of them didn't really do what we wanted there was too many manual steps or just something that we didn't like doing so to explain that a little bit I'm going to talk about just kind of three big requirements that we have for the two, the three main problems we want to solve so I've talked about fallback and the IT and NGM actually provides a fallback backend that you can use it's very easy to use so that's fine we want it to fit into a development process and what I mean by that is we don't want to be having to do very much outside how we'd normally develop code and we definitely don't want to be doing any using YAML around or any of that and we don't want to be having to run scripts to sync translations down we want it to work very smoothly and as I've kind of touched on before we want it to be usable by translators with no technical knowledge we're a distributed company so it has to be a web app but most of our translators are all over the place as well and they have to be able to use it easily we don't want to have to explain Rails to them or to explain HAML that kind of thing so I've talked about the first one already I'll talk about our development process we use the GitFlow model so we'll develop features on a feature branch when we're finished with those we'll merge them into our develop branch when we've got enough stuff on the develop branch and we're happy that it checks out on stage we will do a release to the master branch and then deploy that to the site so that works fine and the problem that we have to solve with this is that when you're developing a new feature you're going to add some keys you're probably going to remove some keys and you're also going to have updated a key that's on the live site so what you don't want is to be changing translations in whatever tool you're using to manage translations and having that break the live site because you haven't deployed a feature yet so that's so we need some way of basically blocking certain translations from being sent to our live site to avoid that until we've done the right deploy but that kind of conflicts with the other requirement that we want which is that when we've deployed a feature in English when the translators do their translations and we are happy with them we approve them we want that to automatically go to the live site without us having to do anything so there's two conflicting requirements there we haven't quite solved it with our app yet it's the next thing we're doing we think we've worked out how to do it technically but the hardest thing is really the user interface to make sure you don't either have fields of checkboxes or lots of confusing stuff on the page that you don't use all the time this is something we want to use ourselves we're pretty confident we can make it kind of smooth and make it fit in with what we want to be doing so those are really business requirements and developer requirements for the translators themselves there's two sides to it really the one side is finding them which I'll talk about next but managing them seeing what their progress is improving translations that's all standard web app stuff so it's not that interesting really for the translators doing the translations themselves I've said the ideal is for them to be able to just look at the page they're translating and type it into that web page that's not something very many people have very many tools have so we want to add in other ways of helping a translator find context like where specific keys were used, what URLs, what path what routes even maybe so that's a bit about tools it's a changing landscape there's a lot of different requirements because it really depends on how your development process works and whether you're happy releasing English versions first and having it fall back if you weren't then you might not have the same requirements as this so I think there would be more solutions over time but that's lesson 7 think about what the requirements are and make sure that the tools you pick and the tools you use fit into your development process so next I'm going to talk about finding translators so the important thing to think about when you're going to go and try and hire some translators is how good quality you want that's one thing you need to kind of have an idea of how many locales you're translating into is it just two or is it 20 because that will affect your decision is it ongoing are you going to be developing this site like we are and you need ongoing translation work done or is it just a one off the most important thing is obviously budget how much money you have to spend and all this so there are different types of translators which one you might pick really depends on your answers to those questions obviously if you can speak all the languages yourself and you're happy handed out in the YAML you can just go and do it you might have friends or colleagues who can speak multiple languages and they'll help you out you might want to hire a freelancer and the website that they all hang out on you can go there and I think pretty much find a translator for any language you want there are translation companies and I'll talk about them in a second you can do machine translation and Google Translate as long as you're happy that people might laugh at your translations when Google Translate there's English ones for you so another thing you can do is try crowd source translations and there's two real ways if you're big enough like Facebook or GitHub crowd source translations can work or if you're small but you're prepared to be patient and find a customer who wants it in their own language and they'll do it for you that could be a way of doing it as well so you have to decide kind of where you are this made up graph here the LSPs or translation companies they are language service providers and they're kind of traditional enterprise-y companies that the kind of people you'd hire if you wanted a printer manual translated into 50 different languages and we looked at those when we were looking at tools and they wanted like $10,000 up front and us to be spending $3,000 a month with them so really it was kind of out of our league and they weren't interested in us below that so I think a lot of people here are going to be not wanting to spend that much money on having translations done so but the best software as a service solution is another is probably what's going to work best for kind of Rails and Ruby applications I think and like I said, keep an eye on what tools are out there because it's a changing landscape we don't have freelancers on that slide but if you hire a freelancer a lot of them use a system called Trados and it basically you can send them if you're doing a one-off project you send them a text file full of strings they'll import them into Trados do their thing and send it back to your translator and that could work very well if you are doing a one-off thing and the way they tend to work to what that software gives them is called a translation memory so every translation they've ever done is stored in there so when they get your file they'll just run it through that see what matches and it works so it's faster for them but translators tend to get paid by the word anyway so them being faster helps them it doesn't really help you very much apart from having faster translations so that's the lesson 8 think about sourcing your translators before you start so assuming you have started and you've localized the site and you don't speak the target language you're going to have to think about the quality control on it so the highest quality we have on our site is in English and in Spanish and we have people who can speak English and Spanish and we can work that out the others aren't quite as good because English and Spanish are our main traffic so that's the ones we concentrate on we'd like to improve them but there's a cost-benefit ratio there and we're kind of happy enough with lower quality for them it's important that we have the site in nine different languages so it's not quite as important that those be very good translations so with Russian we how do we know that's good none of us speak Russian none of us read Cyrillic so what can we do to figure out if that's even in the right ballpark with the translation and the first thing you can try is just running it through Google Translate backwards just to check that it's roughly right so that's just a quick sanity check if you're not you can hire another translator for that language separate to the person who's actually doing the translations to do QA for you just to make sure it's right and if you hire a learning service provider a language service provider that's part of what you're paying the extra cost for is they will have a team of people and some of them will do the translations and some of them will check them for quality and the last thing you can do is just wait for people to complain and see and change it then so it is important to check quality though because this example here is from the Russian Academy of Sciences which is a Guardian article about this not so long ago and they were doing the English version of their site and they were translating the Institute of Protein Research which is the Institute Belka and Belka is also the Russian word for Squirrel so on their site the Squirrel Institute so it's important that you have at least some way of checking that it's roughly in the right ballpark so that's the final lesson is check translation quality so that's all my lessons I'm just going to sum up before taking some questions so if you know you're going to be localizing your site internationalize it from day one it's a lot easier when it's done that way and you might get some side benefits out of having that layer of interaction between your templates and the actual content think hard about what quality level you need does it need to be perfect or not the IT and NGM is not designed to do everything for everybody but it's designed that you can build on it quite easily it's got a fairly simple API so you might have to do that if you're doing something that is not covered by it you might have to write some code yourself or see if somebody else has done it so like I said this is the third time I'm saying this thing but keep an eye on the tools and what's around what's available they're getting better they're improving all the time and hopefully we'll build a good one that you'll want to use and lastly is of course don't use string concatenation so if you don't remember anything else remember locale app and don't use string concatenation thank you very much for listening thanks for having me speak and if you've got any questions I'll answer them now yep yep yes so it's not actually a similar question it's a war story but which is if you're internationalising just do one language first and do it from the start because we had to do this we had to go through all the templates when we were changing to IT and N pulling all the text out and replacing it with keys it's horrible it's done now oh sorry yep we just basically we launch a feature and it's in English and it will appear in English on the Spanish side and on the French side and on the Russian side until we've done the translation so we're happy enough that nobody's really going to complain or if they do complain it's only going to be for a couple of days until we have the translations done so it's just the simplest thing to do we don't have to and then it also means we can work out the bugs in the English version first and we don't have to wait for all these translators all around the world to have everything translated so yep yep yep do you mean functional? I'm not sure what you mean yep the question is about having multiple code bases or one code base and that's really in the definition of internationalisation is making your source code using tools to make the source code work in all different localisations so you only ever have one source code you don't ever have two versions don't start another branch for French you're dead if you do that Dave well the agents can submit if they submit a feed to us they can use most of their systems have little check boxes for near beach that kind of stuff they send that through in the feed and we automatically translate that and we also give them tools if they just want to type in the text to do that so something we'll probably offer further down the line is doing automatic translations that kind of thing the question is do we put markup in the translations and the answer is yes we do in some places yeah in general they are most of them especially ones that are doing web translations are okay with html and probably even erb if we wanted to use that because they're probably used to translating PHP apps as well so they're generally okay with a little bit of technology yep right we don't at the minute we use subdomains to decide on what the locale is our roots are all in English and then what we do when you do a search we don't have a query string it's basically a browse code it's just a little code it's basically all the search parameters squished together and we put localised text after that based on what the code is okay I think that's my time up so thank you very much and er I can drink tonight no


Češtje, ki so tudi zelo, da so vednočili, da so smo vzelo v sezivnih sezivnih sezivnih. Jaz sem Johanni Merlino, rečetnik, v Universitetu of Messina. Zelo sem prezento s mnim kolegjem, Josef Petricomi, s Sebastian Dupont, z Cetik belgijom. So we are presenting here quite a number of results and achievements within a European project, which is called BICON, and it is about enabling federated cloud networking. So, OK, here is a quick outline. There's a lot of ground to cover, so let's skip to the content. So what is the problem? The problem is really about globally operating companies that may need really to deploy tires of their application across different time zones, and about diversifying their choice of cloud providers, of course, for a number of reasons for minimizing cost. What is the approach then? In this case, we are talking about federating the cloud networks. So what we have really is a project which embraces two middlewares, in particular OpenStack and OpenNebula. And what we have here so is a number of developments that really provide us some basic mechanisms for federating cloud networks, as well as advanced features on top of that, such as automated high availability, location aware elasticity, and automated service function chaining. What are the benefits of federating virtual cloud networks? Of course, we are talking about virtual networks. So we are talking really about getting flexibility, having security by means of isolation. Network federations, what we have is the possibility to avoid to manage single instances for networks, but possibly manage all the networks as an entity via API and tools. What are the cloud federation types we identified? We identified three, which are the period federation, where we have really cloud peers that interact through the administration APIs. We have a hybrid kind of scheme, where you have a cloud which really interacts with other clouds via user APIs. And the third type, which is called Brokered kind of federation, where we have an external entity, which is really a multi-cloud orchestrator, which is in charge of interacting with all the clouds. And in this project we have covered the second and third option in particular. And what we have enabled so are loosely coupled scenarios, where we have on one end the possibility to, of course, federate private clouds, such as on the left, the Open Nebula case, an instance with public cloud instances, for instance, AWS. And as we see, a number, for instance, of Open Stack instances. But we also tackled the interop challenge. So also the opportunity to have really Open Stack and Open Nebula in particular interoperate. And so here we have a high level overview of the beacon architecture. Let's focus on the blue boxes, where we have a federation management system, which really interacts with the cloud management subsystems on both sides, on both clouds, which are depicted here. And underneath the network management layer, we have the federation agent, which is really the component tasked with establishing actually a federated networks. That happens by means of interacting with the federated data path and establishing a federated tunnel, as we'll see in a moment, because here we have depiction, a high level depiction of the workflow in terms of federating networks. So we have really the federated network, federated SDN, which is the component which gets a list of network segments. They get passed to the federation agent under the guise of a number of tables, where we have really tenants, where we have networks when we have sites, so really clouds. And what happens is that then the federation agent interacts with the distant federation agent and they get to agree on a number of networks which need to be federated. And then it passes really the information on one end to the federation data path below, so southbound API by means of the open flow protocol to establish the federation tunnel and then relay that information also westbound the corresponding SDN controller in order, of course, to attach the local networks to the federated data path. Ok, here I hand over to Giuseppe to present. Ok. Hello everybody. I won't start giving you a picture of the scenario in which the beacon broker works. We have several clouds spread on a wide geographical area and one or more federation tenant that is used to provide to its then customers the possibility to have a fully federated user experience. This federation tenant has the contract with all or a subset of this cloud and the customer of this federation tenant won't deploy their application only via the selection of the components, the areas where the components will be deployed. To make this possible for the customer of the federation tenant we have created the beacon service manifest that is a custom extension to the hot standard in order to provide the possibility to manage the geographic placement via a new orchestra template resource named OS beacon georeferenced deploy that is a sort of container for the geojason element that describe the area of the space in which the cloud needs to be present. The second important element is the component grouping functionality that is provided with the resource OS beacon service group management. This resource has two important properties field that is the geodeploy, the link with the previously described resource and the group element that is a list for the resource contained inside the stack. And the last element is related to the elasticity management. The resource is OS beacon scaling policy composed by three main element that is the policy type, the link to the geographical area in which the cloud the resource is deployed and the group monitoring that is the link to the service service group management. What the beacon process does is it sets the federation process for the network sharing the network federation sharing. This means that the beacon broker invoked the fedsdn services in order to create the network table all the stuff needed to share the network between the two clouds. It instantiates the resources and activates also the whole process linked to the system manager for these resources instantiated. It manages the geographical placement and the deployment. We can go in detail of the geographic geographical deployment because inside the beacon service manifest the beacon broker takes the georeference resource and make the query versus the MongoDB that extract a set of cloud and the cloud endpoint of these clouds and also the credential the borrowers credential used to interact with those clouds. After this we create the hot manifest starting from the beacon service template. The hot manifest is contained inside the beacon service template and the beacon broker extracts each hot manifest and provides it to the right cloud to the deploy. Now we will start with the video of which is described below. We have here two clouds this cloud is disconnected for the moment and we will show via a simple test with a ping between two VM deployed one in each cloud. In the second cloud we have this VM named connectivity one. I try to go a little bit. Now we will launch the ping and what we say is all the packet is lost. Now we are instantiated the service manifest on the two cloud. What is instantiated with this action is simply a stack named federation and it contains a single resource a zero VM that has a fixed EP. The second stack instantiated is no more that a replica of the first one. We follow this approach because we need to minimize the moving time in order to minimize the latency versus the operator of the customer application. When the instantiation of the VMs is finished the monitoring policy is started and the replicas VM are powered on.  In this moment we have the stack instantiated on the first cloud in the second cloud and it needs some time. I try to speed up the VM is generating now there is a check in order to verify if the VM is the same created from the system and now we can see the output produced by the gcloud connector inside the beacon broker. All this information is a result of the gathering data from the to cloud in order to create the table needed by the federation agent the fedsdn and this table is used to create a connection between these two clouds. The VM in the second cloud is closed and the policy management will... The policy is started. What we can see is the string where there is a display shown the idea of the cloud, the working time in that cloud and the gap respect the UTC time of the active cloud. After this I'm going to change the hour of my laptop where the beacon broker runs and this is translated in the working time of the active cloud in the second line we have a static is 182. This means that the threshold is overlapped and the migration is started. The VM is moved from the first cloud to the second cloud and it will translate in shutdown on the first cloud and activation in the second one. Now we will test the ping between the two VM after this creation operation. We have the VM created via the service manifest and now we will ping the VM in the first cloud that is the connectivity VM in the second one and the ping works. That's all from my side. Let's hand over to Sebastian. Hi everyone. I'm going to talk about a bit of security consideration because when you do federations of clouds new security problems arise. You can have problems of trust between the different clouds that you want to federate. For instance, you might have some clouds that you don't trust in the federation. By default, you should not trust any of the clouds and explicitly specify that on the global level at the federation administrator level you may want to put in place global security policies. For instance, you may want to add intrusion detection on the whole federation independently of the different clouds. There is a bunch of security tools. You may want to use firewalls and so on. Since we are doing cloud stuff here, we may want to use something called network function virtualization. Basically, if you want a simple explanation, when you put a firewall inside a virtual machine, that's a virtual network function. You can scale up to have some elasticity. If suddenly you have a lot of traffic going to your cloud, you can just add more virtual machines to the firewall and it will accept all the traffic and when the traffic scales down you can just remove the superfluz virtual machines. Something else that you can do is change in this example. We have three virtual network functions. One is a firewall, another one is the intrusion detection system and the third one is some kind of monitoring just for logging stuff. In red you have the default data path. Basically every packet will go through the firewall and it will go to the monitoring and then synchronously be delivered to its target inside your cloud. But you may also want to introduce intrusion detection on those packets but oftentimes intrusion detection is quite heavy so you don't want to do that in a synchronous manner so you do that on the side. We never look on a very concrete example of a network function virtualization and service function chaining, which is anomaly detection. In this example we are just looking at one of the clouds of the federation because basically we will apply the same rule to all the clouds in the federation so in this case we have two network functions and what happens is that we may want to detect anomalies in this case the policies we don't want FTP to be done on our cloud because it's insecure and nobody should be using that anymore so the deep packet inspection will detect that there is a FTP transfer and from there it will on one side use the open stack security groups and quarantine the VM through those security groups and on the other side it will tell the firewall to the software firewall to drop all the traffics from that VM so it's properly quarantined. So that's one use case another one would be encryption where one of the clouds you don't really know what's inside you don't trust anything what's inside and in this case two of the VMs in that cloud are actually compromised but we still want to be able to talk to VM number three so what happens is between cloud one and two which trust each other you can have traffic which is not encrypted but as soon as you want to talk to the VM number three and trust it cloud all the traffic will be encrypted so that's another use case ok so now back to open stack because that's why we are here open stack provides a nice little tool to do network function virtualization and managing all those network functions chains that you can have and it's called tacker here is the architecture of tacker but basically you have two big blocks which is orchestration so that's why you chain the different network functions together and the management itself of the virtual network function where you create, delete or update the network functions ok on the software side we are looking at doing intrusion detection for that we are looking so first we looked at N-top and DPI which is a deep packet inspection library and then it will talk to the open stack security groups in the future we will be looking at another tool called snart and on the firewall side it's gonna be probably pf.sense and opn.sense will allow us to in this case detect FTP traffic and quarantine the compromised virtual machine according to a little demo so we are just creating what we call a virtual network function definition which is basically a template for your DPI network function it's created and now we can instantiate a virtual network function which is a DPI virtual machine there we go so next we want to check that it's actually created inside the open stack dashboard there we go so that's my compromised virtual machine which is simply a virtual machine inside my cloud then we have the virtual network function definition and finally the virtual network instance themselves so what we're gonna do now is try to access the FTP server which is on the compromised virtual machine and look at the DPI output to see if the FTP traffic is detected and once it's detected what we want to do is quarantine the virtual machine so on your left you have the DPI output the logs and on the right it's somebody that just will try to access the FTP server there we go the FTP has been detected and now we have a look at the virtual machine and check that it has been properly quarantined so we look at its security group and instead of the default security group it's now associated with a security group called Quarantine which doesn't allow it to be accessed or to access the outside of the cloud ok, so getting full circle we'll also look at another important topic related to the networking part specifically we're talking about having the tools to really debug and troubleshoot the networks so in this case we're talking about networking visualization there's a project for that which is called Skydive and there are connectors for OpenStack what's interesting is that even in this case the beacon project has extended this tool as we'll see in a moment what we have is the federated environment we have the analyzer and a number of agents that are deployed on the hosts and the networking elements here is an architecture of Skydive so as we said the analyzer and the agents there is really a graph processing engine and graph language, description language to have ways to really capture flows a way to enhance on those flows so being able really to make some sort of detections here we have a bird's eye view of a beacon environment which is actually deployed in this case in two clouds and we can see here so the web UI of Skydive and we can see that we can have this very high level overview of the network but of course we can zoom in and so get to the single cloud view so in this case for instance we have a controller and two compute nodes and we can see the little two element blocks which are the VMs which are deployed in this screenshot on the computes and here we have a single node view which is useful to understand what we can see here because we have color coded visualization so both the information about the hosts where the agent is deployed which is depicted here in green we have for instance ports which are down at the moment and they are depicted in red we have the tunnel endpoints which are really instantiated by OVN so OVN really establishing the overlay networks by interconnecting OVS switches instances and those are depicted in light gray and so on the information is really available and when we click on one of the elements in this case is the element on the center which is gray but has an open stack icon on top of it which is black red we can see on the right side bar that there are all the information and metadata which are neutron related so what are the contributions in the project to Skydive we are talking about real time traffic stats visualization which are overlaid on top of the topology we are talking about calculating aggregated traffic over the federated tunnel and showing a button consumption on that tunnel visualizing network load so showing really L2 bandwidth on the topology highlighting as we said color coding really network links based also on thresholds if you want determining bottlenecks in each cloud and on the cloud interconnect on one hand on the other we also have the tools in this case to have a real multi region network topology visualization because we have enabled the definition of multiple separated clouds and their network interfaces so we can explore them and we can also group each cloud network with all its components in a specific area and that is good for let's say usability of the tool what are the impact and the benefits of the activities within this project the integration of course of network virtualization and software defined networking mechanisms and tools with cloud middleware it's very important to stress especially in this venue that the code originating from research is being published under open source licenses and some results are being fed back already upstream so we are talking about for instance OVN we are talking about OpenStack especially the Neutron subsystem and to open Nebula which is the other middleware here is the beacon website is www.beacon-project.eu where you can find all the information you want including also the deliverables which are available to the public and here we have at a glance the consortium so the University of Messina CETIC, IBM Open Nebula Systems University of Complutancy the Madrid and FlexiOps and Lufthansa sorry and also I want to stress that this is Horizon 2020 funded project by the European Commission I want to ask you please to help us by answering our brief survey if possible that is the link and we are open for questions any questions? Thank you very much


Hi, okay. Hey, good morning everyone. Thank you for coming We're gonna kick it off this morning with capacity and stability patterns presented by Brian Pitts Welcome him welcome him on stage Hey, so thanks a lot for that introduction This is gonna be a talk on capacity and stability patterns But before we get into that or get into me I'd like to ask a few questions and like get a show of hands So I kind of have a feel for the audience I'm talking to you today How many people here consider themselves to be like primarily software developers? Cool, so most of the room how many people would say they're primarily like systems administrators or operators All right, a few of us are represented here How many people like don't find that either of these groups, but just want the websites you use to actually work? Yeah, and if I open that up probably everyone would say that's really what they want cool so Hi, I am Brian Pitts. I work as a systems engineer at Eventbrite in our national office So I drove up yesterday. This is my first time at Pi Ohio and first time in Columbus as an adult It's been wonderful so far I'm really looking forward to the rest of the conference and getting to know you and this great city more So I am skyrists on Twitter if you're the sort of person who likes to follow people on Twitter And I have a website at Polybyte where you'll be able to find these slides after the presentation So I work as a systems engineer like what is that means? Let's what does systems in here mean? So this is an unweighted visualization of Software that I took by looking at our configuration management repo. I'm pulling it all out so this is what powers Eventbrite and it's a lot of stuff and My team basically is responsible for the care and feeding of all of these systems and infrastructure Hopefully is at least a few things on there that might look familiar like Python sentry you whiskey If Jane goes not on there It should be in big bold letters because Eventbrite is primarily a Python primarily a Django shop Now what about Eventbrite? The fun thing about talking at tech conferences I usually don't have to introduce Eventbrite because if you don't know what Eventbrite is how did you get your ticket? How did they let you in the door? So we call ourselves as a global marketplace for live events, which is big ambitions in the last year We had 600,000 active event organizers sold over 150 million tickets and did two billion in revenue from those ticket sales Now here's the interesting thing about events Some events are bigger than others This is a graph of calls to one service endpoint involved in purchasing a ticket and In Eventbrite vernacular this shows what we call an on sale just like a sudden extreme extreme spike in activity due to a popular event going on sale so a big part of the job for the operators and developers at Eventbrite is Designing systems that can cope with this sort of traffic pattern now We are far from perfect at doing this and this presentation I'm going to give you is like far from comprehensive about ways you can tackle this problem But my hope is that I'm going to give you some concrete ideas You can take to improve your own systems that you're developing and operating and the patterns I'm going to talk about today generally fall into one of two categories The first of those is stability, which I'm defining is keeping doing processing doing work in the face of impulses stresses or component failures And I said this is how I'm defining it, but actually I'm cheating This is from Michael Michael Nygaard's book release it, which is a fantastic book and one of several that I'll recommend throughout the presentation So in his definition and impulse is a rapid shock to the system It's that hockey stick graph that just is chugging along and then goes up up up when that event goes on sale And stress is force applied to a system over time So maybe that data data store you're using that's on a bad disc and it suddenly started dropping in performance by 10% What's the effect of that over time? So stability is number one the second category is capacity And this capacity is the throughput a system can sustain with acceptable response time and Accessible is a key point here when I've given this talk before I've got questions about like where how you define Capacity I can throw, you know a hundred thousand requests of the system and like they'll come through But that's not a good way to measure your throughput if that first request only comes through after processing for five minutes Because you've saturated your resources and all your users are long gone by then so it doesn't matter that technically you could sustain a hundred thousand requests a Minute if all of those requests are going to a browser that's already disconnected so now I'm going to dive into some of the patterns that we've applied here and after each pattern This is like in the past. I've had people open up the questions after each pattern But because this is a big group and I don't want to go over time I think what we'll do is if you have questions about a pattern or ways We've applied or thoughts about how you might apply it if it's like a generally useful question Hold on to it and ask me afterwards. We should have some time if it's like a more specific about your scenario Or you want advice just come find me afterwards I'll be here over the next two days and I'm like super happy to geek out over any of the stuff Otherwise, why would I even be here doing this right? so pattern number one is bulkheads But the idea behind bulkheads is partitioning systems to prevent cascading failures So this is a nautical term. That's why I threw up a diagram of a boat up here So you can see that within the hull of this boat. There are walls separating Each compartment of the boat. So if we imagine Someone coming up and poking a hole in the hull of that boat Water is going to go in the boat sure, but it's going to be stopped, right? It's not going to fill the entire boat instead the damage is going to be contained and This same design that we can apply to building ships is what we want to apply to the infrastructure we run so Let's look at what I think is like the sort of simplest architecture diagram You'll see in like any presentation someone's giving about designing web systems You're going to have your load balancer in the front that the users talk to you're going to have a pool of web servers Sitting behind that and those web servers are going to talk to a database where the data actually lives, right? And so right now we're happily processing requests. You see I've got all my boxes and cylinders and they're all green Everything's good Now what if some of these requests end up taking much longer than others Let's say the average request is takes a second, but we have some requests that take five minutes Maybe we're generating some very complicated report for an organizer who has lots of events on the system well when the first of the month comes and they decide it's time to go generate all their reports and They start submitting these requests. Oh wait, the web servers aren't so happy anymore The worker processes that were happily chugging along, you know, filling tons of users requests are now All busy trying to satisfy these series of reports that came in and our web servers are overloaded Can't handle other users requests and this one user has essentially brought down our entire pool of web servers Not very cool. So what could we do about this? Well, this is where bulkheads come in handy So we can imagine having two separated pools of web servers one that is handling the general request One that's dedicated to this reporting example. I gave you and so now Yeah, things aren't so great. The reporting web servers are still bogged down but only the people generating reports get that bad experience and our regular users trying to create events purchase tickets They're still chugging along happily on bulkhead a while the reporting on bulkhead B, you know, it was bogged down So at Eventbrite, we do this a lot. We have tons of different pools of web servers and we're Routing to them based on URL patterns the user agents just different ways to classify our traffic So that we can guarantee like quality of service to certain types of requests Now this bulkhead design pattern applies to like stateful services as well I can example I gave you about reports. It would be kind of disingenuous to say well I just bogs down the web server probably one reason it's slow is the web servers waiting on this complicated sequel query to return Well, what if we have a database slave dedicated just to answering those reporting queries and another database slave for serving general traffic? We can make the scope of our bulkhead like as large as we need it to be in order to bound the failure cases that we're concerned with So in this reporting example now bulkhead B and its database labor unhappy generating those reports But again, the other traffic is chugging along just fine So next after bulkheads, I'd like to talk a little bit about canary testing or the gradual rollout of new code Now canary testing is fun because it's probably one of the few terms and like software development that we stole from mining particular coal miners used to suffer like pretty terrible deaths from buildup of gases in the mines Slowly over times a gas like carbon monoxide could build up people would not realize it was happening And they'd be suffering the effects of it, which would make it even worse because you don't notice when you're being poisoned by carbon monoxide So what they realize they could do as a safeguard is take a canary down into the mines with them The canary is much more sensitive to the effects of the gases than humans are So when you're down doing your work, someone is keeping an eye on the canary and if the canary starts to act strange or as our poor bird has happened here kill over, you know that it's time to get out of the mine So we want to apply this same principle to the code we're developing We don't want to subject all of our infrastructure and all of our users to the new code at once Because this is a problem, well then it's too late, everyone is going to suffer Instead we'll roll out the new code of functionality gradually see how this gradual limited subset of it does and then only then if it looks good and we build confidence after a while will we finish the rollout So there's really two ways we do this at Eventbrite One way is what we call baking releases And I've thrown up part of a dashboard that people would be looking at or making a release I intended to show you the whole dashboard but it turns out I'm really bad at CSS So this is all you get So when we're making a new release which we do multiple times a week sometimes multiple times a day of our core product it goes out to a set of servers with any of our bulkheads first and then we'll take a close eye at the health of those servers and compare their health to the other servers for things like response times, error rates, user actions on those servers just generally like does this look okay and only once we're like yes this looks okay will we finish the like kick off the rollout to like the rest of the server fleet The other way we do this canarying is through feature flags For this we're using a fork of gargoyle from Discus and I showed some code here The basic idea is that new features are really many different types of changes When they're released they're off by default So enabling a feature running your new code which is the risky and scary thing is not actually coupled to the process of releasing your code So we see in my example here we just have a gargoyle is active check for our cool new feature if this feature is active we do it otherwise we keep doing what we were doing before which we know is safe and performs and works well So we almost every feature that goes out gets wrapped in some sort of feature flag and this is like chat transcript showing like how someone might submit a proposal and the flag might get turned on So a sample rollout might be First let's turn on this new feature for internal users like within our offices or on our VPN only Then maybe if it's something that changes the behavior of the product opt specific organizers into it who want to beta test it Then start a ramp up of like an increasing percentage of users 10%, 20%, 50% Finally, take it global so everyone gets a new code or new experience And the actual ramp up process will vary depending on what the feature is and how risky it is what's actually changing but the idea is if at any point it turns out oops hey a new era popped up or oops you know wow this got a lot slower we don't have to do any rollbacks this could be weeks into the process we discover a problem we can just turn it right off instantly So next after feature flags like talk about graceful degradation this is turning functionality off in response to failures or load so on a site like Eventbrite and probably most of the sites that web developers hey would work on some functionality is more critical than others we care a lot more about someone being able to create event or purchase a ticket then we care about someone getting recommendations for events they might want to check out for instance we tend to do a lot of graceful degradation through our feature flagging framework which is not the most sophisticated way to do it but it's something that's worked out okay for us so an example might be doing in response to failures so in the dark old days we used to run MongoDB for page tracking and like some reporting features based on top of that so organizers could see like how many people came to their event on different days through different channels if MongoDB went belly up we had a longer feature flag we could go in and turn on just like disabled page view tracking and so instead of servers waiting to talk to MongoDB and getting errors back they just stopped doing that another example would be turning off recommendations during some of our larger on sales so now like our recommendations are generally served by Ajax and have like a dedicated set of resources to it but in the past we've been in line with the event page and so if we wanted to have maximum capacity for people to show up to an event page and purchase events we'd like to avoid doing the work and extra CPU time request time it to generate those recommendations so if we were worried about that we could just go and flip a flag recommendations would just stop showing up at the bottom of event pages there's been some development some experimentation at automating some of this work we have a tool internally we call the velocity engine that right now we primarily use for fraud purposes like tracking rates of actions across the site and one thing people have played with is also piping certain types of errors through there so we can say hey if this sort of error comes through turn off this flag or hey we detect based on rate of certain events we have a volume situation right now we want to conserve resources here's some optional functionality just turn it off automatically so we're sort of playing with that to see if it's worth it or not kind of related to graceful degradation is load shedding now load shedding you're purposefully not handling some requests in order to reserve resources for others so what's different compared to graceful degradation is with load shedding with limited experience to all users you actually serve a completely different experience to users depending on if they've been shed or not for us the primary example of this is what we call our waiting room so during a very large on sale we don't want to have to over provision to the extent that we could literally handle 100,000 people placing orders at once that would be like prohibitively expensive for us to do for the amount of times we need to do that so instead what happens is we have a system where after a certain number of users have entered the order flow for an event any additional users coming through aren't allowed to enter that flow instead they get redirected to a totally different set of systems different load balancers web servers like decoupled from the main side main order processing flow that simply says hey you're in line as soon as your spot is ready we'll send you back into the order flow until then just sort of chill out here and so this is like a really critical way we have to sort of protect the site and keep up and running even when 100,000 people do show up and you still get a good experience like you know you showed up on the page you're not just spinning forever you're not just hitting refresh like can I get in, can I get in which when your customers do that this creases your load and makes it even worse you get a nice message sort of explaining what's happening and then when you're ready you get put through the order flow we also do this occasionally just in terrible emergencies let's say something wasn't load tested properly it may be a really bad query but it's not going to work out and there's no way we can handle traffic to a certain feature or a certain event that's going on right now we might just literally go into our load balancers at the edge and just block that event say before it hits anything that actually runs the event write a code return an error response say we're sorry this page is unavailable to you so that's an extreme measure but a way to give people a really bad experience if it means protecting the experience of everyone else so next is rate limiting from defining as controlling the amount of work you accept so the idea here is you want to understand the capacity you have and prevent exceeding it because it's better to fail fast and give your call or a message that limit was exceeded then cause cascading failures because your system is overloaded the callers of your system are now bogged down and overloaded waiting to talk to you callers of them are now bogged down and waiting and excess of work and overdoing capacity in one area can quickly cascade beyond that and take down lots of systems that you might not even realize are related so it's better to not accept that work than to say yep just wait I'm going to give you a response and then they wait and the areas start to propagate so we do this in a few different places at the very edge of our infrastructure we have some rate limiting in IP tables and in Ginex as well just to catch extreme cases no wait one IP address shouldn't show up and start submitting like a thousand queries a second that's just not legit let's just block that flat out we do this in our application where we have more smarts we can make better judgments about the requests that are coming in and do we want to handle them or not although truthfully right now most of the stuff in the application is more centered around handling and preventing abuse than it is around protecting the systems and you really want to be sure and have this anywhere there's a queue within your system or if you're using external or like third party systems if they have queues as well find ways to limit them an example of some software that does this really well is Elasticsearch for instance and I'll give the pitch now my co-worker John Berryman is giving a cool talk on Elasticsearch tomorrow you should check out but one thing that's wonderful about it from an operator's perspective is for every operation you can do in Elasticsearch a search operation an index operation it has a bounded queue the amount of work it will accept and beyond that it immediately rejects your work which is great because that means you don't get a dump 100,000 requests in there and then have 100,000 processes waiting for a response it's not going to get you can only dump say a thousand requests in there and beyond that immediate failure which you client code can then understand and handle instead of waiting and causing those cascading failures so kind of the flip side to rate limiting would be timeouts so timeouts are limiting the time that you will wait for a request to complete so rate limiting was on the receiver's side the timeouts basically the same idea but it's on the sender's side and it's the same underlying principle that again it's better to fail fast than wait and contribute to cascading failures so again like rate limiting there's lots of levels you can apply this at so at the edge we'll have general timeouts like on requests to web servers and then internally as we have different servers handling requests just timeouts on how long they would wait to talk to the next tier of those web servers or load balancers would have timeouts on request to data stores as well for example if you ever have to wait longer than a second to get your answer back from Redis like something is horribly terribly wrong you probably shouldn't be waiting so wrap it at a second and crucially you won't timeouts on any calls you make in remotely the systems that aren't under your control so a great example for us at Eventbrite is our web hook system organizers can register to say whenever someone purchases a ticket to my event I want you to ping information about that to my web server there's all sorts of events they can subscribe to now what if you haven't heard this presentation and applied these lessons and their web server goes down we don't want our web hook system to be tied up waiting on requests to return to some web server that has been crashed by the load of the web hooks it's received from us right so we'd need timeouts there to ensure that the other web hooks don't get bottlenecked behind those failing ones so that those web hooks can properly fail after a few seconds and then other web server organizers can keep flowing through next would be caching the idea of saving and reserving results to reduce expensive requests and it's kind of a tricky one but it's a powerful one so at Eventbrite it tends to take one of two forms we're saving like computed values from within our code base in either MIMCached or Redis because we have both because reasons or you can save an entire HTTP response and serve that back not do any work in your application and for us we do this a lot with varnish with other caching proxies like Squid can do it for you IngenX even has some limited capabilities this is like the holy grail if you can do this right not even touch your slow application code okay maybe I should back up Python doesn't have to be slow but it might be slow so if you can run it in some fancy fast C code you can track the pre-computer result like that is really keen but it's also tricky right because are you sending back the right result when you're doing caching you've got to think about invalidation strategies and this probably deserves its very own slide so it has one so there's like a few ideas that we've played with this at Eventbrite and this evolved over time like the first idea was just your TTLs need to follow the script stupid so if you imagine that hockey stick stick shaped graph I showed you earlier if we were caching something for even 5 seconds that could make an enormous difference in the amount of requests that we had to actually handle so that was actually like the genesis of most caching at Eventbrite was let's just try caching it for 5 seconds both within our code 5 seconds both our application like event pages let's try caching it for 5 seconds it's a trade-off people can see stale data for 5 seconds but that's such a short time window that is probably okay but once we wanted to get higher cache hit rates and move beyond that we had to come up with more sophisticated ideas for calls within our service layer we built out something that is pretty neat I think it's a centralized invalidation logic so we have a daemon that consumes the binary logs from mySQL which is our primary data store with like the canonical representation of anything within the system is stored and this daemon understands that hey if a change to like a number of available tickets on the event page went through there are these type of cached objects in redis that I should go attempt to invalidate if they exist so building out logic for each service that wants to cache whatever it's responsible for whether it's tickets or venues or whatever else to track the representation in mySQL and then build this one centralized place that can do the invalidation was helpful for us because otherwise with our twisty mess of code base there was concern that we wouldn't be invalidating things in proper places when they were actually updated and we would be caching and serving still data which would not be good but with this centralized invalidation system we were able to over time get confidence in it and ramp up to where we were actually caching things I think for some service calls where we actually got a good hit rate as long as 12 hours so that was a big win and other thing that was even newer and has been a really promising pattern for us is what I'm calling this wrapper strategy to allow us to have a dynamic TTL on pages and this is something we've used in our HTTP caching layer so in that layer if we're just doing straight calls to varnish to decide based on a URL if it should serve a cached response or not that doesn't give us very much control right we can't write too crazy logic there we can't really talk to any of our other data stores we can't really understand like is it safe to cache this for longer or not so what we've ended up doing instead is building both an inner and an outer version of some key pages like our event pages for our inner page for the outer pages this page is never actually cached but it does a minimal amount of work it's designed to be super fast it's only doing really two things the first thing it's doing is it's looking up some data in Redis again based on that sort of centralized bin log consuming process we talked about before like when was the data that makes up this event page like last updated and the older this event page is here I'm willing to cache it for we think it's worth storing it in the cache but hasn't changed in a while it's less likely to change in the future but if the page was to change because we're hitting this outer uncached page first we would still pick that up quickly the other thing the outer uncached page is doing is it's calculating a normalized version of a URL to represent that event page internally let's say you're an organizer on our site and you advertise your event through different channels maybe you own a music venue and you are using our Spotify integrations so people can click through from Spotify and see what what contrasts are going on in their area and land on Eventbrite you would pass through an affiliate code there that we would want to in our code like process and track you might have other affiliate codes coming from Facebook and other places but here we were not caching as effectively as we could because we had different cache keys for each different affiliate code that came through and we were actually missing data that our analysts cared about because guess what, when we did have a cache hit for that event page with the affiliate code because it never hit our application code it didn't get pushed through to our analytics logs and the analysts realized they didn't have a lot of data to properly understand and this is some of our sales channels so they weren't happy so the data engineering team wasn't happy and then eventually basically no one is happy and we have to fix it so this outer event page as part of the lightweight work it does gets to take care of logging that analytics data and then stripping it out so when we make a request to the inner version of the page it no longer has to care about anything variable but it does affiliate codes and now the outer page that ran in the Django application code makes a request for the inner page the user never sees this that request for the inner page gets filled in by varnish based on the contents of its cache so kind of in some ways it feels like a Rube Goldberg system and if I've explained it poorly corner me afterwards and I'll try again but what this has let us do is for like really active events have better cache hit rates because we have that normalized cache key now and for less active events but that was still being requested by users or by bots crawling the site actually have much better cache hit rates because they're still in the cache maybe an hour after they were last accessed instead of all being fleshed out after five seconds finally capacity planning so this I feel like is actually fairly hard it's hard in a way that's different from the other patterns I've showed you the other patterns like as a general operator developer you can sort of figure out how to apply them based on knowledge you have but to do this well it helps if you have people who are actually comfortable with math and statistics and it really helps if you're actually gathering the right data about the behavior of your systems and so for us we're not the greatest at this but we try and do okay there's a few things we do one thing is load testing it's very helpful to have an environment under which you can create load you understand and perform experiments like if I do X I expect Y to happen and oh if it didn't happen why is that let me go address the bottlenecks in the system let me understand how much work I can get done per unit that I want to scale up then have your production capacity be informed by the results you got from your load testing the other thing is collecting those stats on your servers on your application around throughput, around latency around the request and queries and look at the growth of that over time and also look at the seasonality of that which is an interesting component so for us that turns out to be things like we need to spin up more API servers on New Year's Eve because everyone goes out and checks in to lots of New Year's Eve parties on the mobile application just on New Year's night and New Year's day a bunch of stuff like that falls out of it so to recap at Eventbrite we have a number of patterns we've adopted to ensure the capacity and stability of our systems such as bulkheads, canary testing graceful degradation, rate limiting timeouts load shedding caching and planning so I hope that I've given you some useful ideas for how you can take these and apply them in your own systems if you want to learn more about this sort of stuff here's a few books I would recommend these are all quality some like release it have whole sections dedicated just to working through examples and helping you figure out how to build and apply some of these patterns others like artist scalability and the rest through there have good sections to focus on this and put it in the context of like broader systems architecture or operational work thanks for listening I gather I have a little time left so at this point I am going to open it up to questions from the group yes you first I was curious I probably shouldn't talk about that on camera find me later yes yes yes yeah actually there's a few we've used a bit this is something we're not as disciplined about as I feel like we should be I think Varian might be one if I'm getting the name right that we can sort of like interject as a proxy in-between request and like force timeouts force slower responses and that people have tried that there's also we do sometimes just apply strategies like what happens if we use IP tables to drop a certain percentage of traffic or drop this service completely and sort of like see how we handle that so we are the question is do we own our infrastructure do we use cloud hosting we are at this point totally on AWS yeah we have a primary site in US East and a DR site in US West do you want to follow up real quick no not really in depth there's some interesting strategies around services like Lambda but we are more comfortable with some of the patterns we've been applying already yes yeah so that's a good question so the question is you know if you're building out a culture of like pervasive feature flagging like how do you actually clean up your code base because it can be like quite confusing to have to code around all these flags you might not even understand are they still used or not so that's generally up to the responsibility of the developers who have like the team that owns that feature at a certain point they'll feel like you know we're not it's been global for a while we're not going to roll it back so they'll do a code cleanup we do have some tools that we sort of attempt to determine if this is a dead feature flag and encourage people to look at those and see if they can be cleaned up because feature flagging is not without its downsides like I wouldn't want to work anywhere that didn't do this again but it does make some of the coding more difficult for sure and also makes like integration testing for instance more complicated you have to think about the different scenarios and flag states hey sorry so you know I'm not absolutely the best person to answer a question around like how do we test that like flag changes don't like break like user-visible behavior the basic answer is that there's a lot of manual QA and test plan development that would go into like user-facing flag features and so people would be testing different conditions and sort of different scenarios but I would be lying if I didn't say like in the past when there's been like really intensive development across teams in a few areas that that has gotten a lot trickier and we haven't had a good sort of like automated testing strategy around it which means you do rely more on the human testing strategy and things can slip through yes so so yeah so the question was like do we have any analytics on our flags to tell them like when certain conditions are being hit to understand like if they should be cleaned up or not and for the way we use the flags like generally I don't think that is applicable or helpful maybe you could explain to me more later what you're thinking but generally we're setting things in particularly for like a future ramp up in such a way that we understand like if we're saying we want 50% of users to get variant A 50% to get variant B then we sort of know like that level it's usually it's not a flag in which a condition isn't set selective in a way where we don't understand like what the intended behavior is going to be yeah so the question is like how do we handle outages like things can go wrong 24 hours a day I'm not awake 24 hours a day to answer all of them right that would also be terrible so right now we do have team members on my team the systems engineering team who are in an on-call rotation for the entire infrastructure we are in different time zones but not too different so for us the way we tend to do it is we'll have one week rotations we'll all be on call for 168 hours and then someone else will be on call generally like things aren't failing left and right fairly mature company good processes good infrastructure but yeah things do go bump in the night and then someone has to wake up are there other questions are there any hands I'm missing ah yeah that's a great question so the question is like we want to learn more about how Eventbrite does this are there resources we've published we do have an engineering blog which has some good posts actually great python posts I think someone's been doing a series on how we're redoing our like python packaging and distribution internally recently but to be honest I don't think we've published a lot on this so like this talk in the video is going to be probably the main public resource um you know if you want to talk to me more about things and ideas spring running around in your head find me and the other thing I would say is although Eventbrite you know is in a business where we have to deal with this lots of companies are and that these books like really are great resources to draw on and we didn't invent all the stuff ourselves and perhaps we did invent maybe we shouldn't have well looks like that's it so thanks again this has been a lot of fun and find me