eing with each other in terms of what the hashring might look like or what the GFID is on the hashring and we can resolve these cases without any kind of data loss so yeah in this case we're picking by size yeah so traditionally typically what we do is we use the majority policy so we'll pick the majority case and we'll we'll go with that and since we implemented this we really never had any customers come doesn't say hey you bastards you lost my data it's been actually working pretty good so the next thing we had issues with was access control and if you go look at vanilla Gluster and you're not say Ganesha is now an option I'll throw it out there but vanilla Gluster this is kind of what you had to to live with which was this like off allow off deny system where you give it like a list of IPs you can even use wildcards and it would you know yay or nay access into an NFS demon this you know when you're faced with like tens of thousands of clients you can clearly see this doesn't work so well so we hired an intern and we said hey intern solve this problem it ended up being like a really good compartmentalized problem for a summer and you know it's kind of like already a solve problem out there in industry and there's a lot of like well-defined ways of solving this and we chose using net groups to do this and it was something that we actually had used you know an enterprise system so we knew it worked and we had a lot of infrastructure in order to to kind of like generate net group files and effectively how it works is we take you define an export file like this and we have a job in the background that basically just scans these these exports and will create net group files against a really huge database that knows in say a tier called you know a host scheme called my tier for example it'll figure out what all my tier hosts are and it'll generate a net group this is then sent to the to the machines using chef actually and from there this actually like scales and you can actually like control access on thousands and thousands of machines which are in turn limiting access to you know hundreds of thousands of machines okay so another internal thing this is probably one of the scariest charts a PE at Facebook can be faced with you I would wish this was like you know dollars in my bank account on the side but it is not it is memory and this is like this is a memory leak and you know probably last year and the year before you know kind of a lot of the low-hanging fruit problems were being solved and we were faced with stuff like this and you know at the beginning we really didn't know why this stuff was going on you would find this you'd find maybe a a brick that had high CPU high memory you might re-kick it maybe do a state dump beforehand and it would drop down and then you know it may stay down or it may go up but this was really bad and you know machines running out of memory this is even worse than like a machine dying because it's kind of like hobbling and in distributed systems like a zombie machine is like way worse than like a dead machine because you don't know like should I boot this thing out should I keep it in these are like really hard things to automate so ideally you want to like make code changes so they really just can't happen and in this case we this eventually got tracked down to like locking you know you know there's basically some misbehaving clients or a piece of cluster out there that is controlling that has a lock on a file or directory and it's not giving it up and these are also like really hard problems to debug and because they're hard to debug it was really hard to write patches for them because we would see this on a system we would come look at it we do a state dump we'd see tens of thousands or hundreds of thousands of locks pending and then you're trying to like piece this all together to figure out like what series events actually made this happen so kind of what broke the log jam on this was creating this feature called monkey unlocking this is like a developer feature what we do is we we actually like on purpose will drop 1% of lock of unlock requests and this creates these really rare cases into like really common cases and then the idea was okay Gluster must handle monkey unlocking with it on and it must be able to like not block when this thing is operating and once we once we like created this it actually became like you know pretty straightforward on like how to make the patches how to make sure they worked and how to ensure stuff like that graph back there never happened and we created lock revocation and the idea behind lock revocation is if no one contends for your lock we're okay we won't go after you everything's good but if anyone is contending on your lock we're gonna revoke you based on two parameters one is time the other is how many people are blocked behind you you can use them one or the other or both it's really up to you and it's also positive when we revoke a lock we're gonna send you back E again if you choose to ignore you again in your code you may crash that's your problem not ours write v clearly stays in the docks this can happen this is why it can happen and there are the options down there in practice we don't see too many people crashing any again we are blessed with pretty good coders so they're handling a pretty good all right final internal change I'll talk about which is replication so Gluster actually has kind of went through two generations of replication we don't use either of them because one of the things you find when you're trying to like scale things to really large numbers is you need simplicity so the fewer moving parts the better and this was like a case of really that so we created this thing called halo replication which is really a collection of patches that enable this form of geo replication to take place first patch we multi-threaded the heal daemon this we've given upstream this could be useful for other people if they just want to make things heal faster depending on the hardware using most people may want to heal slower not faster but we have pretty beefy hardware so faster is better and this is important when you're like geo replicating obviously because if you're dealing with high latencies you want to like get as many packages you can in the air at the same time the other thing is people come to me and they say okay rich this is really great the first question they have is like what happens when actually I'll get back I'll get to that in a second non-destructive GFID split-brain resolution this will be nothing until probably the next slide but it's really important and basically what it is is if you have two copies of the same file that have a different GFID if you're not familiar with Gloustra it's basically like an I know number on a file system what do you do and we created a patch to use similar techniques as I discussed on the split-brain to resolve these cases in this case though it's non-destructive we do not we rename the data and finally the halo feature itself at its core requires only one option it'll actually figure out for you how many data centers you have and it will you know take its best guess and form replication zones and within those zones you will have asynchronous IO reads and writes if you're a geek like me you may want to like tune this and you can tune things like the minimum number of replicas that you want before you acknowledge or write as well as the maximum number of replicas you may want even because maybe within a zone you've got six replicas because you're Netflix and you're you need like a lot of read capacity so you may say you may limit how many maximum replicas you do synchronously and then you can decide if you want to fail over enabled or not which is enabled by default which is basically if some failure happens you can decide that even though there's only two replicas in your zone you may want to bring in another replica from another zone at the cost of hire of latency reads and writes but you want that extra durability so so you may want to fail over enabled and also we have this notion of min samples which is there's at its heart as we look at these pings and we're kind of this is how we figure out like where all the data centers are and min samples is like how many pings do I have to see before I actually start making calls and the system goes in like a synchronous state until that many samples is is received so this is the way you can think of Halo geo application we've got like three different data centers here we've got kind of maybe this is like the west coast this is like the east coast and on the east coast maybe there are 12 milliseconds apart and then it's like you know 6570 to the west coast and Halo is going to form replication zones based on what Halo setting is set in this case it's about 10 milliseconds so it says anything that is within 10 milliseconds and this is like the brick nodes themselves so if you're like an nfs demon you're trying to figure out like who should I be talking to synchronously it's going to do this based on the Halo value so it's going to say okay I can see you know maybe 20 bricks in my zone and they're 10 milliseconds away from me that becomes I'm going to talk to them synchronously and everything else I'm just going to like let the heal demons do it asynchronously but maybe you're a weird customer and you that's not good enough and we do have some some folks like this that they're like one data center not enough I need two data centers Comet hits takes one out I need my data safe but I also want a third copy over there for maybe some other reasons you can actually do this using a fuse mount and you can actually just say this is you know I want 20 or 30 milliseconds inside for my Halo and you'll get two data centers synchronously so fundamental thing it's like extremely flexible and then of course we have heal demons these are the guys that are actually pushing data between the data centers and for these guys they just use infinite halos so they're just they will see everything and talk to everything and be able to shuttle data around so just kind of some of the the reasons we like Halo and I think our customers do too super easy to use all the standard tools work you can use a little manifest you can use GFAPIs NFS CLIs and the fuse mounts it's got some cool behaviors as partition tolerant if two regions are up but disconnected they can both receive rights and how we do this is actually using that GFID unsplit logic that I mentioned before will actually allow you to write to two regions simultaneously and we will be able to handle the case of figuring out like who wins and when we do figure out who wins we're not going to blow away the loser we're going to just rename them on the way and we do it pretty much like what a local file system would do which is like the last writer wins and it's pretty performance so six hosts we can get up to a gigabyte a second 450,000 files an hour if you're more interested in files per hour and this like this scales perfectly so if you add 12 18, 24 whatever you'll you'll you'll get it so future work and current challenges hardware raid this is something that we are looking at removing from our stack we do not know really how we're going to do this yet it's actually a really hard problem to get rid of NVRM so you know my first inklings are kind of I think this maybe one of those things that if we can actually solve and get a ratio coding into production and really hardened these guys data labs in Spain did some awesome work creating the disperse translator what we're going to do is try and get that in production in our systems and really really hard in it so it works at our scales and we're hoping that maybe that maybe the key to J bought as well we don't know yet and then the other thing is multi-tenancy we're basically making all of our clusters look the same act the same if you become one of our customers you're just going to get a standardized sell and you may or may not live with other customers and another pretty hard problem because we have to like QoS and provide things like self service and we've got QoS down pretty good we actually have a patch that will be upstreaming probably on this quarter which is to do throttling at any directory level and that's kind of been the key for us to get multi-tenancy to work that's it yes yeah so that's cross-country yeah and they're both writing the same file so if they're both writing the same file and if you're doing the read async they're both going to they'll both be reading the data that they're seeing within their region um we allow so depending on the customer some people want they have really really high consistency requirements and for those we have to basically tell them listen like you can't really have it all you can't have if you want 100% consistency we can give that to you but you're going to be in sync mode so you'll have the georeplication but we're really going to have to consult all regions in order to like answer that perfect to give you that perfect answer to your question in terms of consistency so so Gluster is going to Gluster does granular locking so as it's replicating that file well it depends so the the very common case to say it's a brand new file that one's like way more easy it's you know a file might be being written in the west coast and red in the east coast the first thing Gluster is going to do is create a file of some of the it's going to fall fairly significantly sized file whatever it sees at that moment on the other end and it's going to begin back filling the data Gluster then has like granular locks that it's going to enforce on readers in that region and they should be able to they will not be able to read past the lock so they should actually get pretty consistent reads in that case the more exotic case would be like the random right case this is something we just tell our customers don't do this we do not really support this the granular locking in theory should should protect you meaning that while the replication or the heal daemons are actually replicating data into that regional file it's going to be locked you won't be able to read it but this would not be something you're going to want to run like a database on like we'd tell them like you know use database database systems have their own replication mechanisms because their requirements are very very specific so this is more for like very you know like photos or videos or things you basically you open you write it and you want to replicate it go ahead not open source yet so we had a pretty down pat on three four on three six we're we're almost there I think but I think three six is really what most of the community is going to want to run it on so there's kind of a few final things we're kind of like touching up and we don't like throwing patches over the fence that are like not hardened because we're since we're like writing the patches we can roll a bit faster and looser than like an end user so we don't want to like get patches out there that are really not baked so what yeah you know it'll probably happen probably before the Gluster Development developer summer we'll get it out there yeah yeah so internal customers so like what's that might be one of them instagram might be one of them like these are all to us customers yeah yeah yeah so photo video to be clear this is like ever store haystack this is their bread and butter we may store bits and pieces of that but that's usually for like not for front end access this is usually like people might be doing like experiments or trying out different you know video codex or or these kinds of things but like you know we you know someone comes out and says I want to store photo video boom we shoot them over to the team that is best suited for that go ahead so it varies on the cluster I would say the number I do have taught my head is probably 80% of the F ops on a Gluster cluster in our systems are actually metadata and only 20% are actually reads and writes in some it gets as high as 90% in terms of reads and writes I think the last time I actually broke down that stat it was almost 50-50 actually which actually kind of surprised me because using on like NFS style systems it's like you know there's usually a lot more reads than there are writes but for our users they're pretty heavy so it I think last time it was about 50-50 yeah like Gluster at Facebook yeah just those six guys that were at the front of that presentation so yeah yep so currently we depend on our hardware raid cards to actually do that at a block level the background scan is all done by the hardware raid cards at the at the block level so Gluster actually has bit raw detection coming we don't use that yet what we're probably going to do is roll that into our J-Bod project because with the without hardware raid that is something that like you have to own and do so right now we could pretty much like delegate that to hardware raid and hope it does its job yeah we actually have since we run about 20% in ButterFS it's been a good gut check to see like is hardware raid like lying to us and the answer is not really like it's actually fairly rare we have run into a few cases where ButterFS has pointed out corruptions that the hardware raids that did not catch but they're rare enough that we were that we weren't like concerned and we never have never seen it on three replicas so in these cases usually just drop the bad data on that replica and you just reconstruct yep no so not yet this is something we we actually did our own snapshot work using ButterFS we've done some experiments with LVM to see like talking to the Gloucester guys at the last summit these they seem to be pretty confident in LVM snapshots and it is like file suspect non-stick so it's got some nice properties there but in our early experiments with it we're still kind of not super pumped about the performance of it so our customers are like really they don't like blocking like if anything blocks for almost any reasons we hear about it so yeah at the back so it can be it can be a hundred percent or it can be 10 percent it's really up to the operator of the cluster we used Gloucester is actually I think one of the cool things that Gloucester designed in I think pretty early on is they've got multiple queues for different operations so you've got high-pride queue you've got a normal-pride queue a low-pride queue and a least-pride queue and you can actually operate heal daemons and a least-pride queue and that enables them to not like suffocate out your production workload so typically what we'll do is we'll break down we assign threads to queues based on how many cores our systems have and then we typically would do like two or three threads on a least-pride queue so not all of our clusters are running that today because we kind of really only got comfortable with the least the notion of least-pride queuing in three six and three four I think it either worked and not so great or not at all I can't quite remember but yeah at least-pride queuing would be your your best bet yeah so if I had to pick I would probably keep it because I actually feel it's like a hard problem that's been like well solved and you know I feel like the you know I you know utmost respect for the guys that adapt tech in LSI I think these guys know what they're doing and are really good with it so I think a lot of this is driven by like people wanting more metrics of like what the individual drives are doing so I think like the message to Navy the hardware raid vendors out there is like expose a lot more metrics because there are power users out there that want to know like individual like latencies to individual drives what are the drives doing all that kind of stuff so I think some of that is really driving it from like an engineering standpoint yeah so the the throng feature that we're putting out is designed to really handle those situations now this is brand new to even our team typically in the past what we did is for if we saw a really heavy hitting customer we'd actually put build their own cell and move them there in kind of the the new world to really kind of like help us scale is we really need all the cells looking identically if they are truly big enough like they're a multi petabyte kind of use case they may get their own cell but it's really by convention not like from if you look at our config layouts for example it's going to look like any other config so with throng what we'll do is we'll go to a customer and say hey how many fops do you need odds are they're going to be like what the hell's an fop and I have no idea how many I need so what we may do is be like okay run with your like run with your workload we'll then track in their namespace how many fops we see and then we'll say hey that's great this is the cap we're going to place you at before we're going to shunt you to that least price you I just talked about and and you know and in some cases they just may need to change our workload because maybe you know a lot of cases you see people that are doing like you know a 4k read or something you're like well why are you doing a 4k read and they're like I don't know and we're like well go look at your code because like you know sometimes they're so far abstracted using various libraries they have no idea what like that read actually looks like when it gets down to the syscola all right I think I'm out of time but you can pull me aside and I'll be happy to answer any questions thanks test test oh are you able to hear me oh it's loud is it kind of loud is it okay all right thanks so I uh excuse me sir yeah we had one spare connector so it's all set now all right thank you all right should we get started yeah 430 all right good evening everyone thanks a lot for coming to the presentation for me this is the first time at scale and I've been having a blast here the organizers as well as the community has made this even such a huge event and a successful one thanks again for coming all right as you can tell from my accent and just by looking at me I'm actually from Sweden just just kidding I'm actually from India and let me take you back to your time when I a long time back actually when I did my undergrad in India I was I used to remember I still remember actually that I used to love going to the computer labs that was my passion but unfortunately in my university for my class we were allowed to go to the lab only on Tuesdays between 4 pm and 6 pm things a lot better now but back then it was in the case and I had to find a way to make my way into the lab at that time and get a computer and do whatever I want to do and then come out it was very constraining but at the same time I kind of learned how to make it work in that environment then I moved on to U.S. where I wanted to do my masters and here the first thing I did is the registration desk I asked about okay what is the lab schedule the registration day was coincidentally on a Tuesday and they said okay here you go here are the keys to the lab and I was like thinking I was shocked really? so no not only do I have to come on Tuesdays between 4 pm and 6 pm I have to come here get the keys every time and then return it after I used the lab like that's ridiculous that's worse than the public restrooms and gas stations this is how I was thinking and then the lady clarified to me and said no no you get to keep the keys you can use the lab anytime you want this is just like your facility I was like wow this is awesome this is dope y'all peace actually I said it more like thank you madam really appreciate it but you get the point you get the point so I was I collected the keys I was super happy I was doing my happy walk to the class and I was thinking in India the safest place to sit in a classroom is the last row next to the back wall so that it's like you don't see the prof a professor and the professor doesn't see you it's a good setup that way it's like the lines marking their territories if you will fair you don't ask the professor any question professor doesn't bother you you just get your degree done get the heck out all done all right but I came I stepped into the U.S. classroom and I was looking for the last row and it happened to be the highest row in the entire room the room's kind of elevated and I was like what's going on here this is like a zoo animal a lion from the zoo getting exposed to the forest animals now so I was like I remember thinking the very first day I'm screwed I'm screwed so basically I had to I had figured out a way to learn about learning in India and when I came to the U.S. I had to unlearn everything I had learned about learning and then relearn the ways to learn here given that I used the word learn like 50 times in the last 10 seconds you guys probably figured out I'm going to talk something about lessons learned yes my name is Victor Gajendran I'm from the ticketmaster team and I'm here to talk about lessons learned by doing DevOps at scale at scale all right that was a complete coincidence anyways so before as soon as I mentioned ticketmaster to people there's a popular question that comes up let me throw it out there clear up the air I'm sorry I cannot get you free tickets terribly sorry about that all right so now that we got that out of the way let's talk about what I'm going to talk about in this in this presentation obviously I'm going to set the stage give you some idea about about give you some idea about I don't know why this lights automatically moving give you some idea about why why ticketmaster is special what's our environment to where we come from and all that and then I'm going to take you to our DevOps performance and then finally I'll also take you backstage to show some lessons learned things that we picked up along the way all right let me pause at this moment and tell about tell a little bit about what this talk is not about actually let me show a little bit of code I'm showing the code just to show that I'm not going to show the code in the presentation it's because trust me I've looked for programmatic ways to solving DevOps in your enterprise scale but there isn't any so if you're looking for starting a great open source project to address a market segment that has not been fulfilled yet untapped opportunity go for this programmatic solution for DevOps for enterprise and this is a project that will not get you anywhere sorry give me one second slides moving automatically let me fix this all right it should be back now okay so so let's get going so who are we we are ticketmaster as soon as you think of the name you think of tickets and these are people that sell tickets right but what we actually do is we connect fans to great experiences that we kind of bring them together and that's the specialty and basically what it takes is a deeper understanding of what the fans needs are and what their preferences are profiles are and they're matching them to the right opportunity so let's take the experience like even 10 years back like how was the ticketing ticket buying experience back then so some friend in your neighborhood probably told you hey there's a big show coming up and then you wake up that morning if you get box office opens around 10 o'clock so you go stand in the line as you as you walk into that place where they have all these counters your mind makes split second decisions as to okay there are 50 people in each line that this line moves at this rate so i'm going to get my ticket faster if i go to line number three so your mind makes this complicated decisions very quickly and then you pick a line using probability expectation that you're going to get the ticket faster than the other guy right as you are in this line and the line progresses slowly you start to wonder okay am i going to get the ticket when i reach the friend of the line you're anxious you're just waiting but the guy in front of you as it as he reaches as you reach closer to the counter a guy in front of you picks up the phone and start talking on his cell phone and you wonder what the just get out already get out of my way right and then he goes to the counter he buys his ticket at some point he picks up his big bag and then pulls out his checkbook and starts asking you who should i address this check to and then can you spell this for me oh it's like come on just get out already please don't judge me i think all of you have been in that spot where you have somebody in front of the line just blocking the traffic so just just compare that experience to what it is right now with ticket master now you don't even have to worry about which shows to go to in fact ticket master makes those recommendations for you and and then when you you could buy it at the convenience of your home and then it is also verified tickets meaning if you buy a ticket from ticket master it's guaranteed that you will enter the venue and once you're in the venue we also sell technology solutions to enable you to order in menu solutions solutions like you could order food and beverages from your seat right to be delivered to your seat so things like that so we have a wide range of technology and products that we offer and and that's what makes our company unique now we've been doing this for over 40 years so you could think of us as not just ticket master but it's more fair name I would call us the experience masters all right so our technology view so we have a wide range of technologies this is a company that's been around for 40 years like I said so we even have wax computers actually not the physical ones looks like they don't sell it anymore but we have wax emulators running on lx machines on which we run some a small piece of our software and that's on one end of the spectrum on the other side we have modern containers Docker and we use configuration management chef and we use Kubernetes we've started looking into that so there's a wide range of technologies we have pretty much everything in the middle as well so you got a note one thing ticket master has been the leader in the ticketing industry primarily because all this technological innovations have driven its business innovations all right so we do have we are a big fans of technology and there is a lot of investment that goes into building great technical products at ticket master we have about 20,000 plus virtual machines and we have a whole sale warehouse type massive data center that hosts all these virtual machines and to so one might wonder like how do we how do we operate these these this type of massive infrastructure at scale without automation so obviously we did build our own automation this is back in 2002 before chef and all existed our engineers got together and they built an internal configuration management solution that is that is still being used in some cases in the company some of our engineers went to the chef training a couple of days back and I saw a tweet in our internal Slack channel a ticket master's internal Slack channel that hey the way chef does configuration is very similar to our internal tool and that that to speaks a little bit to the type of the engineers we had back then we also had another set of engineers that tuned to the kernel of the operating system primarily Linux now you might ask why why in the right mind would somebody go and tune an open source code well it breaks the it folks you out so you won't get any support and you can't like and because this is a special use case you can't give it back to the community and get it supported henceforth but there is a unique reason for there's a specific reason for why this happened and that has to do something with our traffic pattern ticketmaster the the the website ticketmaster.com it has heavy demand at times and we go from zero to 40 million transactions in less than minutes this is a very unique property to ticketmaster because there are lots of people out there companies out there that have much broader load on on average but at the given time this type of spiky traffic pattern is kind of unique to us if you were in the auto industry we'd be like beating Bugatti Veron which is like zero to 60 to one five seconds you'd be faster than any of that right sometimes our traffic pattern so spiky that we thought that we are being d-dost by by some hackers from some university right but then it turned out to be our customer traffic so there are two factors that explain why we have such traffic pattern well obviously the number one is we have 54 million unique monthly visitors coming to our website and these are fans the loyal fans that are keeping coming back to us for buying tickets and then we have something called on sale events what's an on sale event the little bit of refresher on the technology sorry ticketing industry the way ticket ticket sales works that an artist announces a show and the inventory is made available it's made available typically at 10 o'clock eastern time and then it is made available to that that people in that region and then they open it up at 10 o'clock central time for that that group of people and then mountain time and pacific time at 10 so from the west coast perspective you start seeing peak loads at 7 a.m. our time 8 9 and 10 a.m. so that's why you see huge traffic patterns so basically think of it this way on sale events are events where the tickets go on say for a broad broader group of population that usually the venues are smaller I mean relatively smaller for the demand that that they some of the artists see so a lot of people competing for a small set of tickets so that's what causes this kind of demand spike and and crazy all at the same time kind of a influx traffic so and and also most of these key on sale events have happened on Fridays in other words every Friday is pretty much like Thanksgiving Black Friday for us where you just have to don't have to fight over a $50 TV but you can do it at the convenience of your couch but at the same time from our perspectives like lots of people coming to us to buy our inventory all at the same time and that results in a traffic pattern that looks something like this this is an op-net graph from one of our real on sale events that happened recently a couple of weeks back and as you can see here at seven AM up until seven six fifty eight or so there's not much traffic but seven AM you see a massive spike in traffic it rises up like it's nobody's business and then comes down a little more gradually as we start serving so this is this is one of the reasons why our engineers in the past had to go down to various to low levels of OSI and start tuning so that it's this faster processing of the incoming traffic tickets all right so at Ticketmaster we have around 150 products these are products of different kinds and and there's like 400 plus API or so is groupings and this is pretty much the tech stack I couldn't include everything it's already crowded hopefully people in the back can see all the letters so let me let me start reading one by one actually I'm just kidding we're not going to go through this one by one the goal of this slide is just to show that hey we we have a lot of technology it's a complicated environment and we are primarily an open source tech shop our we encourage our universe I mean our staff to use open source technologies primarily and we believe that a corporate can cannot solve the problem like the broader community can solve and also keep pace with it our resources are limited so obviously we use the open source technologies first we try them out see if it works and then we also encourage our engineers to contribute back in fact that we have engineers on our team who participated in developing Chef and there's an engineer from the Java Spring team and so forth hopefully this gives you an idea about what Ticketmaster is what are some of the technical environment attributes all right now that the stage is set let's let's get into the actual show so the DevOps performance that I wanted to walk you through is actually a journey that Ticketmaster took it's I'm going to describe it as starting at the inception point A and then I'm going to walk you through a few Pivot's B, C and D that kind of drastically changed how we progress along in this DevOps journey which it's a journey of seeking true north B, C and D are called Pivots Pivots is actually a lean startup term we wouldn't get into too much detail on the lean startup principles but I'm happy to take questions Pivot is basically a structured course correction designed to test business hypothesis or a business assumption so that this journey is not drawn to scale it's actually just to show how we progress along in our DevOps maturity model and point A is roughly 10 years back and I'll be bringing you back to where we stand today all right so the journey started in A it was pretty flat and then we went through it had soap opera style journey there's lots of ups and downs and happiness and sadness along the path and then we went to the peak of confusion we thought things worked and then turned out it didn't work for very well and then obviously I'm from Bollywood so I'll definitely end with happy smile and dance all right so so in this presentation you obviously kind of figured out that it's not about the journey or how it started sad and ended happy it's more about what we picked along the way that is going to help you take it to your environment and then do something with it so hopefully I could share something meaningful to you today okay so let's break this journey down into four components four phases so first one is pre-developed and then I have given names for the others as well and we go from red to green and getting bad to good right okay so now the like I said journey the point A started roughly ten years back and as I described we have a chef-like internal configuration management solution we have Colonel Tune we have smart engineers that could really solve any complicated problem well we we started building all these internal tools and kind of started falling in love with what we built rather than the problem we were trying to solve the product that we created happened to be our babies and we started caring for them more so we started stagnating meaning we fell victim to our own success like in many industries even in technology field if you are not rising you are falling right so we started not enlisting and modernizing our technology and therefore we started falling and that explains the direction of the of the error that you see here and then pre DevOps era and there are several attributes this particular to this particular phase I'm sure you most of you know this so let me quickly script through them like any dysfunctional organization that does not build products collaboratively we also had Dev and Ops that was building a tall wall of confusion and between Dev and Ops right and the developers would finish their product they would throw it across over to the Ops with some labeling on it and some Runbook and then the Ops guys would look at it and go that doesn't make any sense let me throw it back to them because the Runbook doesn't have the proper punctuation marks right so they send it back to the Devs and the Devs then look at it and then they keep finger pointing at each other this results in a lot of wasted cycles and lots of rework and therefore eventually you don't end up producing quality products quickly at a reasonable cost so this was one then we had TechDat I mean it's a decades old technology company we have all kinds of versions of technology so we obviously have TechDat like any other debt it either grows or it falls right so if you don't invest actively in cleaning up your TechDat then it's gonna build up and it's gonna exponentially impact your productivity down the line that's what happened to us as well obviously we had long cycle times meaning the time from code commit or even the requirements definition to all the way to a customer seeing that feature in production was was getting longer and longer so if an environment is characterized by these three things then obviously what's gonna happen is we're gonna get our butts kicked by somebody right miraculously for ticket master that's exactly what happened we did get our butts kicked but this time it is by a bunch of nimble agile small startups so these companies started coming out away and then they started eating our market share away from us and ticket master couldn't compete with the with the pace at which the features were getting released by these small companies and and that we were starting to get concerned about that from a business perspective and that's when we reached the point B or the pivot B so this in this deep bit of sorrow we had to make a decision it's a sinker swim moment right so you either change your ways of operating and get software out there quickly that meets customer needs or you kind of lose your continue to lose your market share ticket master has been the leader in the industry so that that's something very tough to swallow obviously okay so we we attempted DevOps and a few examples of our first statement are shown here obviously we all know that multiple silos is a bad thing great so we said let's go and build a cross functional team where the developers and the testers and the operations folks and they just get together and build they have a common goal and all they need to do is quickly get the product out to production as well as keep it running it in production just it just works it's expected to work well that's great and the next one is we we I talked about cycle times one of the deeper problems of cycle times was that we were not taking any feedback from our internal customers our development teams it's not like we did not have any build pipeline or automation that will reduce cycle time it's just that the automation that we put in place did not meet the development or the development or the delivery team needs so they they when they came to us we'd be like okay we already know what you guys want because we've done this so many times in previous companies and whatnot so we know exactly what you know and I'm going to build it obviously they did not adopt the solution and it continued to have poor cycle time okay and some of the folks even quoted Steve Johnson as the customers don't really understand what they want so as I present these things I also want to caution you that these solutions even though it worked for us may not work exactly for you so you might want to take these these findings and and then understand your environment and make it your own so there's got to be some customization required here all right so we made this feedback problem disappear and we started listening to the development and the delivery teams and and we got their perspectives and so this way we were able to drive adoption of the build pipeline automation that we had put in place and we started to reduce cycle time well the results are shown in the developed journey tracker that I have here we we started making good progress in cycle time and and even the customers NPS scores this is what I would call the functional domain meaning we are able to operate in a sustainable manner and we are able to deliver products as planned well almost as planned but kind of functional a lot of companies are stuck in this place this is like it's okay but it's not it's not quite good but then on B dash or the point that I've marked in the middle something started to happen our performance started to degrade we were not that productive anymore and we were confused as to why is that why is that happening so we dug a lot deeper we conducted some root cause analysis and figured that all right things that we thought were good solutions turned out to be bad like how is that like well we talked about cross functional teams right if you know Ticketmaster it's a very siloed organization so we have a massive teams that are broken down into specializations meaning we used to have let me let me list all the teams in one breath see if that's possible so we used to have DC ops team data center team cloud team infrastructure development team a platform team development team QA team business analyst team and so on so forth right so we have all these teams and they are specialized in their areas and they don't know about the technical difficulties of the other domains so now you have to form a cross functional team how do you go about doing that so we picked one from each team and then said okay you guys work together you're the cross functional team and then for ridden then see when some guy gets sick or takes a vacation or whatever we just want to double the number just for high availability right and the team size became massive Jeff Bezos point was like two pizza size team ours was more like 10 pieces size team and it's also extra large pizzas so so this wasn't going to work the team was so big that the team members would not know each other's name it took a couple of weeks to just to get familiar with the people's names the funny thing is the card wall was for so long we used to call it the great wall of cards so anyways it was chaotic it was dysfunctional it started to get dysfunctional so we had to break the teams down into like smaller teams so we experimented with a couple of models we tried Scrum of Scrum's model and we also tried independent parallel teams and we kind of settled for something in the middle wherein we had the teams broken up into infrastructure service and then that will serve the next team which is platform as a service so they add their component and then it goes on to the product delivery teams and between these teams there were unwritten contracts that okay I promise to deliver this feature to you or I'll give you this code base with this functionality so you can build against this interface so we looked at the microservices architecture and the best design patterns out there to enable to learn from some of those things for example we have tolerant reader we have consumer driven contracts and those type of design patterns microservices domain we just adopted the same thing for team designs so the results were course corrected and it started to get a little better then the next one is listening to perspectives like what's wrong with listening to different perspectives right everybody should do that but we started listening so much that we kind of became DevOps psychologists if you will so we started listening too much and if we wanted to get everybody's perspectives we want to solve the problem for everybody and we soon we figured out that a single solution is not going to meet every need people are going to have different needs so we said okay let's launch an initiative to really understand our customers better you cannot have 10 different customers and all of them are exactly equally important to you it's just not realistic a deeper understanding of the customer segments and what their needs are helps you understand and prioritize how important each customer is and then walk walk the list down within each customer again there are different requirements with different priorities things that solve bigger problems for the customer and things that are they just like to have so we had to break it down so that we produce incremental product feature releases that'll really help and and have a better deeper customer understanding helped us with this all right so at this point we reached the point C or the pivot C pivot C is characterized by I would say we are devopsy in the sense that yeah most of the companies try this by reading quick books like devops for dummies or get devops in 72 hours so we bought all those books read all those quick tricks we leveraged from benefits of all the low-hanging fruits out there we've gotten cross-functional teams we got we started using CIs basic things right we saw results but at the same time we were kind of not feeling well at the core we were feeling like how's this going to turn around a 40-year-old company how's this going to make it nimble such that we can deliver products quickly like a startup would right so this concern started to raise we knew that this there's got to be more change required for us to launch into this contributing phase what is that how is that different from functional phase it's a contributing phase is somewhere where you not only operate and deliver the products as per plan or within budget but you also understand and think from customers value perspective how much value you're adding to the customer and therefore you're delivering it so this type of value oriented thinking and needed a much deeper broader change okay so change is required but how much change is the question right well it is new year's time we all make resolutions so let's talk about a quick resolution apparently in a recent survey they say that health and fitness are top of the list that's the category where most of us make our resolutions followed by career growth obviously with this picture up there I'm not going to talk about career growth right so I'm going to talk about health and fitness let's say we all we all know that eating an apple a day is is going to give us great health it's going to keep the doctor or whatever but is it going to get a physical transformation like you expect if that's what you're looking for I was talking to my friend and he said okay if I let go of my other six pack every Friday will I get this six pack not quite not quite you have to change your eating habits your sleeping habits you have to exercise regularly you have to start counting calories and macros and whatnot the entire lifestyle has to be changed it has to be a much deeper and broader change all right so the point here is minor tweaks don't produce major results they really don't if you want something big then you have to go big all right so we want we said that we wanted a more cultural organizational change and that's when we discovered the culture of promises okay what is a culture of promise let's say you have a company vision I'll break it down first and then I'll give you an example let's say there's a company every company has a vision so you break that vision down into strategic promises and each promise then gets further broken down into functional promises and then when it goes to the team it gets broken down into tactical promises let's look at an example let's say our vision is to connect lots of fans to great experiences and if the strategic promise that maps to that is something like okay I promise to make Ticketmaster.com the favorite ticketing site for the fans you could break that down into smaller promises like to provide I promise to provide the best online and mobile experience user experience for our customers which could further be broken down into tactical ones like okay let's I promise to develop interactive site maps basically it's a feature that allows you to view the the auditorium or the stage from each seat so it's a feature and it could be other operational promises like I let I promise to build a site that always works just simply works high up time all right so so now you see that we have taken this one top level vision and I kind of broken down that into multiple hierarchical smaller more actionable promises what this does is there is several benefits to this first one it aligns the entire organization towards with a common goal at the up at the top so if the CEO and the senior leadership team decides to change the direction in the next quarter or next year then it's easier to make change at the top and then it kind of trickles down to the bottom everybody is now thinking about value and customer more importantly than they are thinking about their products so the code or their process everybody is metrics driven in this in this framework and more importantly you would have seen this in studies the best way to improve team morale is to make people feel like they're part of a bigger problem right so this helps the team members feel that way everybody even though the person who's setting up a virtual machine is just responding to the tickets for virtual machines on his queue even he now is able to think of the virtual machine as okay this is going to help me deliver the interactive seat map feature which is going to tie to the best online experience which is going to tie to the makingticketmaster.com the favorite site so the promise is kind of rolled up every task is now more meaningful all right so we tried this and then we got excellent results definitely external results or turnaround times were faster we not only were able to deliver the products as promised but we also understood whether it really helped the customers or not so business business value delivery was happening on a continuous basis then we reached pointy the pointy is not necessarily a pivot it's it's just an accelerator that brought us into the next phase is transformational meaning in this phase we were so tuned to thinking with customers in our center so that and not with our product recorder process as the center of our ecosystem that we are able to quickly pivot if we need to based on company strategic decision on time yet yet deliver to the diverse custom customer needs we were able to own the customer's problem and and always be keen on solving customer's problem rather than the products that we built the term product manager is a misnomer as some of us know if you call somebody a product owner then after a while they start loving their product and what they created right but if you call them a problem owner of the customer and the goal is to solve the problem for the customer then that the focus is different now they are putting the customer in the center of the equation and and they produce whatever products whatever it may be version one two three whatever to solve the problem for the customer so this is a this is a mind ship so because our organization was already set up for this structure by this time because we have been practicing the culture of promises for a while we started seeing some cool transformations that that were organic for example let me give you a few examples so we made a promise to say okay let's free developers from the constraints of on-prem infrastructure well they have and they have a need they submit a ticket they they wait for X number of days depending on the SLA and and and they get a server but at that time they don't have it properly connected no they have to restart the whole thing it it it causes too much delays so we don't want to constrain the developers so that the team now comes up with this idea of okay why bother with this massive infrastructure and how the constraints here why not just go to AWS or web services right so that's exactly what the recommendation that came out of the team it is groundbreaking for taking master because like I mentioned we have a massive infrastructure and we have a big team that runs this and for them to like accept the fact that okay we're going to go to the cloud because it delivers better value to the customer that is significant that that is that is a huge shift the next promise I wanted to find out is reducing cycle time of the software releases so we said we'll believe we'll build a continuous delivery pipeline now as you know continuous delivery pipeline has several stages you check in the code and then it runs the unit test and then integration tests and then spins up an environment using scripted automation and then it runs the user acceptance test and then it takes on to further performance testing or security scanning or whatever and the code ends up in production all in a fully automated fashion but it's the same series of steps that takes your code through all these steps right now we built a framework actually we are in the process of building a building a framework that lets people plug in their own custom build script in there and their own unit test frameworks in there but we still have that high level unified build pipeline or the continuous delivery pipeline for the entire set of 150 products now what does this offer now you have a common scale using which you could drive tech maturity well how's that now you could say let's say all the teams are using the same build pipeline continuous delivery pipeline now team A might have 95 percent coverage in the unit test step well you could measure that because it's the same standard pipeline right but team B has only 45 percent and you could do the measurement and use metrics for the various stages and throw up a dashboard that says okay team A has higher level of maturity in adopting or practicing continuous delivery because their coverage is better they have more automation their environment creation is scripted and the cycle time is highest and so forth so you could use that as a model to motivate the rest of the teams and drive everybody towards a higher level of tech maturity the next one is we promise to eliminate environmental inconsistencies obviously Docker came up so we wanted to contain containerize our applications so besides the obvious benefit okay eliminates environment consistent inconsistencies we have we have this other unique benefit that comes out of this remember we talked about migration migrating our applications to AWS the team not only proposed that they came up with this idea of creating a cloud enablement team that will go ahead and build and on sorry a self-service AWS onboarding kit that includes things like cloud formation templates the VPC layout and the connectivity all established and proper recommendations for technology choices with an AWS product let's say it's a package that's easily consumable for the product teams now each one of the teams can just go through that and get their products up in the cloud so this enables us to move faster to AWS on top of that if we Dockerize the applications on the on-prem infrastructure while it's running in our private cloud then it's much more easier to migrate to the cloud so Docker was again a beautiful transformation that we recently figured out so anyways so this pretty much sums up our journey in this direction and as you can see we have come from all the way being pre-developed in the pre-developed domain to now in a transformational phase where we are willing to see past the technologies that we'll use and look at what's more important at the end of the day which is customers and customer value so that's it that's pretty much what our journey is if it were a Bollywood movie this is the place where they would roll the credits and sing the happy song and dance but I'm going to summarize the key principles for you to take with you the two things mainly first one is automation drives throughput right so you don't want manual operations in your day-to-day operation things that repeat most often obviously you want them automated because you get the best bank for the buck so you drive for more automation as much as possible and then automation if you produce crap then your automation just produces more crap so you need the feedback loop right the feedback improves the quality of your throughput now there are a couple of examples of different examples that I want to give the first one is seeking feedback improves empathy for the dev and the ops team to understand each other's perspectives if dev teams understood how difficult it is to maintain and operate a software after it goes to production if they understood then I had that empathy then they would definitely build applications that are HA easier to operate easier to back up and whatnot easier to monitor right so asking for feedback improves empathy the other one is we should always seek to use operations as a input in to close the loop into the strategy for example product managers when they create a feature and make a business case for building a feature or a sub product or even a big product they make a lot of assumptions they say okay if I build this feature the customers will come to this website and click this button so many times and therefore expect so many dollars to come out of this every quarter so they make they make these assumptions in the beginning of the project and then we go ahead get funding approved and then we build the product product goes to production but very few companies actually take actual measurements that will validate the assumption that were made in the ROI phase and feed them back to the product management product management pretty much runs blind that they they rely on quarterly finance numbers to actually find out much later on whether their assumptions that they made in the first place were right or not so operations teams can help enable that to close the loop okay how many of you know this guy is Alvin Toffler he's one of the famous American writer and he's also a futuristic thinker he's written several things about like revolution in terms of technology and communications he's the guy that said change is not a mere necessity of life he said change is life but the thing that caught my attention was this one he says that the illiterate of the 21st century will not be those who cannot read and write but it's those who cannot learn and learn and relearn what it is is this is an age where the smartest guy or the loudest guy in the room is not going to win it is those guys that can quickly learn and adapt to the changes that they see that's going to win to sum it up I just want to say that when you go back please encourage and try to create a culture of learning where every team kind of feels like it's safe to make some honest new mistakes as they try to push the boundaries of technology because when the environment is not safe or it is not suitable for learning what happens is then engineers start repeating the same thing that they did yesterday because it's safer and you don't get much learning in that process I hope I answered some of your questions thank you and I'd be happy to take more questions go ahead yeah it's a great question let me repeat it for the recording the question is so you showed wide range of technologies in the tech stack and as you do automation are you trying to work towards converging that tech set obviously yes because we do find that it is very wasteful to have like duplicate technologies for example you don't want chef and puppet both running in the environment right we have an internal config management tool like I mentioned and now we are starting to adopt chef so one is going to replace the other obviously we're going to use open source tools so so that's the direction we are going but it takes a lot of commitment and because we have 150 products or so it takes a while to get rid of legacy technologies and replace them with modern ones but that's the direction yeah these cross functional teams usually are self organizing and they usually drive the technological decision making that is we don't practice the chief architect that reviews every single design and approves the design we don't practice that anymore instead we have communities of practices meaning the architects who are interested in a particular topic they get together on a regular basis and they discuss what's the best practice what's the modern ways to do this and they talk peer to peer and make decisions most of the time all the technical evaluation as well as selection is done by the corresponding teams yeah please so so there are several different frameworks that enable one to implement the cultural promises one of the frameworks that we adopted is ThoughtWorks Lean Value Tree Framework you can look it up online it basically goes around by breaking down the vision into strategic promises and exactly the way I showed but they have constructs and proper artifacts in place that helps you and they have a step by step of rolling out such a program into your company we had to train a lot of people on what Lean Value Tree is and we got a lot of pushback to be honest with you not everybody is gonna you're not gonna have the same 100% team past this cultural rollout there's gonna be some resistance there's gonna be some churn you gotta plan for that and be more realistic yeah yeah so so there are two questions that you asked the first one is did you just go across and like instead of one team did you go deeper into a product or did you just say everybody just go ahead and use this that's one the second one is a question actually the second question let me ask you again once I answer the first one it depends on the solution we were trying to solve like for AWS migration as one example we want all the products to migrate to the cloud and we have an aggressive timeline to make that happen so we created this cloud enablement team that creates this onboarding self-service package so how do we know this package is gonna work so we partnered with one of development team whose technology is suitable for to be the first ones and we might we help them migrate we are helping them migrate to the cloud as we are migrating them we're creating this self-service package which is our product now we're gonna go take another product on the other side of the spectrum and test the same self-service toolkit at that point we know that okay two different use cases have worked so I'm gonna now open the gates and say okay product team just go it helps you just self-migrate your own product see you in the cloud so it depends on what actually we're trying to roll out we've tried different things and this is what seemed to work for this occasion do you please repeat your second question um if you remember the the page where I slide where I showed the hierarchy of promises note that I didn't start it off on the bottom I went from the top and that is that is how the model works like you cannot have a bunch of teams trying to drive this cultural change upward it's gonna be extremely painful and won't even go anywhere and we have 80 plus development teams so it's not gonna go much further our change fortunately happened from the top so it was the team of senior leadership executives sitting in the room wondering okay these minor changes minor tweaks that we are doing is not gonna give us that much results or that much cultural change let's do something big let's go all in and that's when we brought in the lean value tree framework so fortunately it's from top down yeah yeah exactly precisely yeah yeah it needed a bunch of these little school of fishes out there trying to eat the shark we needed that kick in the back to like get something good going please yeah so very good question how did you define the KPIs and how do you go about like tracking and making sure it's in the process you're constantly looking at it like I mentioned the lean value tree framework does exactly that in fact we have several artifacts that we identify the goal leads when I say these are different the strategic promise leads the functional promise leads and the tackling promise leads they are supposed to fill a template like this is a framework on how you how you roll out right that template has one of the entries in there is KPIs success factors right so the team leads they have to think through okay how am I going to deliver this value and then they have to come up with ways to measure it every cycle that we review that that particular goal or the project we will review the actual metrics and we have to show the teams are supposed to show the trend or the numerical change and progression it's okay if I mean it's not possible it's possible to expect the graph to be always like a hockey curve graph let everything is improving it's not the goal of looking at the metrics is mainly to find out whether the methodology is working or not it's not really to make sure that it's working so if it doesn't work totally fine you just move on to the next idea and see if that works that's the culture of learning I was talking about currently we don't have any special tools that captures this in fact this is not this is a new framework so there's not many out-of-the-box tools out there that helps you do it we are currently just using Excel at this point which is works sorry oh absolutely yeah so A is like 10 years back right and then B started roughly three years back and then you could think of it as one year one year one year right so the A to B is the one that we had the like I showed the graph like dropping so fast it was not that bad it was more gradual in fact that those are the tough ones to track because every day you're looking at the mirror you didn't lose that much weight but then if you take your picture from two years back you were like 10 it's like what so that would happen you don't realize the gradual changes as obviously as you do the drastic changes all right thanks a lot for coming hello I guess I'll get started here okay well thank you for coming I know some of you are hungry and thirsty and so I won't take up much of your time I'm not a technical expert so if you do have any technical questions I can take them down and get you to the right person or get an answer back with you okay so my name is Douglas DeMaio I work for OpenSUSA I've been there for about a year I'm fairly new to Linux I have a background in military so my wife's German so I don't have much of a sense of humor so bear with me all right so this is a basic overview of what we'll talk about today with OpenQA OpenSUSA if you're unfamiliar with it is a community project that's sponsored by SUSA it has two distributions and it has multiple tools a lot of people will ask you know what is the difference between SUSA and OpenSUSA OpenSUSA is a community version and SUSA is the enterprise version so why automated testing well the answer should be fairly obvious right maybe that's why there's not too many people in here but you know we're talking about innovation we're talking about progress we're talking about getting driving technology forward so this is just a brief summary of the changes that have taken place using OpenQA in the past two years you can see how many changes have taken place in SLEE12 you're talking 600,000 packages added and nearly 2080 removed right same with OpenSUSA 13.2 very similar numbers there and then our latest release OpenSUSA LEAP 42.1 also with very similar numbers some might ask what's the difference between jumping from 13.2 to 42.1 I kind of just want to clarify that because I'm sure some of you might be thinking why that happened so Sousa or OpenSUSA in its latest release used a lot of source code from Sousa Linux Enterprise 12 ServicePak 1 and they used that as sort of the base and then community built packages on top of that so it's it's a little different than CentOS it's more dynamic there's more of an effort from community in that base and Sousa has a long history of starting something new with 42 reference to the meaning of life yeah so YAST was its first version was 0.42 and Sousa's first version technically was that they released was 4.2 so I hope that answers the questions that you might have in your mind all right so yeah change is good right change is good because you know you you fix security issues or you enhance performance you you fix bugs you have a newer hardware that you can accommodate newer features you know upstream upstream is very fast paced and and so you want to be moving just as fast as them change is also bad right so you get new bugs and you get new security issues you get new functionality you know some people understand change some people can adapt others sort of live in the past right okay so automated testing helps open Sousa develop fast and you will see that in the in the slides that come a little bit later with tumbleweed I'll give you some basic numbers and you can see how fast we're actually moving with upstream what are the problems with the other testing tools so how many are you how many of you are actually using some testing tools that are out there right now solarium anyone solarium okay cucumber no jankins I hear a lot of okay jankins heard a lot of people use that so so what are some of the issues yeah well this is just the basic list of it right so some don't render well some are entirely web interface some focus strictly on packages they lack a gooey really there's nothing out there that compares to open qa so open qa to the rescue a little bit of background this started out as a hack week project in 2009 by a guy by the name of Bernhardt vitamin and it's developed over time that was something that they continually focused on on the hack weeks that followed so we eventually got to a point where where we could start to integrate it and when we looked at integrating it you know this is sort of the dev ops issue and discussion the person in here previously was talking about is it's really a cultural change right so you need to look at I guess you're at least with open susan we looked across the board we saw how how our processes worked and we thought well how can we integrate this in and and so that's what we've done is we've we've taken this tool that we've developed and added it to create and enhance our dev ops this is sort of an overview so testing testing testing you'll see the submission come in gets an automatic review goes through a pre-integration tests manual reviews and then it goes into what we call factory and within factory you're looking at multiple staging levels of tests so this is where some of our processes has have been enhanced there's a huge flowchart in my co-worker's office and it if you look at it it just it doesn't make a lot of sense I guess when if you don't really know the whole thing but you have the yes no answers and it all kind of cycles around and you could start it or you could be at the very end and get a no and it'll shift all the way back to the very beginning to run through that process but as you go through some of these testing phases where we use open QA and within factory and the state and certain staging areas when we get to it to where we can get to the final test and everything's worked out and it gets QA approved you get a release or an alpha or a beta whatever you want to say whatever point you're at in your in your development cycle so a little bit of overview is about the about open QA it doesn't really touch the software it really acts like someone is actually doing the testing manually and so this is this is a great benefit you're talking about opening up all different programs you're talking about testing the console a variety of things you're actually you can you can create a test with open QA to basically do what you want so it's it's an installation yeah installation testing it's from beginning to end but really when it hits a failure doesn't need to go forward right so it just stops right there however you could potentially if you wanted to in the scripting ask it to continue if you'd like so it gives you a graphical interface you can see on the left there you get the green means it's passed and it works on to the next level right um one thing I would point out with this so in in your development process you know when when when we have something like our rolling release tumbleweed you get a lot of version changes right so so you might wonder how do you account for that in your testing scenarios well basically you have these green boxes that you can focus on certain areas that you say hey I need this to be a hundred percent that this this area cannot change however you have version numbers that might change you can give a a percentage of acceptance of change so you can account for those little things that would happen and obviously if you're doing routine cycles right you know those little things would matter and so that was thought of so all right so you can see the script running too that's another good thing below this if you actually go to the open qa website you can track a live test that's taking place it's in you'll see a little yellow button and if you click on that you can actually see it see it moving and and see it the text the test taking place and see the script below it and how it's actually being tested yeah move on so this screen as I said I'm not a technician so to some of you you might understand what this is but this is supposed to be from what I understand an output of the previous screen so I'll give you a chance to look at that move forward so this is sort of the architecture the way it's set up we have various workers goes into a pool and it gets I guess it's call up and and sort of waits in line because you'll have multiple tests taking place and they're happening every day I mean there's there's tests every day going on over the over the day over the night and they all kind of wait in a queue and then eventually when it's their time they they go through testing and you can you can adjust that too to if you need something that you need to rush through you can I guess subvert the other ones and and go to the top of the list if that is what you need to do so it gives you a quick emulator it it supports so with ours it supports multiple architectures which is is very helpful for a lot of people right because because they're working on on several different architectures or we're working with so many people so we we have to make sure that our stuff is running on a variety of variety of hardware and it does run on bare metal so most of it is done tested most of it is tested in a VM but you can do it on on bare metal this is sort of the dashboard that you have from it where you can see all your tests and it basically this is in the development cycle of of a distribution you could you could see that I guess years past right you get six month development cycle fairly fairly quick at that time I guess but you might have introduced something into it as you're building as you're building your distribution and you're toward the end of of it and and then you introduce something else in there and then you find some bug or some issue and and and in the past it was kind of difficult because you go where did this happen you know it might have happened like four months ago it if you got a track and go back to all that well the benefit is you know you get to keep all this information in your servers you have all that data you can go back you can figure it out fairly quickly if you need to but a lot of the testing um you'll see it right away you'll see the issues right away so there's a failure that you can see right there on the left you'll get that you'll get that red check mark and basically it stops from there so who is who's using open qa well obviously open SUSE is using it we use it for tumbleweed and how many people here are familiar with tumbleweed how many people here are familiar with raw hide okay so tumbleweed is what some of the people that I work with believe is actually the future of Linux distributions and that is that the latest software is being tested and coming out very quickly so it's it's the latest packages the latest applications it's it's just that's what tumbleweed is you you have the latest and greatest and it's stable and it's tested and that's that's the most important thing that I'd like you to take away from from that idea of tumbleweed it's tested a lot of people will compare it to arch and you know arch is good you know but you have to compile arch right and there's a lot of stuff you have to do with tumbleweed you don't tumbleweed you get snapshots and it's tested and it works we used it in a leap and that was that was a quite a a difficult thing right to bring two source codes together but we used it for validation testing system testing SUSE is using it they're using it for SUSE Linux Enterprise 12 they used it for 11 I believe 11.4 was the first one they actually started using it a little bit and red hats using it the person who wrote this who's our QA engineer Richard Brown and the open SUSE chairman one day he was a little bored and so he thought well let me let me see how how easy this is to make a test and so he wrote a test for Fedora and in doing that test he or in writing that test and creating it he ran it with Fedora and he didn't do it to kind of yeah like I I guess what what's the word just not be confrontational he didn't do it with any intention like that he wanted to take a distribution he knew but not not so well and just write something for him and he did and in that test it found some bugs that Fedora had and of course he knew the the release manager for Fedora and he said hey I just wrote this for you I don't know if you want to take want to take a look at it and the release manager was very yeah thankful and he said yeah okay well so they started talking about it and they started collaborating and now you have Red Hat using it and Red Hat is using it and they're contributing back to it which is the most important right because if OpenSUSA has some has some issues and Fedora has issues you know you have certain bugs well in a lot of ways right there they were late so we're kind of working together and we're we're helping each other and that's I think that's a positive good news story for for the community and for Lennox and you can see right there in the bottom there that's the release manager and I did that I took this screenshot just a couple hours ago but they so they used it five days ago and they're using it with Rawhide you know that you might see that progress develop a little bit further based on based on them how they integrate it into their processes I don't think they're there yet but they I'm sure they will get there soon so OpenQA and Leap I've briefly explained a little bit a little bit about this earlier you know you're taking source code from SUSA Lennox Enterprise and then what we had in OpenSUSA is sort of the development going forward and really bringing those two together goes through it went through extensive integrated integration challenges right and you're looking at over 25 different installation upgrade scenarios and OpenQA allowed us to do that you know you have an OpenQA engineer he's twice as productive right he's running his tests he's doing his manual testing this OpenQA is just a really really beneficial for everyone and as I mentioned earlier we have it in our processes it is an integral part to OpenSUSA development and really it's allowed us to to be very fast in our releases so the very bottom there testing testing testing new tumbleweed releases as I said a rolling release they're expected about one every two or three days when we have the when we have the building of of our regular release it kind of it kind of slows down right because because areas are focused in other other places but it still does move forward during that time it's just we have to dedicate more resources to to the development of our regular release so here's an example of a quiet week for tumbleweed this is something I guess has happened back in April or something like that so they had three snapshots 146 package updates 15 new packages packages on the DVD and we're also removing that some of those packages right so we're cleaning up after ourselves and one new kernel all tested and 118 different installation and upgrades this is this next one is a little more interesting right so and this is what we're talking about with development you have five snapshots 298 package updates 47 new packages on the DVD 42 packages removed two new kernels and it was sort of just another week as someone described it when they when they did this and this was again this was back in spring but we did the same thing last week so we had five snapshots within tumbleweed last week yes I'm not exactly sure on that number I mean I could explain the top ones but that one I'm not exactly sure do you know George do you know anything but so yeah as I mentioned earlier the first one the first time they started using OpenQA SLEE did or sorry SUSEA did was in SLEE 11 SP4 and they integrated that into their later processes with SLEE 12 which was released last year in SLEE SP1 which was just recently released so you how you incorporated into as you do it with some pre pre-validation and staging so and every part I guess of the development you're kind of looking at OpenQA and you're using it to your advantage so it's inside our processes is inside SUSEA's processes and they don't with SUSEA they they don't really show their their stuff you know it's sort of hidden in the back but with OpenSUSEA you know we do make those tests available for everyone so over 20 over 100 different validation scenario test I think I would think that would kind of be along the same lines as your question earlier you're using it for we're using it for alphas, betas different different scenarios matter of fact when we released when we released LEAP I think we just went straight to a a beta I think that's what we did and you can use it for post-validation as well so go back OpenQA and you so if you're if you're thinking about using OpenQA I basically there basically here's some some information for you on how you can either contribute or learn about it or contact someone that can help so uh the documentation is available the bug reporting is available at that website and it's located the main project is located on GitHub as I mentioned earlier you know OpenSUSA is transparent they're making all their test test publicly available for everyone so everyone can see it and the best way to start and think about how you would describe the steps for someone someone else to do it so if you're writing a test you want to think about how am I going to get how how am I going to use this right and so you want to write those steps down assume that that person doesn't know how to do anything and then use those steps to help create your test code write the test code for each step and you also want to think about what you can do with OpenQA to enhance or develop your processes to aid with the DevOps transformation if that's something that you're here and thinking about with this presentation this is some basic background sort of I guess advisement whatnot yeah organization I it's a British guy that did this so there's no there's no z in it yeah so I just left it the way it was I don't know if that was if that's necessary or not and here's some other coding actions for you I guess here's a example test console it's for the console this next one's for the graphic and so as we're looking at at that we see how we're open using OpenQA with their distributions but it has a lot more potential than that right so here are the some of the things that we are actually looking at doing the thoughts the thoughts are already there but the they're actually working toward this right and my understanding is that possibly they might be looking at salt stack too so and that was pretty much all I had for this presentation if you want to find out more you can go to that website and you can contact either Richard Brown at susah.com or hop on the IRC OpenSUSA factory and all those guys will will be able to answer any question you have pretty much right away they're they're on there all the time and and they'll get back with you on on any sort of help you would need Richard Richard's very helpful he can he can answer any questions you have he does it as his day job so what's that he's the guy that made the slides and yeah he's the British guy the slides no I don't think they are I did send them to scale but I did make just a couple of changes not not much I mean the content is for what he did was was there so so does anyone have any questions about OpenQA is it being used to test anything other than the installations I believe it is I think that was actually in the slide let me see if I can find that I I I did see that somewhere you can use it to test you can use it to test network cards you can use it for basically a lot of different things open cv yeah let's start bring that up to read the actual screen and output and compare the predefined needles that isn't the question okay okay I couldn't answer that question for you I wouldn't I wouldn't be able to well I mean I don't see why you couldn't make a test for that I mean if it's I can't tell you if we do that but but judging from what I can see in the documentation and and the way that they've discussed it it's you can test for pretty much anything you want it's just a matter of actually writing it right and that that's brings up a valid point if they hadn't thought about that but I'm sure they probably have but who knows you know I I can pull up I can pull up IRC and and ask if they would if they would know that I mean probably everyone would on that oh that's that's actually a test sorry yeah that's that just no that that was something I had running in the room before before people showed up so they could actually zone out or look at it it wasn't there was nothing no intention behind that other than just give them something to look at rather it is yeah did you want to see it yeah okay let's see I don't know if it would be in doc again I'm not a technical expert I'll go to the documentation and see if there's anything that it's would be all the information needed to install and set up tools here I start a guy general operate that's probably in that starter guide I would think I mean I really yeah I I couldn't really tell you I know we have our security team so that I know they look at stuff as as far as like incorporating security tests in it I don't I don't I don't really see anything but it's an interesting thing to bring up and I I can ask Richard and see I give you my card you can reach out to me and I mean those that's a good idea right why not I've heard of it uh-huh yeah yeah that's I'm sure that that would I'm sure bringing it up would probably make someone do so yeah any other questions all right thank you