services are interacting with each other with various set of queues in between message brokers and other stuff I do not want to get into the architecture building of those architecture so just assume that you have 30 different services interacting with each other and let us take an example if there is interaction then let us take an example that your user is trying to access a billing he is accessing his billing or a cart or something from your application the e-commerce site which you have built so when he is trying to access there is a latency there is a performance issue for that user when there is a latency issue what happens to your application the user will leave your application he will not try to use it anymore say things like it is pretty slow let us move on so that is what the user mentality is so there is a latency issue there is a latency in your application now what are the solutions how do you solve this problem today so first option what you will do is you will go and figure out logs you take the logs and you check the logs but there are services more than 30 services there is a lot of logs and looking into the logs is obviously painful when you are looking into many logs between different services it is pretty difficult to correlate sometimes it does not make sense and most of the critical parts are not logged at all I do not blame developers but they do not usually log most of the things what about metrics you have metrics but metrics will give you something happened at the time there is some issue happened at the time and there are 20 different services in between and there is obviously there is an issue with that metrics again so you will not be able to find the right set of information you cannot debug the request which you are getting this has been the problem when you are thinking large scale and each request is involved and developer is not able to debug so the more the number of services the complexity is more so as service A talk to service B and service B talk to service D and it also creates another child called service E and there is again X, Y, Z it is pretty complicated when all these services are creating logs all these services are creating interacting with each other and there is absolutely no way to figure out what happened to your request where did it all touch so so microservices and new problems so these problems are already there but microservices will complicate the problem so what is the problem root cause analysis so let's take one more example you have a merchant who goes and updates his the product information and product pricing once he updates that what happens to that it actually publishes that particular information to the catalog the catalog again publishes to elastic so that user can search based on the price or whatever the product etc how do you figure out there is a touch point between different services service A, service B, service C, service D and how do you follow the path of that particular request a single request I am talking about and again transaction monitoring so microservices have different databases and there is transaction on these databases there is a commit which is happening so this the commit has to happen from the backwards because the last commit has to tell okay this commit is fine and then it goes to the previous one and then say I got the last information and I need to commit this one so there is a transaction monitoring of a particular request is not available in logs and service dependency the dependency between different services is not available so I showed you the first slide where there was dependency graph which I was showing you it is pretty difficult to figure out what are the dependencies between the microservices and of course to improve the performance of your application to improve the optimize the latency you need to know what is happening in a particular request this is the most important thing of microservices in what context a request is talking to different microservices the context is the most important thing so if I am accessing buying or selling and there is a transaction which is happening and this context is to transact selling this context is to transact buying so this is the information is not available in logs unless and until the developer is actually coding and also the different correlation there is no correlation between the service touch point so to solve this problem what happened was people came up with so log analysis we still do we do not do away with it and we do not do away with monitoring as well but phrasing is something which is they brought in to manage a particular request to see what happens in a particular request so all these people started designing their own phrasing and zipkin became more popular but it had problems but all these things were based on three core steps they were tracking code and they were storing the data so basically they were storing what was happening in the case and they were storing the data so basically they were storing what was happening in the code what your code was doing when the request comes in and they all they all had what to query what not to query and the display for a UI to query the request but that did not solve the problem the microservice is again it's not just you it's not just you're talking to your own services right you're talking to services which is external so a book my show talks to pvr so so everyone started designing it but it did not solve it and also you can see logging, matrix and phrasing had overlapping parts because even tracing was doing logging even matrix was logging something and logging anyways logs but you had different systems there was no interoperability then there is obviously when there were so many vendors involved they were actually building different services and one service was on zipkin one service was on something else they were not able to share data right they cannot share data between them so there is obviously there is no logging standard when they are not able to share data so what happened so k and cf got together they brought in something called as open tracing and this is nothing but standardizing the apis so basically everybody can talk to each other if they implement this apis okay one of the key information open tracing brings in is a request from where it is coming for a particular request where it is going and some references and type of request that is nothing but the context right the type which is required I just want to bring in some kind of context related stuff so you have your service you have api you have design and you have implemented open tracing and you have shared libraries so you are sharing data in a particular context with not your service basically if it is not your service within your organization as well within organization as well there will be lot of different services which want to share data so using the open tracing standards you can actually share the data and also define context so we will see in the demo context the things which we can do so what is AGL AGL is a monitoring tool which is built on the api standards okay what are the three things it does is it will actually give a information for you guys on what is your code doing and patterns basically it is just to figure out at what context the interaction is happening at where there is a bottleneck and where there is your MySQL query is held up where there is lock issues are there so basically it gives you a complete pattern and you know there is one component to another component so basically you can actually trace the complete interaction between of a particular request from service A to service then let's take an example here okay so this is a bad picture okay this is a e-commerce site and you browse or you do some stuff it actually goes to service A service B, service D and then there is a comment happening in database okay when it comes to service A it is actually storing the information of the transaction and service B, service C all information is stored and this is where the AGL comes into picture it has a UI and a database you can actually query and you can actually do the integration of the transaction which is happening so some of the concepts a trace is a collection of spans this is the concept of open standards open tracing and so a trace is a collection of span what is a span so basically you have a request that request will create a span for a particular microservice A and then that is one span and then there is a service B there is another span then there is a service B service C service F then there is a span of a thing you know span gets created for a particular request and some of the properties a particular span has is operation name and start and end time and tags tags is nothing but searchable key value pair logs and baggages and references baggages is a metadata type and all those stuff and references is tell off and follows from and etc. so let's take an example you have a request to check your bill and user comes and he says I want to access all the bills and when that happens there is a span created so these are the spans and this is the authorization microservice so once the authorization provides information to billing how does it provide information it provides information via tags so it also logs of course so it provides information to billing and billing calls various various other spans and there is lot of there is lot of it can call a number of spans and then script could run and information is provided to the user but notice that it's over a period of time and everything is collected in the time series database and it is available via Prometheus and you can use it any time series database so let me do a live demo I hope it works so you can see there is a command which I am running which is creating the it is like all in one but I told you that I am not going to talk about the architecture it has a lot of architecture like elastic or you can use kazandra for storing this data and all the stuff so you should check out the website for that let me do this also notice the command so basically I just want to give you a kind of understanding there there is a UDP UDP port so basically what EGGER does is it just sends the information the code which is packaged the EGGER client it just just sends the information but will not wait for a response from the EGGER client this is not right so basically this is how it has been built is that you know you have a web client where Redis is there it is giving you a session ID and there are few set of kind of microservices and back end I have been to do this demo there is some kind of a simulation and all those stuff so this is something like we are all in root plan and we have to get back home we need to book a kind of cab so let us do this so when you click on this ok I told you pretty brave cool so this is now post to the EGGER client ok so EGGER client is this I hope you guys can see this ok some some information on EGGER client this is the graph which we were actually saying when I showed you the first slide of Uber since this is pretty we have very less microservices it is small we will see directed graph I have a front end it is making so many calls to Redis and it is making some calls to customers some failed you know we did not get a car at all some failed as well so we go back to this so here you all the services which is available you can check it out and so there were 9 trays created so basically I clicked 9 times and you know only one got through rest of them failed so if I click this this is how your trays look like front end call it call to customer again customer and there was some issue in my SQL and if you go there you know for acquiring lock it was waiting for acquiring lock right so and there is some tags which you can create it also shows you what you know a skill query it is running and you can see lot of stuff this is where the you know it is also showing what is the execution time and all those stuff let me go back you can see there are failures are here right if you see this this part you can see the failures here and there are some request which went through which now it went through few of the services but it failed to provide required information so only one got through which is here right so let me show you how it will look so this is basically the trace if you see it created lot of trace one after the other so you can actually figure out from here that there is redis something happened in redis and there was a session time out and all this thing for the particular driver so it could not provide the driver at all you can actually figure out for a particular request so that is the magic of tracing you can go to the level of single request so with one request you can actually trace a touch point of so many micro services which I showed you in the first slide right so coming back to this first one this is the first one right so this first one will actually show you the complete history so you can see the complete history here what is actually happening this is an algorithm which is running in the back end it is actually calling the route to figure out the route it is actually seeing which driver is available and all those stuff so there is a simple you know I will share you the repository as well so you guys can go back and you know check it out so there was only one car which was provided let me go back to search and you can use here driver this is how you can query for a particular request and you are actually if you see you can actually write a lot of log search code here and all the stuff and you can also see an R-back and all the stuff let me stop here and let me go back and give you guys some references so that you guys can stay around with it and learn a lot of stuff repo architecture framework and concepts and terminology is available here so you guys can go back and learn that's me we can discuss I think we have a question okay any questions hi Vivek so if I have an existing application consisting of microservices what would it take for me to add this distributed tracing tool into it and what is the minimum thing that I need to add so I can enable this yep I did not touch upon that because it was more from the developer oriented so I wanted to keep it more ops but if you want to do that so it's a standard so basically you have to index first you will have to initialize your code for these libraries to be used and then you will have to use that standard and create those but as it says microservice is complicated when you try to implement that will take some time for you to actually implement this because it is from the developer perspective right developer has to sit and code refactor his code when he is trying to put the distributed system with this phrasing which is already existing so if you are building from ground up or if you are building something new then this is pretty easy to implement if you go you can check out the reports and other stuff the code is pretty simple you can actually initialize and just use those libraries and start using it again if you are using microservice you will have a small team as well so small team of developer and if it is pretty small application if you are made sure that the microservice is it is not too heavy then you can easily implement it because it is not going to take much of the code as well if it is very small can you just hold on to your questions so we have a Q&A session where we will be there at 3.20 so we have a Q&A on dealing with legacy systems so we will be able to answer your questions then so in interest of time we will move ahead for a lunch break just few announcements the merchandise counter is at the venue as you enter the venue on the left if you have already paid for your t-shirts make sure you collect them if you want to buy them you are welcome to do so we have given you feedback forms there are a lot of talks that happened this morning please do rate the talk and give your feedback and solutions at 3.30 we have flash talks from the audience so if you want to give a talk anything open source any tips and tricks with DevOps we have Mehul here he will be here after lunch you can just give your name your mobile number and the talk so we will see you back at 2.15 thank you welcome back after the lunch break I hope you had all your delicious food before we start the talk announcement that on the first floor we have the BOF and office hours going on so if you are interested in attending that you can move to the first floor coming back to the talk now we have Kashif Radhakir he is an eternal father figure to his child and also to the engineering teams that he manages when he is not in front of the computers he digs into the technology landscape and the history of the Delhi city he loves his kebabs so don't pick a biff with him let's have Kashif talk about his topic now hi thanks so I think referencing Pukraj I have the post brand deal talk I think everybody is asleep which is fine you can continue to sleep I just wanted to understand the audience that I am talking with how many of you have heard of immutable infrastructure that's about a quarter of the population good that will make it easier cool I will begin right away so you know I work at while this talk was originally made I worked at a company called Kayako since then the company has gotten sold and I am moving on but this is an experience from Kayako that we will talk about so just quickly to kind of remind ourselves of the context and the problems that immutable infrastructure solves here is a slide that talks about what is our typical infrastructure so we think of our servers as pets we give them names you can call them Naruto or Bankai or whatever it is and you know those servers personally you kind of build them up yourself you know on this server I have done something special everything all the changes are done in place on the server itself automation is completely optional and testing is very very important because each server is different you know your infrastructure is susceptible to drift because it's likely that somebody made a change on one server but forgot to or wants to make that change on another server later it's difficult to kind of revert state you know in our setups typically and it's slow to recover from disaster so those that I think are the sort of problems that we face today when we do infrastructure so I know a lot of you will probably relate with this I think this GIF captures our lives so well and I hope it will pay off a bunch of you may have already seen this this life of a DevOps guy that's when you think you've got this shit working there comes another challenge I think that while it's very exciting to kind of see on a GIF it's not at all exciting at 2 in the night when you end up you know solving these problems so quickly what is immutable infrastructure right so immutable infrastructure is us treating our servers as cattle and not treating them as pets so you know farmers who have cattle they don't name each and every one of them because they don't want to develop a relationship with the cattle because one day they'll be responsible for killing it for its meat or other resources so the decisions are not hard makes the decision much easier and that's the paradigm shift that happens with immutable infrastructure so you know every time you want to make a change you recreate the whole infrastructure on change automation becomes compulsory you can't do anything without automation and testing becomes optional because really what you're doing is not different from what was originally tested there are a bunch of advantages that come with this one you can't drift there is no changing of the servers manually so there is no question of a drift arising in just one server you have fast recovery from disaster because it's completely automated it's easy to revert, it's easy to scale you know your staging environment can be made to look exactly like a production environment maybe with lesser resources and it's obviously self-documenting because it uses infrastructure as code or uses code to write out infrastructure so that's just setting the context for this the key idea here is that you don't repair things if a server is screwed up and how it's provisioned or its hardware has a problem or whatever you replace the server you don't try to repair it so very much like lego nobody's I think ever tried to repair a lego they just throw that lego piece out and they use a new lego piece if one doesn't work so what I'm trying to do here is talk about give you a case study to kind of relate this with broadly so that this is not just theoretical that case study is from the company and we lived the mutable infrastructure life long enough and then we decided to make this transition and we did successfully make that transition so this is just me sharing how you could do something similar if you choose to so what is kyako? kyako is a help desk it's very much like fresh desk or zen desk you know at kyako it used to run on soft layer which you know is not the cloud and every time we needed a server we needed to raise a ticket so it was usually there was a two or three day cycle time and if the rack that we wanted the server on you know there was nothing available there then it's likely that it would take even longer so hardware provisioning was done via tickets you know we use chef I don't know why we use chef but we used chef for automation and obviously we had SSH access you know to make changes because nothing was ever shipped out perfectly there were parts of things that had to be done manually because they were not automated yet and we had a bunch of you know running business so we had different types of servers that we provisioned workloads had become specialized so you had different types of servers, you had different components that you had provisioned for the whole thing was obviously fragile like I think most mutable infrastructure is but we've just become used to it and we don't think of it as particularly fragile but it was the example that I'm trying to present to you is not a small toy example it's real business, real servers behind that is actually the image of servers as they were in a mutable infrastructure that's an actual image built using cloud craft by sourcing actual servers from putting in actual server data essentially so it was a large scale operation so the key question we had when we set out to do this was how do you do this with immutability how do you build infrastructure with immutability so these are the key questions that you know our team had which was how do we provision a machine you know how do we connect various parts how do we do releases in this world how can we keep our costs in control how will we find out you know where are all the services gone because the services will be spawned at different places then we would have spawned them manually and then you know who's going to have access these are like a bunch of key questions there are lots of questions I think these will set the tone for understanding the space well so you know in order to pull this off we had to change our service provider from software to Amazon because immutable infrastructure requires you to do everything programmatically you can't do things by hand and so hardware provisioning needs an API and you need to get some kind of response to that request and so we move to AWS which obviously everyone knows has APIs for all kinds of things including hardware provisioning then AWS helps you know to a degree but there are some parts which it doesn't automate completely you know if you want to do immutability and so we choose the HashiCop stable HashiCop for those who don't know is a company which is helping with distributed computing and building out a lot of tools for it and I'll talk about them in a little bit so these are the two key technology providers that we use to kind of implement immutability and they might also be relevant to you if you choose to do this so you should look them up so I'll spend two minutes on what HashiCop is so that everybody kind of gets what I'm talking about later so HashiCop has this you know has taken the infrastructure or the running the operational process and broken it down into five parts and they have a tool for each of these parts and that's building so that's building of your images, the software that you want to deploy on your machine it's provisioning hardware and software that you might require it's making sure that everything is secure then there's the process of actually running those servers and making sure that they are able to talk to other servers to accomplish tasks and cohesion and so there are five different tools that they use for this so packer for building terraform for provisioning hardware largely terraform is very much like cloud formation if you're familiar with that or do much better and less painful then there's vault for managing all kinds of authorization and access even offers encryption as a service then there's the actual orchestration or running of machines and workloads that's Nomad so Kubernetes also does this for example and console is the service discovery or the service registry which you'll use for service discovery so this just to give you a brief context purpose of the talk is not to teach you Hashikov you can learn it with a lot of documentation out there other more proficient people talking about it but it's important that you know that these are the various components that we'll end up using because I'll end up referencing them in the talk later so again just to get back these are the questions that we want to answer and we'll pick each question up and say how do you do this in an immutable infrastructure world this is what provisioning a machine like is in a mutable infrastructure starts out fine never ends well because you feel like this time I'm going to get it right then you realize that somebody has changed the Java or PHP version in the repo and that happens so how do you do it with immutable infrastructure one obviously I mentioned we had to move to AWS and we used Terraform to go ahead and describe our complete infrastructure and then Terraform would use AWS APIs to provision it on the fly so we would build this out with Terraform somewhat like specifying JSON declaratively and then there's a Terraform command line kit that you would use to kind of deploy ok software was provisioned with Packers so we would make every release would kind of eventually end up in a software image which would get provisioned as an AMI on our EC2 instance and obviously the key point to all of this is because you want complete automation you want to make it infrastructure as code you can't have any manual parts so everything has to be codified and Terraform plays a pivotal role in that process in this table of tools ok so here's how do we connect various parts so here's an example of infrastructure as code in Terraform so that example on the right is basically helping you provision a load balancer and when you run this it'll provision an EC2 instance a T1 micro with a certain AMI that you specified and set up that configuration on it so that's like setting up a load balancer without doing anything by hand and once this is done you know everybody in your team can just run this whenever they need to provision a load balancer they don't have to keep doing that activity through the GUI console or by hand ok so Terraform describes infrastructure as code integrates very well with its other tools whether it's Packer or anything else in the Hashtag Cops table and AMIs it actually even works with Packer images it works across cloud providers so you're not bound to use only AWS you could use AWS or Google or AWS and Google and Terraform would still work for you it's modular and that modular aspect is I think extremely powerful and we'll spend a slide on it you can preview changes before you apply so it's not a case of you fired the command and you don't know how the outcome is going to be so you can see what changes Terraform is going to make before you actually execute them and obviously very importantly it's collaborative so typically in the cycle of absorbing a new DevOps guy there's a lot he doesn't know in context if you give him a task he has no way to collaborate with somebody who does know because there is no common workspace that they're working on lots of things are being done by hand because this by hand is eliminated he's going to commit to the Terraform repository and it can go through a pull request and somebody experienced can actually see that he's doing the right things before you push that code out otherwise you'll find that out by trying to cross the waterfall and falling down this is what is I think the fundamental change that is going to happen when you use individual infrastructure is that you'll be actually using Lego bricks and see Lego bricks are awesome because you replace and not repair and you can use them to craft any kind of architecture this visual is just to help you understand that Lego can be used in any way in case you haven't used Lego in many ways but that's the advantage that you have with Terraform as well and this whole stable actually they'll you can use them to build any kind of architecture there's no kind of architecture which is not open to change where Terraform an example of that modularization is this so if you go on their site they have a registry where people can submit modules that they have made and then there's a quality process also around that submission a review process but nonetheless here's a module in Terraform for Kubernetes if you use this if you see the section on the right this section here this is all you need to do to add it to your Terraform code base and get it to provision of Kubernetes infrastructure if you told someone to do it especially if you've never used Kubernetes before even if you've used Kubernetes before there's a very big chance that the provisioning will take hours and you will get it wrong and here they've solved the problem by bringing it down to 5 variables can you see that 5 required variables that's all you do insert the snippet of code and that is all it is and at the end of this when you run this 10 minutes later, 15 minutes later on your infrastructure there will be a Kubernetes cluster up and running and that's extremely powerful and the thing is everyone in this audience does that it will show up in exactly the same way that standardization is extremely powerful as well and so there are modules for everything they've truly been able to deliver the lego-like promise so they're modules for AWS they're modules for Google Platform they're modules for Azure for couple of modules for Oracle the Alibaba Cloud and even GitHub and so now you can pick these modules you don't have to reinvent the wheel each time around this was so possible earlier only for software engineers extremely difficult to do for DevOps or SRE folks and some of this was solved by tools like Chef and Puppet and all but they didn't really get it lego right and these guys kind of allow you to do that this is how one typically releases code with a prayer to the almighty who seems to be looking away anyway nowadays yeah this is just if I just wanted to put in because I thought this explains blue-green deployment better than anything on the planet with a little bit of sense of humor we'll talk about why blue-green deployment is important but that's blue one is blue and the other is green that's an existing infrastructure and he's replacing it with a new infrastructure and that's what we're going to do to release code so we're talking about releasing code and how we release code is with blue-green deployment so what happens is how that process works so I'll quickly tell you what is blue-green deployment there might be a few people who haven't heard about it so blue-green deployment is this idea that you have an existing infrastructure we'll nickname it blue we'll code name it blue and whenever you want to make a change any kind of change you don't touch that you make a new version of that in both hardware and software call it green and then you just switch the traffic from blue to green that's a blue-green deployment and these are just the steps that we did it we used to kind of work with the tools that we used so what would happen is every time somebody did commit a bunch of good commits would kind of go out as a release so whenever a release was built GitHub would hit or pack a setup and that would convert and build out an AMI that AMI had everything that software needed to run for example we had an app server so this was a PHP PHP based organization and so you had every time there was a release we needed PHP to be installed as well so that AMI had PHP it had all the libraries that we needed any other components and it had the latest piece of code so when you did the release say 15, 20, 30 minutes later and AMI would become available in the AWS repository then Terraform would then pick up pick up that AMI essentially and build the green cluster out the blue already exists and it would build the green cluster out with new code and then two other components would come into play console and Nomad and they'd basically be used to switch traffic from the blue cluster to the green cluster and it doesn't need to be 100% switch you can choose to switch some weight that's fine that's supported you can also choose to just do a canary the rest of your system needs to support that your software needs to support that but you can do that as well it's supported here so console and Nomad which I mentioned a part part of that toolkit console is used for service discovery and Nomad is used for orchestration of running resources they would together manage that then you know once you felt it was fine you would tell Terraform to an event or however else you would like there are many ways to do this take down the green infrastructure sorry the blue infrastructure so now your green becomes your new blue and your new releases out also you know rollback is quite easy because if you decide while you're releasing this that there's something wrong with green you can just switch the traffic back to blue and go back and running so that's basically the process we followed for release the next big question that we had is how do we keep costs in control because today now anybody can just file stuff automatically and it might he or she may not realize what they're creating so there's no easy answer to that if you automatically choose to create a hundred servers they'll get created but there's another aspect of costs which is that how do you know you're using those resources well and that Nomad helps you accomplish so Nomad will help you optimize workload for utilization and efficiency so you can say you know only spawn new boxes when this CPU is over x percent otherwise try to use the existing resources don't spawn new boxes easily possible to create infrastructure only on demand that's another way to kind of manage costs so what happens is a lot of businesses you know have very cyclical volume so in the day they'll receive a lot of volume and in the night that volume will dip significantly but we don't change the number of EC2 instances we have running in response to this we're always paying a large fat amount for the day you can actually bring down 30-40% of your cost maybe more if you just start following the cycle so in the evening after a certain time you can just set up an event which will trigger new deployment with lesser servers or with different size servers and you'll save cost and obviously you know Docker's helped improve utilization and reduce cost and this works with Docker so you can use that as well so in the earlier world we had you know when we provisioned a service we knew what IP addresses it was available at and we would then tell the developers that somehow usually through Confluence or something else and then they would kind of read and write there to that IP now what's happened is that service discovery has become essential for us in a mutable world because the IPs will change it won't be the same IP every time you deploy something you would you know it'll show up somewhere else so console from the Hashacop toolkit is the service discovery or service registry essentially and it supports both DNS and HTTP so you can you know have DNS records and ensure binding to IPs you could bind to DNS records which you could do in the mutable world as well it has the idea of service health so you are able to check on whether a service is healthy or not before you switch traffic to it so whenever you deploy so you can write custom tests if those tests turn green follow them turn green only then will terraform think that oh this service is healthy and I should move traffic to it it also provides key value store which is primary use case of which is you know configuration runtime configuration and you can also use it for orchestration because you can put watchers on it and locks on those keys and use that to kind of coordinate all your moving parts but yeah so you can even store configuration we moved our configuration from disk we had a per tenant so you know Kayaku is used by businesses each tenant has their own configuration to their own API limits the number of users they are allowed etc and so we moved that from disk to console and console is based on raft which is a distribution technology essentially that will allow you to make this you know not central and fully distributed our configuration was available all the time even if a partition in the network occurred both the partitions would have a copy of the configuration and be able to serve whatever traffic they are receiving if they are receiving any and you know it will allow you to do some advanced networking as well not critical the next question obviously we had is what about access right so the pool chain access right people to deploy stuff we RSRE team obviously will have access to that and they will have their own pool request based mechanism on which they will be able to release services which will cause service to change but the big thing that we were able to achieve is that we live with zero SSH access and what that means is that nobody can get in and run an accidental command from history which will screw things up nobody can leave a screen running or not run something in the background all of those errors disappeared completely because just nobody has access and each time a new machine is deployed you know if you wanted to maintain SSH access you would have to make sure you gave permissions there as well but that all that need is gone and then obviously we used Vault to kind of keep all our secrets and deliver them to the right applications at the right time and Vault is another tool by Hashikov that helps you do exactly that okay so that's it it's a short talk and that's what we used to kind of achieve our immutable infrastructure I wanted to keep a large portion of the talk open for questions and we have the time to do that just to quickly kind of summarize it for you we built to do immutable infrastructure we built images using packer that are formed, picked up these images along with GitHub post hooks post commit hooks deployed them we managed access through Vault we had Nomad worry about the resources are being used well and we had console kind of tell the application layer about configuration as well as services where they exist and if they have moved where have they moved to right and all of this is possible only if you're using a cloud provider which I think is now become the default so everybody is you can't do this with software and we are happy as that baby is things have been working well that's it open for questions and you can always reach out to me in case you think of a question later that I could answer Hi you talked about blue green deployments so one of the you talked about blue green deployments one of the problems with blue green deployments is when you have say 50 servers and 50 more are coming in I can't hear you this volume is a little too low for me hello can you hear me now yeah partially yeah so with blue green deployments you face a problem where you have 50 servers and then you have 50 more servers that come in all of them let's say they are connecting to a database and you have too many connections on the database so your database goes down that sort of problems have come up so how do you suggest we solve these problems yeah so that's a very real problem right the first time you do it that's the first iteration of what you do you take your database server down second iteration is obviously you move traffic slowly so the key thing to understand here is that traffic is actually not changing so it's not like your load is doubling anything that is not going to blue is now going to green so as long as you disconnect from as long as your application is not maintaining too many connections when the load goes down which is usually not the default behavior you will not have that problem so the way we solve it is essentially do it piece me so you can move 25% of your traffic let this establish its pool connection so on and so forth and then move the rest of the traffic and change the thread is only because you have your connection pools now there are two sets of connections Hi here couple of questions here one where do you store your state store first point and second point is what is your backup and recovery strategies on that sorry I couldn't hear the second part backup and recovery strategy let's say for example you store your state store in a console or S3 or somewhere in your locally and what about let's say if you lost the data then your infrastructure is nowhere in there and how exactly you have the backup and restore strategies implemented okay so yes sure by state store I'll assume something I don't know if you mean something differently so you mean application state terraform state okay so terraform doesn't is largely stateless what it does is it will query and figure out what the current setup is and then it will match that against the new setup and tell you what differences there are going to be okay so if you go and change something manually that might pose a problem when you're doing this because terraform will read that and either just delete that resource because it's not in its plan so on and so forth so terraform itself is stateless in terms of other pieces of state right so packer will make an AMI, AMI is you can version and keep Docker image is you can version and keep they basically end up on S3 buckets which is where they end up so your state ends up there you're not asking about application state but application state is obviously database and is unaffected by our if you have a follow up question please catch them off there are other questions sure maybe I didn't answer it well enough hi so I have two questions I have to guess where you are how can I find you because the voice is coming from there and you're not there is that handed so I have two questions one on the blue green deployment one is on the slide which you talk about cost effectiveness so one is while doing a blue green deployment the IP is changed and I get added to one of the DNS endpoint so how do you handle or manage the TTL issue on the client side right so might possible few of the legacy clients let's say Java 1.7 or other clients might cash the IPs of the infrastructure let's say the blue infrastructure and when they change they might refers to the old IPs in that case they might be having issues that's one question on the D the second is we talk about Nomad right which will we can use to do cost effectiveness so the question is why Nomad why not using a native functionality of the cloud solution like auto scaling groups and like this sorry why not using what auto scaling groups of let's say AWS right so it has cool so I'll just kind of say your questions back I think maybe if others didn't hear it and just to make sure I've heard the right thing so the first is how do you solve the problem of services caching or other clients caching IPs that they're using so that's why you have DNS available on console so you don't have to resort to IPs they continue to use DNS and you put a short ptl on that DNS so when you switch the traffic shortly thereafter they will query again and receive those changes right and you have to sometimes change your application services to do that developers did not write it thinking you are going to do immutable infrastructure they may have bound to IPs they need to make a change the second question was why use Nomad and say not auto scaling you could use auto scaling and you could have asked me the question why not use Nomad that is no good reason one the second is it works very well with consoles other toolkits right so it makes it easier for me to do things it has richer control it has three kinds of jobs it does services systems and bad jobs so it has three kinds of jobs that it does which auto scaling does not give me that fidelity and will obviously not orchestrate my stuff auto scaling will only add hardware which is one aspect of Nomad it will not orchestrate it and third I kind of need things to I need blue green to work so I need events to be generated and they are sent out by say console using a service health check and I wanted to react to that and auto scaling is not by default setup like that you could kind of use cloud watch and do a lot of your own right a lot of your own code and achieve that in the end but it was just easier to use so the short answer is you could use auto scaling with a little bit of work or you can just use Nomad if you're using the rest of the hash it fits very well obviously auto scaling does not do orchestration I have a question here what was the storage back end that you used for vault here was it console or was it fire system we used S3 S3 as a storage back end for vault for all the secrets and my second question was what were the other use cases that you found for vault besides secret storage and encryption as a service so we only use it we didn't use it for encryption as a service it was on a roadmap but it's not playing out now but we only used it for authentic shipping authentication to applications that will need them to perform their tasks well but you can use it for other things I mean there are use cases I'm not an expert on that Hi, yeah now, so my question is that using Terraform was it like your go to place like in the first place or did you like see other options before going to Terraform I've been burnt prior before this experience with cloud formation and if you think cloud formation is good for you do it and then you'll realize it isn't it's very verbose very fragile and doesn't offer any of the guarantees collaboration or planning that Terraform has built for for example Terraform is very clearly in a consumable way tell you what it's going to change before it changes it and then you can approve that so they've built it with all of that in mind secondly they're they've used a tone down version of JSON to do that for their markup which is much more usable than JSON you know when JSON came out everybody said see much better than XML human friendlies but JSON is not human friendly at all and then you add YAML all of these other things come out which are which try to bridge that gap so this language uses something that is Terraform uses something that is human friendly and makes the job easier because you have to live in that code all the time excuse me yeah like you just answered why you choose Nomad like was it that the reason only the HashiCop tools I mean you were you were using or I mean why didn't you use the Kubernetes yeah so Kubernetes is cool because it's from Google but it doesn't follow a philosophy that I personally like which is the UNIX philosophy of one tool doing one thing Terraform does that Kubernetes is like Emacs you know you can get married on Kubernetes just like you could do on Emacs that's my inherent dislike for it but despite that it's a very competent tool like Emacs is also right and people are able to use it very well you're free to use that this is being made by an organization which is trying to get customers for their documentation and support systems are excellent they have professional plans available these kind of things matter immensely when you run it as a business unfortunately we as a side or devops do not think of that we think of what is the coolest thing that somebody is blogging about and Kubernetes usually shows up much higher on on that rating right and so people use it but I believe people have found it competent the only thing I've heard is that you have to do too much work to achieve too little I would still prefer using this over Kubernetes hello sir hello sir whose mic is actually active the two guys competing this is a race conditioning sorry somebody was going to win it I think I'll start so this is the first time I'm hearing about immutable infrastructure and that's really cool to know I'm just trying to convince myself that since I've already trained my developers to use docker images why do I need to train myself or them to learn a new layer on top of it so they already know like this is the Java version I know I want this is the Ruby version I want they can write it down in their docker files and they've learned and trained themselves right why do we need this extra layer on top of it I'll answer that in two parts so I think this is how your developer feels this is how you feel and you're discounting that don't discount that this will end ok that's one reason the second is still not using Java 5 you moved on because you found technologies change and they offer new possibilities so those are two really basic reasons why you should do this because this is a new way to do stuff which solves for some problems and you should do that now as far as training your over-training your developers then they have to remember what knife is in Chef which has no semantic meaning to anything knife means nothing you can either use it to cut bread or kill somebody it's some of the worst semantic naming I've ever come across you've taught them all that they've tried to remember all that they didn't need to so now when you train them again you can drop all of these what I find rather ridiculous concepts and do something much more much simply finally it also offers a career path for your developers so all of your developers are now able to code or you hire people who are able to code but they're able to now grow code to express what they're doing so they can easily their default way of doing stuff or our default way of doing stuff now is to write code to do it okay and not actually to do anything manually because we can't do it manually if we do then the whole thing will come down so I think there's some very progressive reasons and then you have to acknowledge this pain you're so used to it you don't acknowledge it what you're doing today is painful right that's a good reason to change always sorry the other thread so I have couple of questions sir so let's say when you are running this console maybe only one afterwards you can ask him so we have a lot of time my counter says 6 minutes why not I have not even run out of my time for this sure sure sure so let's say when you are so when you are running this console members and all into multiple DC's fans across this multiple reasons so at that time how do you guys take care of let's say one of the member goes down so this was my first concern and the second was like if you are again spanning across multiple DC's with multiple regions and all in AWS so how do you guys take care of this integration of Nomad clustering along with this console in again multiple regions then so those are very good questions so he's used console but he didn't study all the documentation so this is a very good question actually so there are things available that allow you to replicate stuff so one of the challenges so his question is what let me just put it in context he's saying when you run this multi region you know how do you deploy a console because the latency between regions can be very much and the console quorum will think and you know because it's on the internet there will be a lot more partition so the console quorum will keep like re-electing masters it becomes a bit of a problem that's like the technical version of what he asked so the way you do that is you choose an architecture where you have separate quorums for each data center which is also what the documentation advises you to do that leaves you with the problem of coordinated state or same state so it's the same company and so you have to have a tenant in one region and another tenant in another region you still need to know that somewhere so it's very possible for you to both console allows you to replicate the key value store that comes from one data center to another and it can be read only at that data center or it can be read and write so whenever there's a change here it gets propagated quickly across the van protocol to the other data center where it gets where it gets used so the short answer is think about this this is indeed a problem and architect your solution to have multiple quorums each for one region to kind of do this right I think there are lots of other aspects that you will come up with but I could answer them offline and become a lot of question so we had enough questions we have run out of time for this session I'm sure we have more questions you can continue it afterwards we have another joint Q&A session or you can meet him after it afterwards we'll go on with the next talk before that we still have two more BOFs going on upstairs one is on are we concentrating on deaths and second one is on how do you plan your container strategy for your organization both the BOFs will start at 3.10 on the first floor so those interested can move there in 10 minutes so we'll start with the next talk the next talk here is by Pooja Pooja is an open source enthusiast and an automation nerd with a DevOps mindset trying to bridge the gap between the various teams she blogs on open source, creates YouTube videos and tutorials and gives talks at conferences she's also rumored to be a MC at Root Conf she and her colleagues have hired an intern called Alice a talking bot so she is free to spend time playing Pokemon with a friends family and let the bot do all the work for her so she'll be speaking a little about the bot and how she utilizes the bot to help her do more work so let's listen to her relate this image to our continuous integration pipeline I'm assuming every one of us use continuous integration pipeline some or the other way can we relate this any guesses no guesses when I ask this to my friend she says it's your life no let's relate it to the continuous integration do we now what is missing that's our sorrows about test cases before release we know that there is a bug still we release or we know that we don't have enough test coverage still we release it's one less tire but truck still runs and it's fun now but what happens when you remove one more tire is it going to be fun more no 3 AM calls right so that's what this talk is about to be today we are going to understand how at my workplace we try to work on bringing the continuous integration and adding the test coverage as the minimum criteria to actually have the code base pushed to the next pipeline before we move on brief about me which is not brief so I'll just sum up it I find myself an explorer I have explored a multiple range of things in professional personal life I've been a developer, tester and automation engineer and I try to learn, develop things also so that I can fill the gaps between all of these teams basically and which helps us build and ship the healthy product personally I like to share what I know and learn in return so I blog about them and record videos on that and I'm an automation nerd as well as the open source industry so whichever project I work for my I use for my work I make sure to contribute back to it in some other way my latest contributions can be found in Jenkins and automation related plugins I'm a big time foodie and sleep lover but I use yoga so that my the Hulk inside me doesn't come out the same secret I told Dr. Banner that's why you could not see the Hulk in Letters Manager coming out alright so this is about me and so today's talk we are going to start with why I am talking about this topic what are the pain points we face and I see the similarity like you also face and what are the things we did to solve it so that comes into what section what and how it works and then how is about you how can you also integrate in your systems and Q and A at the end sounds fair yes oh it's after lunch talk yes alright so we go over it this is the simple data another pie chart bar graph but I'm not going to bore you I'm going to talk about a very interesting thing here this chart represents the code commits in our system this I just pulled from our code base so back then initially starting off more engaged we had a monolithic repository where we were adding everything in one sounds familiar everybody starts with that because that gives us speed and pace to actually add code faster and serve our customers faster start-up story so that's what we had but do you see any pattern in this graph so green bar shows the lines added red shows the lines deleted orange shows the team size at that time and then you have year on year do we see any correlation or any pattern here so we we grown up exponentially so we were size 15 and we are 50 contributors code contributors so any pattern you see code is growing fast features are going fast a lot of features are going fast but there is a pattern reverse the additions are lesser what can be the reason so this is what is interesting about so one reason is here so since we understood monolithic is not going to help us serve better to the customers faster so we started segregating our services into different pieces that's one reason you see the less number of lines added in the monolithic code base but at the same time you see the deletions also reaching towards the number of lines added what is that we are adding a lot of features we are serving customer needs very very fast any customer needs but still the deletions are growing up the reason is because we are re-architecturing as well we are bringing better coding practices we are bringing better services architecture so that we can solve the same piece faster so that's something but there is one more point here the point is we already have the code there which you call legacy code we already have the code there now you add a small feature the addition will be less because code is already there so now the problem started the code was transforming addition deletion more, addition more things like that everybody can do it that's where a point came when a single line developer has to add there will be a big pain point because a single line can break service at any point at any point and it has to be tested a lot so I'll give you a summary like how it goes so my take lead comes to me let's say for example I'm a core developer he comes and say that what is the timeline which practically if I think takes only one one line to change but in reality I see if I change this one line I have to test in several ten more features it means all ten more features has to be tested I am afraid to give timeline and my take lead is like why does the code frighten you so much and I'm like I'd be a fool if this big giant code has not frightened me up one mistake I'm dead why risk it so this is one common point when you have the code base which is not test covered like we had monitored code and we did not have enough test coverage you can say zero coverage let's say and we actually wanted to add one line we have to test a lot of places one thing and second thing is even to write test cases for them is a difficult choice so I'll come to that point again in the next slides so we wanted to solve this problem and we were brainstorming what should we do one ideal approach is okay you go and write end to end but are they sufficient or are they good enough or can that approach scale and that's where we went to understand what are the best practices so we have seen this slide a lot of times a lot of conferences people talk about it everybody sees it everybody go and pitch in company and what happens it does it work does it work easily any successful example you just went and said we should do it no the US smiling faces chose that yes so that's what happened so we we wanted to actually achieve unit test coverage more than others because they are less expensive more faster and the developer gets feedback faster and so the the entire QA the quality automation works more smoother than having only end to end test cases so we wanted to do this we decided to do this so what we did was like every like instinct we just said everybody that okay we want to do this from now on whatever code we add we have the test coverage we have the test cases for them unit test cases and have coverage for them that's the approach we took and then the real life happens the good intentions works only when you make mechanism to happen after sometime we realize saying doesn't work we did not have enough to test coverage still so the developer started writing test cases some wrote some did not write the people who wrote also they were not enough coverage so now why we have to dig down why there is no enough coverage there can be multiple reasons even the person do not know she is into development she is not thinking about the test cases now so that's kind of challenges so we wanted to mitigate this problem so like continuous integration pipeline creation we also thought okay let's go ahead and we also do it so give a context to brief we use github for our version control and we create pull request and we have a pipeline structure where each stage something pass then only it goes ahead so it looks like this so we call this a shield we named it shield it's shield task to understand everything and then move ahead so it goes like this so developer creates a pull request the first column and it goes into painting state second column if any of the checks fails so we have checks like syntax validation unit test validation any unit test also the test failure or the coverage failure so we have benchmarking if this is below this coverage it should not go away so any of this fails the pull request is marked to be blocked so it looks like this in real so you have the create pull request it goes into waiting and the pull request is if you see here the coverage is missing that is why the pull request gets blocked so this is how we thought yes now we can have coverage the pipeline is set it automatically works you create a pull request the checks running behind and keeps reporting to the github again so we found okay yes this works this setup works completely and it's awesome and we can go ahead and that's where the reality hits again one more challenge can anybody guess what can be that challenge so I'll give you a hint we talked about legacy coding initially and we are talking about adding test coverage for new code only so it happens so that the developer who is adding test coverage for the new code so let's say I have pull request which is having two lines of change and just for example imagine that there is already existing code of 100 lines you added two more lines your test coverage will say can it say if you wrote test cases for both of them can it say 100% no because it is contributing to the existing code base so we have legacy code base for which test cases are not there and we have new code for which test cases are there but it's still the pull request going into blocked state all of them all the pull request gets into blocked again we hit the same point either we bypass the system which we created we bypass and let it happen and if that happens then again we are losing the point of adding test coverage so we wanted to again solve this okay how can we improve on the shield process so that we can encourage developers to write test cases for newly added code why newly added code only because we saw in the first slide we are already knowing that our code changes our code is transforming faster we know that we are removing a lot of code we are rearchitecting a lot of code it means everything every piece every feature becomes new someday so we know that if we are if we encourage only to add for new rather than fighting old this is the legacy code this is not my code why should I write for it without thinking about all these things we can just concentrate on one thing add for what you return so exactly so we did so we saw that we have run checks and we have a test coverage report so we added one simple quick the question itself is answered the question is I want to write test cases only for what I added in the system and your system should detect that and tell me if I miss only that portion so that's what we do before running checks we actually save the code diff and at the end we pass that diff and pass the coverage report find the mapping ones and then generate another report which only talks about the missing coverage in your added code now which goes like this so it's not beautiful it's just plain HTML which is behind the that check where missing coverage was there the coverage is missed only in your added code so this gives us benefit that developers now feel encouraged and they feel more ownership basically because they have added that code and they should write they should be writing test codes for them and this approach works for us because we know that at some point all the futures will turn to the new code so it means every time we will have new code coming in and eventually at a certain interval we will have everything discovered so we are using the system from almost a year now and we and we reached from almost 20% coverage we reached to 56 with this approach and eventually when all services segregates or code gets re-architectured we will have its coverage complete complete for the complete code base that is all I have so this is the how part how can you build if you want to build so we have the open source APIs for everything we just need to connect the dots github APIs, Jenkins APIs for running the checks in behind basically and then we have one more system called Alice which we created last last year and open source as well which helps us automatically to talk between these APIs and do let she'll do only the parsing task so we do not need to build everything together so the continuous integration system could be built on that tool which we already had written last year and there are some things we have credits for the problems I always give credits to problems first because that's why we solve if problem comes and people so this idea is originating from multiple people in my company that we should be focusing on only the diff coverage so I give credit to them then github APIs, Jenkins APIs and photo credits that's all so this is what I want to end with 100% coverage is a myth but at least 60% is better than nothing that's all thank you so I'm open for the question and answers any points you will that you want to integrate or you want to try it out or you also have problems in a different context we can have a discussion on that I can pretty much relate to what you are saying we are in the similar phase right now but I want to hear your perspective in terms of functional test coverage like what is the coverage at your company and like how is it helping having 100% functional test coverage yes so for us a problem set is different because when you say functional test coverage is it developer functional or it's end to end yes so so we are a SaaS company which which actually the endpoints are not in our context so it's customers customers basically so we cannot hit the complete endpoint so that is why 100% end to end is not feasible so that is why we have broken it down into different pieces so this is a big problem but since this problem exists we are able to follow the good practices otherwise we would have again ended up writing end to end and then struggling to maintain them so we are focusing on piece by piece individual services so unit tests should go broader and then we are segregating them into different chunks like this should go into the API test context and then this should go into end to end automation so end to end automation we started and we stopped so we are again going to start that but then again the complete end we will not touch because that's not something in our context am I able to we can have further discussion on this any more questions? if you don't have I have a question how many of have more than 80% coverage wow I should meet you what's your name how are you unit test coverage end to end we are planning to start end to end writing end to end test cases and I mean I don't have questions on that but like what sort of like you guys started it and stopped doing it so that could be an interesting like story why did you stop doing it was it creating more hurdles than solving problems yes yes so that's what so for that I need to give a brief about what we do so probably this is the offline thing we should again get in touch that is all thank you so we'll have a joint discussion Q&A session with Pooja Kashif and we'll also have Vivek Shreeder joining us so three of them before we start the Q&A the social media announcement the hashtag for the event is root conf on twitter and instagram for the twitter handle is at the rate root conf as geek is on instagram at the rate has.geek you can tag use the hashtag on twitter instagram and you can also tag on again on both so yeah we'll start with the Q&A Kashif if you are in the room please come over all the talk videos will be available on hasgeek.tv so you can watch it after the conference any questions we'll start with the Q&A and we'll just get Kashif to join us soon find your nearest mic runner how many of you here building applications using microservices or distributed systems as well so if there are no questions we would like to hear from you and our stories of what you are doing and how do you monitor it what are the issues you are facing yeah the first challenge is like you said the tracing which makes debugging really a nightmare in most of the times even with the tracing mechanisms that we have in place as of now although the services interact between each other smoothly the problem comes with the database because the device ID that we have we had seen in Yager such kind of keys I don't have the luxury to add any arbitrary key for example if I'm doing an SQL query I don't have the luxury to add an arbitrary ID or maybe unique identifier or something in MySQL database so our debugging like the tracing ends at the service level and when we have to go till database we fall back to manual correlation that is one major problem and with respect to monitoring and alerting in when it comes to microservices so we have a lot to monitor and we monitor everything and nowadays most of the tools by default give enough metrics and enough thresholds the problem is we don't know what to alert upon and whether that it's challenging to decide whether should I wake up at 3 a.m. for this alert or can I sleep until 9 a.m. and can it wait until 9 a.m. so that is one major challenge when it comes to microservices for us I would like to hear the solutions on how people are solving the database relation between database and service when we are tracing or when we are building observability so yeah that's my major question yeah I mean that's a classic example right so the tracing ends at the service level and again the database debugging again it's a different story all together so I would I would say just check out debisium whether that solves your problem I am not too sure whether it would solve or not solve but I strongly believe you should go and check it out can I ask you a question just to clarify your question you said you are using microservices yes and those microservices have a shared database some of them help okay then you shouldn't be using microservices and that really is your answer if you are not able to separate your databases then those services are not independent they are dependent on each other and they are coupled to the database not at the service level so this is actually service oriented architecture no I am sorry I didn't follow you every microservice has independent database so you don't share database you should not share a database no they don't have shared databases so then what is the question that you have so imagine there is a data layer service which is retrieving data from elastic search or MySQL maybe now we know that and this is maybe a Kafka consumer or something like that now when we are debugging that certain message got lost somewhere in the transition we are able to find out that application logs we may be able to find out okay this has been the request has been sent to database and what happened in the database or at the database layer the logs are we have to correlate according to the timestamps and according to our query language but the client making the query can log the ray ID for you when it makes the query and tell you what the results were even correlated with that so the application logs will have my query ID which is correct so that can go all the way down until services definitely but when it when that in the database if the query failed due to any reason or especially in Mongolia yeah so we get the error ideas and that's where we actually have to involve manual collaboration to find out there is this database error when this particular query has been made the thing is when the query is going on there is a lot of database interaction is happening so there is a failure at the some level and when that something fails he will have to see which database query actually failed of course yeah so basically I mean that problem can be solved with debisium that's what I am giving you that hint but I am also not sure so I just want you to go back and try it out and probably publish it for us hello puja the question is for you I want to know that what kind of tool you have used to achieve the only line level coverage in the sense you are able to identify what lines developer has changed and what is the coverage on top of it so are there any off the shelf component or any plugin available or how that was achieved yeah I will just repeat the question to understand which one should correctly so you mean the tools and things we used to identify the lines added so basically that's what so everything is available we did not need to think about tool the github api itself is the git diff so when what you see in when you create a pull request git diff is shown to you that's the diff if you are able to fetch you just need to build a parse alright so when I say I am saying just now it took us 2-3 weeks to actually create properly because in your context it can be different for us we focused on only added lines but there is a case that even the deleted lines can affect the existing code to give you analogy let's say in deleted line I delete a variable usage in deleted line but existing lines the unreferenced variables is still there so it's a basically syntax missing it's not best practices this is just a simple example it can be even bad that the declaration doesn't exist kind of things can happen so we had to actually improvise it that what are the things in our code base what should we parse this diff on or what should we neglect so we had to do that so one repository will not have one language it can have multiple languages also so then you have to build your parser in such a way that you basically have the your checks basically have those sort of rules what is okay and what is not okay so for example for running test we are using python so we are using nos test and unit test library which runs that and which generates the coverage and all we need to do is from that coverage we have to map it with the github the same way for java different set of tools are there so we are using core libraries which are best at doing for those languages for example j there are certain tools I can't recall now which gives the exact test coverage we need to find a way to parse them Ghasif this question is for you so every time you you are saying that every time you go for a deployment you use AMI right which means you build those AMI so let's say I have to deploy a simple java system I need to install java and all the other dependency that I need so don't you think this is costly every time you keep on doing it or using something which is all how do you how do you reduce the time of build at that point in time that's my question valid question so you can have pre-baked AMIs and use packer to kind of add further to that so that's also that packer's also that is my voice audible? yeah now it's audible no so when you build an image you don't think in terms of what languages you build a composite image once which is your base image on which you are going to add your code and you can basically use packer to help you just take a pre-baked AMI and add the new code to that and ship it out and also the notion of it being expensive I don't think it's expensive I feel there is no real cost or no real time that is going to take so I don't know where the notion of expensive is coming from can you say that again with the mic maybe I am coming from containers where containers is basically just one line you can literally compile it on your machine and then push it off so we are also using containers actually not easy to instance but it was not relevant to the core point but again there is no addition in cost I don't see any addition in cost or any time being spent which is affecting your service delivery adversely I don't think there is any I think there is a notion that it is expensive because somehow I think we are imagining humans are doing this task somewhere but it's not actually it's not expensive it works fairly fast we deploy multiple times a day from a Slack command people just do it whenever they want there is no prep and very shortly when the deployment is done it's always asynchronous whether you use immutable infrastructure or not you don't wait for the thing to deploy so whenever the task is done you get to know on Slack and that's it nobody knows any different it doesn't cost anymore in terms of the only real cost is if you have to roll back and you don't have an AMI ready then any time you waste is expensive those AMIs are lying because they were previously released and so there is no real I don't think it's costly at all and correct me if I'm wrong so the cost is only the diff right so only the diff actually the cost it's not much yeah privilege matches only the diff is the cost this is to Kashif so we are also thinking to migrate to Terraform infrastructure as code but the problem as of now is probably there are 600 easy to instances with fairly large infrastructure so how do we go about if we are starting today on the legacy infrastructure how do we go about transitioning do we go service ways or do we go probably S3 as a module or there is again a good question there is no good answer actually any answer that will satisfy you is fairly long but I will address it shortly though so one thing you should do is most architectures are not homogeneous they are heterogeneous they are actually composition of parts is very rarely the case that this not so so you can take a part out and move it to Terraform while the rest of it stays in a different way now you will be faced with some problems around service discovery and things like that but you can have hacks around it while you do the rest so obviously the short answer is you decompose your architecture to parts and do it and do justice to the problem because there are a lot of problems when you do something like that if it is possible you should do it all in one shot it will save you over all time and it is very testable so it is not like rewriting code they are not equivalents and so maybe a rewrite in Terraform completely usually if your architecture is not massive is a better answer we will end the Q&A session here we have run out of time for this we will start with the flash stocks now so far we have two flash stocks in case somebody is interested we still have a couple of slots available you will just get a mic and you will have 5 minutes to deliver the talk if you want to do a flash stock right now just come over and let me know so we will start off with the flash stocks Chandan Kumar are you here can you go on for the first talk we have Chandan Kumar for the talk on BGP lag summer training hello everybody my name is Chandan Kumar I will be going to talk about DGP LED summer training am I audible backside am I audible yeah good so I will start with the questions so do you guys contribute to free software or fast open source softwares and cool and those who have not raised the high assume that you you have not contributed to open source or free software but you are willing to contribute to open source so before coming to this topic why should contribute to open source people contribute to open source to get a better job with good salary because salary part is complicated for everybody and ought to enhance your new skills or learn about new technologies so here your problem will be get solved by DGP LED DGP LED is Linux user group of Durgapur which does not belongs to Durgapur we are a group of people who contributes a lot to free and open source software starting from kernel to open stock fedora everywhere and we are a set of friends which are not friends in the virtual world as real friends we help them to become an open source contributors so how and DGP LED started with a motto first learn and teach others it is started by Kushal Das who is a C-Python co-developer back in 2005 and 6 so he and his friends were struggling with contributing to open source and how they can learn and say the languages say the technologies with others so they have started this group and it is a group based on IRC where lots of people hang out and share their knowledge and we used to do 3 month summer training program so that anybody can join it and learn the kickstarter things and get started and become a contributor to open source world the biggest hurdle for everybody to getting started in open source is home to contact and where to start the disability fix the problem and the free leadership for this training is nothing you just require a laptop with GNU Linux installed second you need a decent internet connection I think in India internet connection is available everywhere and you must have the attitude to learn and the training is free so by the end of the training what you will learn so the training starts with how to communicate properly on IRC or in free and open source world how to use your box when I say word box your computer how to use an editor effectively how to use a version control system as well as well as learn a language to solve your real world life problems and apart from that during the course of summer training we used to invite guest speakers who are already the contributors of various projects like last year Gudo who is the creator of python showed up on the IRC and he shared his journey how he has written python on how he is contributing to open source to change the world so if you want to become a part of this journey just save the trade 17 June 2018 at 7 p.m. on IRC channel how many of you know about IRC good about the free note server where lots of open source software will develop like VLC so just join this channel say hi on 17 June 2018 you can find all the information on dgplg.org summer training 18 and we will be happy to help you out thanks any questions we will have next talk by Rahul Bajaj on configuration management at its peak with 4 men hi everyone so let's start with the talk the talk is about configuration management at its peak with 4 men so about me I am Rahul Bajaj and I work at redhat as a software associate software engineer and I have been working on this 4 men project for about a year now so let's see what are we going to look at today we are going to look at the project 4 men does anybody already know about 4 men anybody has heard about this project anybody who has not heard about this project not ever heard about this project and the other people who have not think it is crap so we are going to see what is 4 men how it adds up to the picture key features and functionality of 4 men so yeah before looking at 4 men let's look at its history a bit how many of us already know what is puppet ever heard about puppet not even knowing what it is we have heard about puppet so puppet is a configuration management tool right now what configuration management tools are used for is to configure your machines your nodes right so puppet solves a great and a huge amount of problem for itself but if you see a systems entire life cycle right a systems entire life cycle would consider installation then you know actually go on configuring the machine and then you monitor the machine right so puppet solves a great and a huge amount of problem by doing the initial configuration but what about you know installing and updates and drift management what about them so 4 men is a complete life cycle management tool which looks after your installation initial configuration and also the monitoring of your nodes correct okay so three major functionalities of 4 men are provisioning, configuring and monitoring correct so let's look at each of them I'll just go through very fast because we don't have a lot of time so going back to provisioning we provide all we do all types of provisioning 4 men provides all types of provisioning on bare metals or you know cloud we started doing virtualization now we also support containers so basically you provide provisioning for all kinds all sorts of machines provisioning in the sense we do both pixie and pixie less provisioning pixie less provisioning we do it in terms of you know image based provisioning and yeah so we also provision virtual machines as you've seen and to provide DHCP DNS and TFTP services we have something called a smart proxies which we'll just see in a minute okay so that's our provisioning 4 men also you can use any kind of configuration management tool after you provision your machine like once you've installed the operating system on your machine on your server then you can use any kind of configuration management tool to you know configure your nodes like you can it also supports Ansible Puppet Chef whatever you want to use next is a monitoring we support monitoring we've recently I did Prometheus so we're supporting monitoring to like end to end monitoring of your systems this is the 4 men architecture so I spoke about smart proxies so what are smart proxies smart proxies are basically you can have one 4 men centralized server in whatever place you are right now and say I want to serve some machines in the US right so what you can do is you can place a smart proxy there smart proxy will basically provide you services like DHCP DNS and TFTP server you can use the smart proxy in any location and you can serve your or provision your machines in that particular locations and having one single instance of 4 men wherever you have it like okay so that's how you have a smart proxy which makes 4 men a distributed architecture all together which is which couples up with puppet for provisioning for configuration management in the second half we also have a web user API so 4 men has a API and you can use or you can manage your systems in the way you want writing your own using this API customization so 4 men basically yeah customization so 4 men basically has a plugin kind of an architecture so some people like to use 4 men only for you know reporting you can do that you can you know plug off all the other features like provisioning configuration management you if you want to use 4 men only for provisioning machines you can you know plug off all the others and so yeah it's a plugin plug off plugin architecture API and CLI so we also have a API we also have a CLI we also have a UI interface so whatever you can do on the UI you also have a CLI for it so you can write your own custom scripts and use 4 men the way you want okay last is so you can just check this tool out at the4men.org it's an open source tool so you can the entire repository is present on GitHub you can have access to it and if you feel free to ask questions on the 4 men channel the 4 men dev channel and you can also post your questions on community.4men.org yeah that's it thank you we have our last last talk coming up it's on MySQL 8 the best gets even better right so for people who don't know me I'm Balasubramanian I work as a MySQL release engineering team and this is again as I said this is kind of presented in G8 years and that's by my director and I just want to present something about MySQL 8 the best gets even better okay that's a safe our statement and we have few things to highlight these are the list of customers who run on MySQL Google, Uber and GitHub there are few others as well and we have published MySQL 8.0 on 19th April so that's very recent so I just thought of highlighting few things so that you are aware of what's happening in MySQL with the release part so as you said MySQL 8.0 is G8 and we have got few futures which is NoSQL documents store which means that now we say MySQL 8.0 has got SQL plus NoSQL and we have even got the JSON support in it so for people who are looking out for JSON support we have kind of a solution for it so 801 is the solution and we have got CDEs and we have got Windows Function and again a very nice improvements for the EnoDB and we have got a very nice replication improvement so we are going to see those in the slides and we have got roles introduced and we have got unique code so we are going to see how we know emojis and the last thing is on the GIS the nice improvement on GIS as well so I will go through very quickly just to be able to read few minutes so MySQL 8.0 is SQL plus NoSQL so having said that we have got the full JSON support through our new X-Dev API NoSQL interface and again on top of it we just the other points are like how we do it this is a very neat diagram so we are saying that we have got connectors on top and we have got a MySQL shell and it connects to the SQL API and for the MySQL shell it interacts with the X protocol and internally if you see that we have got both NoSQL and SQL so the document store it is the main thing about the NoSQL support and it has got a full NodeJS integration and it has got auto-completion and MySQL 8.0 shell has got a nice prototyping capabilities and you get a full SQL and X-Dev API support with a built-in auto-completion which is kind of a very important feature and the best part is you can set up a new DB cluster support within few minutes and it is a nice DevOps tool for someone who is like want to try out you can try this out it is available in dev.mySQL.com let me not go to it in depth I will just walk over on all the important points so we get it through the JSON table function which is available in MySQL 8.0 we get the full support JSON support so that is very important and these are the examples on how we get it and as I said the nested arrays it is in laps if you want you can try out these are the list of functions very quickly on the efficient replication as I said this is the graph which clearly shows that how things are improved with 8.0 and we have got this transactional data dictionary which means that all the system tables are kind of moved into the data dictionary and with that we have got an improvement significant improvement which means that MySQL 8.0 is twice faster as 5.7 and a nice support for CTE and Windows function and a better handling of hot row contentions this is very important because we have got very kind of you know things from the community saying that this is a very good future to have and we have got it Invisible features this is something for a DVA in case if I want to hide it I can just you know say that it's invisible keep it for some time and see that things are okay if we are not using anything we can just either software it or do a rollout SQL roles which is very important and again this is kind of implemented in 8.0 performance is very important and you can use all these variables to get the performance right okay world of emojis we get the support using the UDF 8 yes MySQL 8 defaults to UDF 8 MB4 just we have completely built in and ready to use so there is no extra configuration or installation it's very powerful if you got a full geography support and it's kind of you know projected it's flat and across two dimensions persistent in case it's a world of DevOps cloud yes we have got it through this persistent variable you say persist max connection equal to 100 and you are set inodb a nice improvement on inodb we got a document store json and we have got automatic ddl as I said everything is in the inodb database and this is what this community is looking out for they are looking out for an instant add column so what we have done is we have got it it will be available very soon in the next police. Replication there are a lot of replications for it nice improvements and very good improvement on the group replication and this is very important on the inodb cluster part as well we get the high availability achieved through this MySQL and these are few graphs which shows benchmark on read write says bench and a lot of future so I can't cover everything in 5 minutes just all set and last it's upgrade checker which can help you in upgrading from 5.7 to 5.8 to 8.0 and coming soon as we are going to have your Oracle Cloud data center soon in India and MySQL 8 will be on cloud by June May yep few things for you to go over for more learnings any questions you can reach out in our booth we'll be happy to assist you thank you we are done with the flash talks before you head out just a reminder that we have left feedback forms on your seats after all the sessions are done you can whenever you are leaving just fill up the feedback form and submit it the box outside thank you you guys can head for your break now it takes 34 minutes so this here is an example of overshooting job now at 12 outside is supposed to schedule a job but it doesn't because it finds that the previous job is still running so the next job is scheduled at 12.30 and this results in a high chance of SLA bridge if it happens very often besides this we have we face some other problems and limitations in general like there is a continuous process even when no jobs are running it's very difficult to scale this horizontally in this form there is no way to run on demand jobs lack of visibility then logs are all interspersed because multiple threads are writing to the same file and it's only specific to Clojure and Java this is quite important actually so at this point I just need to clarify that since I said Quartzite and then immediately moved on to problems I just want to clarify that it's not about Quartzite Quartzite is an advanced scheduler and it is capable of solving the problems but that would need building all those things whereas Jenkins has those as we will see in the course of this talk so in this slide I just wanted to clarify that so let's move to the problem statement so as I said we were looking at building a platform for all our job scheduling needs and we started with these goals we obviously wanted to target the problems we were facing like prevention of SLA bridges distributed execution of jobs that would facilitate horizontal scalability job pipelines it would allow us to compose bigger jobs out of small jobs UI for running on demand jobs common functionality like the we want the platform to provide all the common functionality easy to write jobs and on board them and finally not just limited to Clojure or Java so why did we choose Jenkins this is a very valid question if you go to the website of Jenkins it says that it's an automation platform but a lot of people only end up using it for CDNCI or for running tests because CDNCI are its most highlighted capabilities but it is an automation platform it comes with a built-in job scheduler it has a very active community which is highly responsive to security vulnerabilities and it has a much better plugin ecosystem we had prior experience with this platform at the time we had to use it as a solution we were already running another cluster for CDNCI it had evolved over 4 years to support 500 jobs and around 20 slaves and it aligns with our philosophy like we want to build something on top of what we are familiar with because we invest in technologies that work for us then we reuse them and standardize them so it helps us just reuse code so now let's look at the Jenkins based approach in the course of next few slides I will be speaking about the various building blocks of this solution beginning with Jenkins of course so we run Jenkins cluster in a master slave configuration like this where we have one where we have one master and multiple slaves and master is the node that takes care of scheduling jobs it has a queue it shows job definitions and it provides a web UI whereas slaves are the nodes that connect to it and job gets executed on those nodes the second building block is the job wrapper script so this is essentially a python script that is installed on all the slaves so you can think of it like a layer between the scheduler and the code that is written by the developers and it is this place where we can pack a lot of common functionality because the scheduler will execute the code through this wrapper so we can we have retries timeouts and monitoring handled by this wrapper and which means that our jobs are very lean and all the reusable things have moved to the platform a third is the code itself which encapsulates business logic and it can now be written in any language the only requirement it has to satisfy is it needs to run like a command line script indicating the success or failure based on the exit code and fourth part is job definitions so now that we have a scheduler and the code we want to make the scheduler aware of this code right so to do that we need to create jobs on Jenkins but so it can be done manually but why do it manually we have the pipeline DSL plugin in Jenkins which allows you to describe your pipelines as code so it provides a groovy script groovy DSL for that so we have developers write these groovy DSL as groovy scripts as well and we keep it along with the source code in the repository and then we have the release integration which ties all these things together so you can imagine the release process to be made up of these four stages in the build stage we package the source code and groovy scripts into a tar wall in the pre-deploy stage we prepare the nodes for two things one is to be able to run the code and the other is to be able to join master or slave and in the deploy stage we just copy the artifact nodes and extract and in the post-deploy step this is important we trigger a special job on master called the seed job this job is going to translate all the DSL scripts into the Jenkins jobs so to build all this we use a lot of plugins like we didn't write much code all everything was there we just had to tie it up together so the first plugin let's look at the first plugin here which is pipelines it allows us to write multi-stage jobs to compose complex jobs out of smaller jobs so here is a pipeline which shows four jobs four stages where A and B run in the first two stages the third stage we have C and D running in parallel and the fourth stage we have E so the beauty of this is that these jobs can run on different slaves and they can be written in different languages and they can be maintained by different teams so an auxiliary plugin to the pipeline plugin is the pipeline DSL plugin it provides a groovy based DSL to describe the pipeline so here I won't go into much details in the interest of time but essentially we are just telling it which label which slave to use and then we are just listing down all the stages then we have the job DSL so this is similar to pipeline DSL in the sense that it provides a groovy based DSL but for describing how to create a job so we use this for the seed jobs so what seed job does is it extracts so this is the last step of release flow and the seed job will extract the artifact and it will just loop through all the groovy scripts and just create jobs for them the third important plugin is the Jenkins swarm slave plugin this is what gives the platform its distributedness and it enables auto scaling so without this in absence of this plugin the way Jenkins master connects to the slave is that it opens an SSH connection to the slave this plugin flips it around so it allows the slaves to initiate the connection to the master and therefore the master doesn't need to know about the slaves now so this is an important property of distributed system where your cluster size is not known in advance and it helps with auto scaling so now slaves can come and go as the load increases or decreases and it also helps in HA which we will see later and finally we have the matrix plugin which helps in monitoring it provides a drop wizard matrix API it's basically just HDP API exposing health checks and this is consumed by a sensor plugin which then sends alerts so what did we so these are not all the plugins we are using I am just highlighting the plugins that are important so what did we gain out of this first of all we could solve this problem of restarts now releases don't affect the jobs because there is no restart happening here when deployment takes place the job would be running on some slave and the master would get the jobs config would get updated on the master so there is no restart here and there is no SLA breach because of this problem anymore the problem of overshooting job is solved because Jenkins queues the jobs so in the previous example we saw that this third job run was overshooting in that case quadzite skip in this case Jenkins will queue and the job will start immediately after this one finishes which is at 12.4 so the interval between the job has now reduced from job runs has now reduced from 56 minutes to 4 minutes and it is horizontal scalable because jobs can be distributed across slaves it's also possible to have HA for slaves like if you need that the slave should always be available and some slaves make auto scaling possible these are some of the other benefits we get UI for free from where we can just run the jobs all the common functionality is not now into the platform thanks to the job wrapper it's easy to write because of this it's easy to onboard because of our release integration better visibility and logs because of the UI again and it also provides this RESTful API ACL and all these things which are usually required for any UI yeah so let's look at how it's doing in production we are running it in production for a few months it's a new project so far we have onboarded around 32 jobs on it we have 13 slaves and so far we have run on more than 15k jobs and so just to give you an idea of the load we are running around 100 hours of jobs per day on across all the slaves when we think about when we talk about running anything in production high availability is very important unfortunately Jenkins doesn't have built in high availability for master because here the most important component is the master as it schedules so we have an active passive setup where besides the master that's actively scheduling jobs we have the passive one it is similarly configured to master and continuously syncing from the master syncing files from master using this tool called unison now let's say something goes wrong on the active master we switch over that is we make the passive the new active and since these are swarm slaves they will re-resolve DNS and they will connect to the new master and again monitoring is also important production for jobs it's handled by the wrapper script for master we for jobs it's the wrapper script is doing everything so developers don't have to worry about monitoring at all for master we have the process checks and the health checks which are exposed by the matrix plugin as I said the senzu plugin which consumes it and for slaves we have process and health checks for the swarm client process so this is still in development and we know that there are a few issues so these are the most glaring issues the HA for master is not real HA and at present the switchover is also not automated so this is something that we know and we can live with it now but this has to be fixed soon second is auto scaling is not implemented yet this limits our use of this platform only for predictable load and going forward we also want to use it for other use cases and future plans as I said better HA with automated switchover auto scaling of swarm slaves we want to implement state passing between pipeline stages something that is not there in a lot of platforms and so it will make the it will make the Jenkins pipeline behave more like a unix pipeline where data can flow through the stages and finally maybe we may write this python wrapper in java like we have rewrite it so that it can be packaged as a plugin that would also mean that we are tightly coupling with Jenkins but we are not sure this we are thinking on this and that is it thank you no questions guys ok must have been a good break no question you mentioned a unison unison plugin that you are using to failover is that open source or it is open source and where can I find it first link like which company is it help shift itself that is developed yes ok thank you any more questions many here so the jobs get queued up right so is there any flag where you can check that not many jobs get queued at single point in time yes so the matrix plugin I just mentioned right it exposes those like exposes that those things over hdbi we can write auto van jobs which pull that api trigger alerts how different is Jenkins from run deck jobs from which jobs run deck run deck run deck so to be honest we we didn't evaluate run deck when we started building this like later on I did take a look at it and it's quite similar but I think there are quite a few things that so the most noticeable thing was that Jenkins has and that it's coming from someone who is not work with run deck at all so Jenkins has a far more matured plugin ecosystem and the kind of plugins we have found it like they kind of very closely matched the use cases that we had so that's that was my evaluation after I came to know about run deck and until then we had already implemented this any more questions in the example that you showed so you showed that 3 and 4 they were running in parallel so does it also handle synchronization that it will only run job 5 after 3 and 4 and it's provided by the plugin out of the box yeah it's provided out of the it's part of the pipeline plugin yeah this is related to the query she asked run deck and the Jenkins I had evaluated run deck I mean last year run deck is mainly for operations like whatever tools we have currently so we can develop any self services or automations Jenkins is more into like development day to day like whatever the development deployment and the feedback we want from the code daily we are checking in so this is the main difference I see so this is not purely I would say maybe apple to apple comparison we have between Jenkins and run deck but that is a lot of overlapping functionality and I don't see why you can't use Jenkins like I didn't see the one advantage we would get by using run deck or rather I would say that I could see the things we would lose by not using Jenkins in this case but we can speak like if you have any more points you can okay thank you okay thank you so next speaker is Swapnel Dubey he is going to talk on compute intensive applications on DCOS so Swapnel is passionate about designing and building scalable big data systems these days his expertise is in developing infrastructure for data science use cases when not in front of the computer he likes to spend time with his daughter he also participates in cleanup drives and contributes to child education Swapnel welcome to the talk on compute intensive data applications so a bit of history for me about me I have more than nine plus years of experience and I'm right now working with with XRATOM software services as the big data architect I have experience working on domains like BFSI e-commerce ad serving with companies like snap deal, pubmatic and slumberj for the past one and a half years my technical journey has taken a shift towards kind of developing infrastructure primarily for the use cases which talk about running data science applications for example how we can run the TensorFlow and how we can have the GPU enable TensorFlow running and dynamic scheduling and all so that's a bit of a background for me the agenda for the talk is that I'm going to take you through a journey where I'm going to introduce a use case which we are handling right now which is primarily a data engineering and a data science mixed kind of a use case and how I will be talking about how we used to do it like eight nine months back and what were the issues with that strategy and then I will talk about how we how we slowly re-strategized ourselves and move towards running something on DCOS few positives negatives what are the improvements we are thinking about how to improve our system what is the aim we want to achieve I will talk about that as well about achieving the aim as well as well as I have few demos for you just to make you comfortable that yes what I'm saying is that simple so our use case as I said is primarily data engineering and data science use case data engineering we have Hadoop jobs we have Spark jobs we have Kafka for doing ingestion pretty simple and standard pipeline for data science we had R scripts written so more on the BA side we have people who are kind of the role of BA so they primarily write these R scripts and we have few data scientists who write these TensorFlow jobs and how we try to solve the whole situation where both the systems can run together is basically thinking about running big data applications into containers now running big data applications into containers there is something like a kind of an overhead as communicated by few of my colleagues especially when we talk about systems like Hadoop or Spark where they talk about horizontal scalability you can add the notes and you can run the applications but I think after the next slide maybe I will be making more sense when I talk about doing continuation for them as well so our client is an e-commerce giant the data size what we are handling right now is approximately 55 GB per hour and it is traditionally a company who has DE and DF so earlier what used to happen was as an e-commerce giant we have the use cases in data science as collaborative filtering and logistic regressions and finding out that what exactly is the sale with respect to the way the price of a product is getting changed but now the thing is that with the size increasing every day and complex user patterns coming into picture our data engineering pipelines are becoming more and more complex and along with that we wanted to have a look into the deep learning use cases as well so we wanted to leverage TensorFlow so this is kind of that six, I mean this is the picture which was like six months back we have various sources, we had written a simple Java spring based REST client so these sources basically hit our REST client with the data which is primarily in JSON format the data gets through Kafka and gets to flume and gets ingested into HGFS so Kafka for those kind of system where SLA is not that primary concern but flume is kind of pushing the data through your system for example placing an order or finding out basically identifying the use cases for fraud detection and also for those couple of small use cases we were using flume as well and for the rest of them for example the user clicks viewing of the product and these kind of scenarios we are primarily using Kafka we are ingesting that into HGFS and from there as I said we have Hadoop and Spark jobs which process the data so basically they do filtering and sampling and all those kind of things and then they finally put the data into GCS and from there we have a Kubernetes cluster so basically we have a basic Kubernetes cluster running and we have the Docker images being created in different environments like if our BA wants to basically run an R script we have the provision of spawning a container which is apt to basically taking care of that R script and once the execution finishes we had written a complete job kind of a script which keeps on checking periodically that whatever pod has finished it simply kills that so our cluster grows in size and it comes back and mind you this whole infrastructure is right now Google Cloud Platform so with this Kubernetes kind of infrastructure on Google Cloud Platform there is something known as a concept of preemptible nodes and all where if you have those preemptible nodes the condition is that they will go away after 24 hours but the price associated is just one third of your actual nodes so those kind of nodes are not proper when we talk about running Spark applications but when we talk about running certain burst of applications like people firing 100 R scripts together I think that kind of infrastructure can work which runs for maybe 15-20 minutes and goes away a bit more on the size on the infrastructure what we were using so the event which we are ingesting is close to 1500 events per second which looking at 1600 to 700 kb it comes out to be somewhere around 55 GB per hour the rest layer we have 10 nodes right now 25 nodes for Kafka so how we do is that even if let's say we have we are trying to do let's say a task 8 and we want the data to be processed from stage 1 to stage 2 if anything fails we try to save the state into a Kafka job so that we can again go ahead and reschedule that task again so that is the reason why we have kind of a lot of Kafka brokers running as I said flume is just a very small use case for couple of use cases where we have kind of an SLA needs to be met we are using flume so it is pretty small at that point of time and for running my Spark and Hadoop jobs as I said we were using GCP so in GCP the managed Spark and Hadoop environment is your data proc and we were close to 100 nodes there for the Kubernetes we have 2 bare minimum nodes so 2 nodes where like our complete job script was running and 2 nodes were running for running all those housekeeping tasks containers apart from that everything was kind of scalable in terms of preemptible machines and it kind of the way we were firing the jobs the bigger that I mean it was a big team in terms of from the BSI as well as the data scientists so at an average normally it goes around 100 to 120 to be frank because at that point of time we were not doing the proper sizes to the containers and all so there was a bit of mismatch at that point of time but if I just refer to the number it was close to 100 to 120 in peak hours kind of now here I want to mention one paradigm shift kind of so I think my journey with Hadoop started 4 years back and at that point of time we were kind of really thrilled when looking at a system that if you want to increase the infrastructure all you have to go and add couple of nodes to it and it will scale up on the same line spark came bring that in memory thing it became fast but how the things are right now is that but then again in the previous two scenarios you are anyways bound by the infrastructure let's say if there is a requirement where you require 500 nodes not maybe not for the complete 24 hours but maybe just for one hour you need 500 nodes so what will happen in the original in the previous scenario is where you need horizontal scalability to be kind of increased manually the number of nodes increased manually there we need to go and we need to increase the number of nodes but we cannot expect the data scientists or the people who are running the job to go and provision maybe five nodes for them and then go ahead and do the processing along with that with these spark jobs and the Hadoop jobs how were they kind of planned was that they were catering to the maximum size so let's say if I have an SLA to meet maybe processing the complete performing the complete processing in one hour and in the peak time if I require 100 nodes so we normally schedule 100 nodes and let it be like that even in the non peak hours where you are you are paying for the infrastructure but like not utilizing the infrastructure completely so the DevOps team which was there five to six months back was four in number and we have a lot of software components as I said we were using Kafka, we were using Spark we were using Azkaban for triggering the for scheduling our spark jobs we had Hadoop, Flume, Kubernetes, Jenkins a lot of things happening approximately 125 Kafka topics 93 spark jobs to be frank different teams sharing the infrastructure as well and the basic problem was that we need to meet the SLA's with the aim of optimizing the cost as well so if you see this was kind of an infrastructure what we used to have we had like three nodes for Flume we had Kafka on 25 we have Jenkins maybe setting on one machine 100 for Spark, two nodes initially for Kubernetes right and they are kind of provision for example the spark machines spark cluster specifically is kind of provision to meet that peak or SLA so the thing was that even in the non peak hours the complete resources which were there on the spark cluster was kind of underutilized and this is true for all of them I mean I agree that with Flume the infrastructure loss I mean the money involved is pretty less but then again if we can optimize on that as well it will be good for us ideally speaking what we wanted is that instead of treating each and every component as a separate resource group as a separate I mean applications requiring separate infrastructure we wanted to have something like this kind of an infrastructure where each and every applications are nothing but just the applications so within this complete infrastructure if there is a requirement for my spark job to use maybe let's say 100 nodes a capacity parallel to 100 nodes it should have the capability to do that and as I said we have a data science data science use case as well so in the non peak hours we had to schedule the training of the models so in the non peak hours when the spark cluster and everything is kind of free in those hours only we are going to kind of do the model training for our data science jobs so with that particular aim to utilize the infrastructure and in talking in terms of infinite scalability whenever it is required our first choice was to move completely to Kubernetes right so we and why it was our first choice because anyways our infrastructure was we had decent experience on Kubernetes we were already creating the images in Google Google Container Registry we have decent experience of handling Kubernetes as well we could have gone ahead and containerized every application via Kubernetes but there was something with a few points missing we wanted to have couple of more things to be covered up with this ship so few points which we wanted to have is deployment should not be able to tie to a particular cloud provider now how many of us had used Kubernetes may I have the hands up please how many have used Kubernetes is your Kubernetes running out of I mean is it running on Google cloud platform or in some other environment anybody who is running Kubernetes outside GCP and that is the pain point that is exactly the thing like Kubernetes being the baby of Google they had decorated it in a way they had decorated that beautifully it looks good when I talk about the preemptible nodes taking one third of the money and all but those facilities are only available with with GCP if you try to install Kubernetes let's say on on-prem the pain points are big and if you see if you try to search that on Google there will be just few blogs who talk about it right so people are not doing that kind of a thing so we wanted to we didn't wanted to tie ourselves with Kubernetes and Google cloud platform to be frank we have the aim of once the applications get stabilized we have the aim of actually moving to on-premise things especially those kind of applications which does not require this certain burst of resource users who don't have a requirement of certain resource users burst in the resource so that was the next along with that we wanted we don't wanted to compromise on adding new components so easy addition of new components is also something that we are targeting and this becomes very important when we talk about hosting our infrastructure on on-prem it is simple to use Amazon and use its product but it is difficult to have them like cross-platform and frankly speaking if you go ahead and search running Hadoop on Kubernetes you will not find any example just one example I mean very small example here and there but nothing like that and we wanted to have our production pipelines of Hadoop being running on on Kubernetes so that is again a negative point against Kubernetes in our case so Kubernetes look promising and initially and as I said it looks very easy on GCP it takes maybe not more than a minute to spawn it the size of the cluster increases flawlessly it comes down pretty well but the thing is that what if I don't want to use GCP right so in short installing big data components so RM was just not looking into the big the applications which are like which are kind of perfect fit for containerization right when we talk about applications which have to transfer a lot of data across the containers I don't think those kind of applications kind of scale that well if you have proper if you try to run Spark job on complete containers it is not going to give you that complete performance as such when you have that running proper on actual machines and the thing is that either to talk about Kafka, Spark or Hadoop they all have that same nature so installing those big data components especially the data engineering side was difficult now not enough support available so if you just go to that Spark 2.3.1 version which is kind of released last month they just talk about a single example on Kubernetes right and if you talk about the previous version there they talked about Kubernetes and for the next one and a half years there was there was complete silence right so it was not it was not happening properly your Spark is not inherently built for running into those kind of applications for TensorFlow as we were targeting so we wanted to use TensorFlow and we were using started using TensorFlow for running deep learning use cases and the and like 5-6 months back when we started with developing of this infrastructure at that point of time the GPU support for Kubernetes was in alpha phase so they had kind of said that don't use it into production and that is something we wanted to achieve and if you talk in the current scenario we have few very good libraries which support this data parallelism and model parallelism and deep learning we have a library cube flow which is picking up these days but talking about 5-6 months back these were not present so it was very difficult to run your models and train them properly using these parallelism mechanism parallel approaches as well as the logging thing so each of the pods have their logs but it is difficult to basically pull all the logs at one place and go ahead with doing the processing so that was also basic but the thing again that I am not advocating that the components cannot be installed they cannot be installed in Kubernetes they can but the thing is that the amount of effort with Kubernetes and not GCP is huge so how we solved it is that we we have Marathon in Kubernetes as the orchestration engines and below that we started using Mesosphere DCOS as it will come in the next slide it is nothing but an operating system so how are when we talk about an operating system on my machine it is an operating system which basically takes various processes and it gives you a feel that all of them are like one you can use your different applications can run on different course and they produce the result with DCOS the only differences that we don't talk about process you have multiple nodes and you have a one big pool of resources together so for this machine I have four processes similarly if I have multiple machines just add all the cores together and that is your DCOS capabilities so why we started looking into this is because big data ecosystem components like Kafka, Hadoop and Spark was kind of one click deployment it is just if you click one and you are good to go it has very good logging framework we were using Kubernetes we had done a lot of working Kubernetes and thankfully DCOS also supports Kubernetes to be the orchestration manager so we have a case where our whole pipeline could have moved as it is to DCOS so whatever work we had done that can be reused and deploying a distributed TensorFlow application was not that difficult here so this is DCOS it is a three layered architecture if you see the very top layer it talks about the services and applications so you deploy your services and applications and those services and applications basically pass through the middle layer which is an operating system and this operating system basically adds up all the resources which are lying in the in the underlying nodes so you have all the nodes here you add all the resources and that will start pulling to the complete infrastructure which is available to your one operating system right so here basically I will go with the demo and my aim is to basically show you that what all steps at the basic level are required to have a DCOS cluster up and running the infrastructure I am going to use is Amazon we have infrastructure as a code, a written and ansible we will be using cloud cloud formation for that first I will show that how simple it is to spawn up for doing your POCs and then I am going to use that same infrastructure for the rest of the use cases as well so we have cloud formation templates that are already available and what I am going to do is I am going to pick North Virginia,