Aha! Good morning, everyone, and welcome to the Fifth Root Conf. Today we have a lot of wonderful talks for you, which I'll introduce, but first I'll introduce myself. My name is Todd McQuillan. I'm an American living in Japan, who comes to India every year for reasons. And my good friend Zainab has asked me to emcee the conference for you today, so I'll be introducing all of the speakers. And maybe I'll tell a few jokes, I don't know, we'll see. So today we have, today's talks are focusing on monitoring. We have a few talks on monitoring, and also on automation. So those are the two themes for today. We also have a couple of talks that don't fit in those themes, just to break things up a bit for you, keep it interesting. Today's first speaker is coming all the way from Nuremberg, Germany, which is a small town of about 500,000 people. He's been in open-source business for about 10 years, and is an expert in all things configuration management, monitoring, and, oh no, I forgot my answer. So, without further ado, I will introduce Mr. Bernd Elg, who is giving a talk on the state of the open-source monitoring landscape. Thank you, Mr. Elg. Now it's on, now it's on. There's a cell phone. Okay, I'll call you later. Okay, so let's start. Good morning, everybody. Yeah, in the next 45 minutes I've tried to give you a little bit, kind of an overview of what's going on in the monitoring landscape. So forgive me if I don't mention every tool out there, because they have too much. So I checked GitHub yesterday evening and there are about 25,000, so I cannot go through everyone, but I picked the most interesting things, hopefully. So giving you a short introduction, like I already mentioned, my name is Bernd. I'm in the open-source business since over 10 years right now. I'm CEO of a company named Netways. We are based in Germany, and it's a service company. I'm also heavily involved in the deaf of the state movement. There are deaf of the state, a lot of local events, and I'm in the co-organizer team, so if you're interested in any deaf of the state thing, please get in touch with me. And the best way to do it is using Twitter. You can find me there with GetHash. So just one slide about Netways. What we are doing, we are a very old company, talking about tech companies founded in 1995. We are doing open-source since we are about 20 years right now and we are heavily focused on open-source data center solutions. So everything open-source in the data center is the field we are working in. And my last thing is doing a little promotion. We're having an iSinger camp on Saturday, working together with Haskeek on that. So if you are interested in iSinger, the camp will be on Saturday, start about 9.30 in the morning. You can find all the details on the website. So please join us on Saturday for a full day of monitoring here. Okay, that's the marketing area. Now we go into the monitoring. So if we talk about monitoring, we have to talk about what it is. So what's monitoring? I don't want to scare you in the morning, but perhaps wake you up a little bit. So there's no default to say what is monitoring. So I try to explain what monitoring is for me. The basic foundation I guess for monitoring is availability and functional monitoring. It means if I'm not able to ping a device or a service or something else, I have no idea what's going on, what the performance is, what the metrics are. So it means figuring out if your infrastructure is available, if your services are available, and if they are basic, it's a functional, if you can log into database, for example, that's the foundation of everything coming on later. On top of that, the metric and time series, metrics and time series are very popular, especially in the last three, four years with starting with Graphite. Previously where there were things out, RD tool, MRTG and all that stuff, but it was never so, people were never so interested in metrics and it changed for the last few years. So this is a big topic for me. And also locks and events means tools together, information out of the system, dealing with submitted locks and that. And another very important thing for me is user experience. It doesn't help you in any way if all your checks are super green and everything is okay, but the user experience is bad and the user is not able to use the web interface or set client applications. So getting kind of a perspective how your application is served to users is also important I think. So what to monitor? When we visit customers, when we are on projects, it's sometimes really not hard to figure out what should we monitor? So that's really a hard question. What is important for me? So there are different approaches to come to a full-featured monitoring. I would say if you have no idea how to start, the best way is to focus on your business. What the service is you're making money with? Internally or externally? But what's the important service? What drives your business? And to achieve that, I think a top-down approach for monitoring is very helpful. Bottom-up means you monitor every device you can find, do an auto-discovery and every IP address you get a successful reply is monitored, but it doesn't help you. It ends up that you get 2,000 emails in your mailbox and you will probably spend half of your day creating rules to move them to the trash. Therefore, a top-down approach is really very helpful. So first of all, focus, what is your business logic? So what are you doing? The funny thing is some people don't know what they are doing. I meet a lot of customers where they say, what are you making money with? Yeah, that's hard to say. So then you have a problem. So you should know what you're making money with. And then starting with a business logic, starting with external service which your customers use, which your customers are unhappy with if they are not running, it's a very good point to start. Then focus on the application. So that then figure out what applications are responsible that your business logic is up and running. It could be one application, it could be more applications ending up that you have a successful service of your business logic. Underneath of that, there are services. Means, for example, a Tomcat, a database, whatever could be needed for an application to be up, and then you come to the business logic on top of that. Of course, there are different perspectives on infrastructure because perhaps the guys are interested in, I don't know, fail this, are not the same people interested in a fail in the business logic. Perhaps your management is not interested in a hard-wise failure, but somebody should take care of because if all your heart is crushed, then you have a problem in your business logic as well at a later point. So there are different perspectives, depending on what is going the top-down way. So how to monitor so how to do it heavily depends on your perspective. Means what's important for you, what you like to see is something nobody can answer for you. So the perspective on your infrastructure on your services could be so different that depending on your employees, on your service guy who comes to your company tells you a totally different story. There's no no perfect rule to do it. One thing I heard a lot of times that people discussing monitoring is push or pull the best content. And I think there's no R, there's an M because there are things push makes sense and there are also other events where people make sense. So there's no R for me and to write this the same with Vim or Eximus better means there's no, perhaps there's a better one, I don't know but definitely there's a push and pull for me. So sometimes it makes sense to go to a machine and get out the metrics. On the other hand it's also important to deal with passive events coming in. Metrics are standard the push way or for example if you're leading with FNP press for example they are still out there and I think we are not going to kill them in the next 10 years. Personally I don't like Outer Discovery. Outer Discovery is very helpful in marketing because you press a button when you have 10,000 of green or red lights but the quality in Outer Discovery environment is most of the time not very good because what do you have? You have a bunch of services you figured out existing in your environment but perhaps it's a laptop, notebook, workstation whatever so it's really hard to make a good environment out of an Outer Discovery service. There are some exclusive services that you can use especially if you work in big network environments Outer Discovery could make sense so if you have a good tool for example OpenNMS which is really good for telco for big network environment there's a very good Outer Discovery ability also creating dependencies on that there could be helpful because creating for example dependencies on a network layer could be really really hard work if you do it with infrastructure as code but I would say in general IT services infrastructure can think about infrastructure as code means you have a process where you define where your services are use a configuration management if it's puppet or Ansible or whatever we have already something where you orchestrate your infrastructure where you say that service should run of a bunch of machines and monitoring should be a part of it so the time where you create all these services and then open up a ticket to the monitoring team and say please take care of the monitoring please don't do it so monitoring must be a part of your life cycle the service goes up it has to be monitored also in the early stage of the development it should be part of the process that monitoring is an important part of that and also if the service goes down then it's going to be removed out of the monitoring system so infrastructure is for me the only thing to do it right with monitoring that monitoring is part of the process because it reduces failure rates it reduces alerts on things that are okay that they're not longer here so definitely for me a good way and also an important thing to choose the tool for you so if you see that you should know and you should figure out what tool can I configure with my favorite configuration tool provide monitoring as a service if you have a monitoring make sure that the other guys in the company the other folks have a chance to to work with that monitoring system so provide an API interface whatever that they have a chance to participate in the monitoring and they're not required to open a ticket write an email monitoring should be a fundamental part of your infrastructure design and therefore you need something like a service interface whatever you use but monitoring needs to be a part of the process and that you can engage people to use it you have to provide kind of a service if you would like to have a service monitor independent from if you are on duty or not but use an interface at your service here and then we have in the monitoring system then you get your metrics also create dependencies on that so means if you don't edit in the right way you will not able to get metrics for your service so that's an important fact so so coming in explaining what I think monitoring is and how a technical approach can work let's talk about the tool because this is why we are here hopefully in that talk talking about availability and functional so what's out there there are dozens of tools out there there's no reliable database what opensource monitoring we can look on Wikipedia they are about I don't know 64 tools there's a monitoring survey James troubled it for a couple of years it's a little bit outdated this is from 2015 but it gives a good impression what's out there so we ask people on a yearly basis usually what are you using what are you doing with your tools what's important for you and in the tools of eye you can see that Nagios is still number one then there are a couple of star services like CloudWatch, Nurellix, some homegroom tools whatever that means usually it's modified Nagios or something like that then a thinker is coming up sent to Subbix and all these tools which are from the open source on premise area here I will talk about except Optu and Sentrum because they are kind of flavors of Nagios and I don't want to cover it again they are of course individual products with advantages and disadvantages but it's also not possible to put it in the 45 minutes talk so I will focus on the open source on premise tools in this survey so what they are Nagios I think are sent to Subbix Riemann and OpenNMS perhaps you figured out that OpenNMS is not in the survey but I think it's worth mentioning it so I put it in let's talk about the first one it's kind of a love-hate relationship I was using Nagios for years I was in the Nagios community advisory board once and the problem is I think Nagios is a good system but Nagios was a good system at his time like steam machines were cool at some time so Nagios has a lot of advantages and it is easy to extend and all that stuff but definitely today I would say they are better alternatives so of course you can use Nagios still for if you have to monitor a 2030 host and Nagios still is really really reliable because the codebase is so old and so many people patched it to death that it really works but there are better options out there so I'm not a Nagios hater to make that clear but if you start new if you think about doing something with monitoring please don't start with Nagios start with anything else but not with that sorry Nagios somebody's here I think the other one on the list also that I'm involved in the I think project we have pros and cons as well and I would like to treat every product fair I think a tool came out of the Nagios for originally so the I think a project for three codebase into I think but at some point we figured out at this time that it was really hard to change the codebase and then the product the team started to rewrite the code from scratch in C++ advantages definitely in I think are that there are a lot of integrations to other tools are built in means if you would like to write metrics to graphite influx to be open to to be all the stuff it's just in future you need to enable means you don't need some other external tools to make that happen it has an application based cluster stack means the rest API to add delete modify services during runtime makes it very easy to have it in an HR environment and that's definitely an advantage and disadvantage could be in I think because there are so many possibilities to have active passive checks and have the checks running on the server on the client that it could be complex to set up especially if you have a system with multiple nodes or the certificate thing we need to make it secure and all this stuff people have sometimes problems with it always room for improvement of course the documentation is not so bad but we regularly figure out that people have problems with so much possibility talking about Sensor, Sensor has in general a very similar approach like Nagas and I think they also have kind of standalone and subscription checks means the server can do stuff and also the client can do stuff a lot of people complaining about RabbitMQ which is necessary to run Sensor I'm not a RabbitMQ expert but if you look at the forums a lot of people complaining that it's not running it's not stable and it's hard to install I don't want to charge on RabbitMQ because I have no idea but this seems to be a problem with running Sensor and production that people often have a problem with a transport layer in there there are no historical data if you would like to do SLA reporting afterwards it's really hard in Sensor because the information is there and locked but you don't have a data model to access it and what's really sad is that all Sensor move to enterprise only dependencies for example there are no dependencies in the open source version SNMP move to the enterprise version it's not wrong to have an enterprise version don't get me wrong but I don't know where is the border between open source and enterprise version here I think the open source version of SNMP is really hard because it's probably not enough for an enhanced monitoring setup if you're able to buy the enterprise stuff in addition to that then it will pretty much do everything you need but this is something you should take care about Subix Subix is also very very popular monitoring tool in Japan for example is de facto standard so a lot of people using Subix there perhaps you know better but I heard so that the Subix is very popular there it's a full featured solution with Subix an advantage definitely is you get a lot of out of the box to install the agents turn them on and you get all the data, you get your graphs everything so it's really easy to start a new monitoring system logging and graphing is integrated in Subix that is easy it's a little bit harder to orchestrate but it's usually the case with all the tools as much you get, when you get more in up front then it's usually harder to extend the later works because all the things are integrated if you would like to extend it in some way then it's really not that easy and scaling Subix could be a problem because Subix also the satellite systems rely on a Postgres database but at the end all the data has to be written in a single Postgres database to be fair also other problems like you think I have issues working on that but that's the scale out limitation I know that customers have with Subix if you would like to scale out that your database needs to be very good to it means you definitely need to have a large environment to deal with all the data coming from Subix Riemann anyone heard of Riemann okay a couple of you Riemann is a project there's not so much going on it's my last point on the slide I figured that out that there's not much going on in the project right now Riemann is a streaming processor you have a server running and all the clients constantly push streams to the server it's based on closure which is really important to know that it's in closure because you also have to write your streaming rules you have to be familiar with that language to really work with Riemann in a good way if you would like to measure different things like a web server application server they provide different Riemann tools to send over the metrics to the Riemann server it's stateless it doesn't store any data it just handles it, shows it in a web interface and you can see the metrics so the advantage of Riemann definitely is also perhaps in a combination with another monitoring tool that you really have real-time information about your system so you constantly get your performance streams out of the system and see what's going on like I said before there's not so much going on in the project I checked GitHub again yesterday there are a couple of small documentation enhancements but there's not much to do on the other end it could be that it's just perfect right now and there's not much to do I don't know OpenNMS OpenNMS is on the market also for a very long time it's also full featured open source solution means you have everything in there like I told you before it's very good in heteros in homogenous environments telco network they are very strong in that area it's based on Java which is not important to say if you hate or love Java but it's hard if you fork out of Java means every time you leave the OpenNMS Java context if you have to execute an external plugin then performance is horrible they have a lot of stuff included means SNMP everything is inside the JVM because you have to do something which is not included into OpenNMS then performance is not so good but this is not a it's kind of an OpenNMS problem but it's a Java problem forking out of Java is expensive and therefore scaling out external checks with OpenNMS doesn't make sense but I also guess that's not a field of expertise Outer discoveries is really cool they are also really really nice guys from the OpenNMS project for years it's definitely a cool tool if it fits to your needs okay so leaving the functional monitoring let's assume we set everything up using our favorites we figure out something is running or it's not running metrics and time series what it's all about counting so we would like to say perhaps it ends up in money or not but we would like to know how the metrics are and definitely I would say so not talking about all the ROD tool based tools which are out there MRTG, PNP, whatever so ROD is definitely also a very cool thing the problem with ROD is for most of the people it's not fancy enough that's not a technical argument I know but sometimes you don't need technical arguments and that's definitely a problem that people would like to have more fancy graphs where it can add different things and all that stuff and that's hard with ROD so what graphite is the database the graphite is the whisperdb it's very similar to ROD it has a couple of differences for example you can also add data not in a serial fashion like PNP the dates have to be arranged that is not necessary for the whisperdb it kind of started the metrics revolution so it really started to be popular years ago something like that and the biggest advantage is also the biggest disadvantage because graphite consists of different components you have the whisper which is the database for the RODs you have a carbon you have a graphite web and some of the original components first of all are really hard to install so a lot of people fail with graphite because they're never able to get it up and running but also the different components they seem very good if you look out at github there are so many different components replacing individual components of graphite for example carbon seba which is like a proxy for the carbon cache there are so many things that it's really like a moving target and it could be hard to debug I know that there are a lot of large environments based on graphite and they figured how to deal with it but to start with it it needs some knowledge I would say definitely it's still kind of the standard I would say another thing is OpenTSTB anybody using OpenTSTB congratulations you made it pardon ah ok so OpenTSTB is pretty cool but it's hard to set because it's based on Hadoop and HBase means you have to know how that should work if you know it congratulations you know it then you can do it it's really able to scale like crazy you can also store all your data you don't have to reduce your data you can live with raw data forever if you're able to pay for the disk of course and if it's up and running it's very easy to scale based on the HBase and Hadoop you just did a note and that's it it's also a lot of monitoring tools provide an API to OpenTSTB so if you already have knowledge about Hadoop or HBase you already have a cluster that would be a good point to start if you just would like to start with metrics and then have to start with Hadoop and HBase if you have time anyway it's harder to start with OpenTSTB another thing is Prometheus and Prometheus originally was developed by a Berlin based company named SoundCloud for their internal metric storage they open sourced it and I think since about a year I don't know they are a member of the cloud native foundation Prometheus is also a time series database with a dimensional model means the database model is not tree based like in graphite for example you have these tree based metrics and Prometheus is very flexible in the database model originally it's designed for web service you can query externally means getting OS metrics or something like that you need other tools installed to be there there was no kind of a plugin mechanism on there for example you would like to get some load information or infrastructure information you need a node exporter to install it and what made Prometheus very powerful that it has a rule based alerting so there's a component in the alert manager where you can based on metrics and thresholds create alerts send it to a user and if you are if you have a setup that is really based on metrics and probably all your information comes out of let's say response time and specific load scenarios then Prometheus is really good to set up a monitoring based on metrics it has to fit your environment definitely in the cloud area means because also Kubernetes is in the cloud native foundation it gains a lot of attraction heavily developed moving forward so it's definitely interesting tools to check in the next year another thing is Influx DB Influx DB has a very similar scope to graphite I think the person would say okay let's make graphite easier therefore it's really much more easier to install and it has a very powerful SQL like query language so if you're familiar with SQL it's really easy to get metrics out of Influx the cool thing the enterprise also means horizontal scale out in Influx DB you need to pay for it and they put a lot of energy and also doing more with it means they develop the stack named TIC stack TIC is for telegraph what is it? chronograph and capacitor there are different components where telegraph is able to send metrics to Influx DB chronograph is the web interface to analyze it and capacitor is something like the alert manager and prometheor so the guys from for Influx data figured out that they more than a metrics database and created components to have probably the full chain from sending metrics to have a web interface analyzing it and also creating some having a rule-based approach to get alerts out to the user Elastic I would say it's the defective standard so who is using Elastic stack in some way here? definitely more than OpenTCD don't ask me why I would say they were the first player and they had made very good decisions in buying other projects means Elastic search is there for a couple of years based on on the scene and the Elastic stack also started in the time series era about very serious I would say one and a half years ago there is a Kibana extension named timeline for metrics and for time series this was pretty cool you have to look out Elastic Beats which is a method to directly send metrics information from your tool there are perhaps Beats for Icinga for Nagios for your application you can directly send metrics information to Elastic bypassing LogStash and then you can use Kibana to query this it has a different model approach because the fundamental concept of how Elastic search works in the database is different to the model of Prometheus or Breffat works and it's more important that you know what you would like to see where the graphite approach is more like put everything in and if you can afford it store everything and look later what you need in Elastic I would say you need to think more about what I would like to have metrics for how the node design should be how the object design should be Statsy could be helpful Statsy in combination with LogStash Statsy is kind of a metric aggregation from Etsy where you can work with with counts and aggregate specific values and putting that information back in Elastic is very powerful but it could help you in some way as well so now we found a way we can store in the metrics different of the tools have their own web interface like the Influx Data guys Kibana of course but if you talk about visualization so getting all the metrics out Grafana is it I would say right now the standard because it works with all these databases Grafana has interfaces to all these tools and more the guys working on Grafana Torgel and the Grafana team RainTank they're bringing new releases I would say every week and it's very easy to start with and it's very easy and it's probably one of the reasons for people using it because it's easy to combine different data sources also from different back ends into a single panel means to get some sources out of Influx DB from Grafana if you have more of that getting information about Elastic and it's a very cool thing who knows Grafana annotations not so many so annotations are a cool way but if you have your graph and storing your metrics in it for example a puppet run or usually a git commit so somebody breaks your system then you can show that event in your graph means you can probably look with Grafana into your metrics database and also look up to your syslog information and that's pretty cool because if you see that something is wrong with your performance, with your metrics with the response time sometimes it's just a puppet run exchanging some software make it very easy to see it you can see depending on the event it's just a test event you can also add more information here you can also say there's a git commit by developer xyz so you can call him so it's really easy to have a quick analyze if the performance changes what happens so if you use Grafana and have kind of a log management tool in the middle you definitely should give it a try first of all we have to start I think what the difference between log and event so a log is just the flow of unstructured data so hopefully we have kind of a timestamp makes it important to work with it but it's not more than we have a timestamp and we have a bunch of information in a log and if you would like to do more with it we have to split it up in different attributes going from a log into event means we have to check what is what what is the timestamp services responsible what is the message and I would say it's always the same process except you only have the by law you have to probably store your logs then perhaps you don't care what's in there if you don't have to store it but if you would like to work with your logs you have to make it into event so the process should always be like going from a log to event thinking about locks there's a great lock you have grog for example you have different patterns where you can split up into different parts most of the services are already out there so we don't have to rethink it again just grab it and then another bit and then take some action if you really work with logs you have first of create really the identify the attributes and work with it later on also elastic in that area definitely is more the standard than in the time series area it's highly integrated so since elastic at the time was the first one elastic based on the scene then they I don't know kind of board or adopted log stage developed by Jordan Thistle and then Kibana so they get all these tools together and also beats was an external project named packet beats before so elastic did very well in getting the right components together in a complete solution that it wasn't now it's the elastic stack it was previously the elk stack but since they added beats elk doesn't work anymore you don't have any user authentication or stuff in it and if you obviously would have everything in a fancy way you need an expect for it but how they do it elastic I think it's very good because for example log stage has a very powerful API which is an advantage to other so you can really go to a log stage instance and see what's going on how the data is processed how quick it is and this is open source if you would like to have it in a fancy way in your Kibana you need to buy the expect but I think the border they draw between open source and enterprise edition is pretty cool because you have everything you have the API you can access them if you would like to need it in a more fancy freeable way you have to buy the expect so definitely log stage API came up with I think log stage 5 a year ago one and a half year ago if you're working with log stage and you see some kind of processing performance problem the API is very very helpful and by far it's the largest community so if you think about log management all that stuff elastic is the biggest community another cool tool is graylock graylock is also based on elastic search so they use kind of the same database but the biggest difference is that all the configuration and the ruling and everything where you have to do buy hand and log stage is provided by a graphical interface also if something like authentication authorization is important for you graylock could be a very good choice because everything is in there connecting to LDAP having user rules all that stuff is freeable and it's easier to start with graylock because you have an interface where you can see your input sources, the output sources and it works also very good in combination with log stage means it's always it's an or it could be an end as well means you can combine all these tools and there are a lot of users using graylock but using log stage to get the information from the system or another tool named fluindy fluindy like Prometheus also joined the cloud native computing foundation it's kind of a unified log layer so I would say fluent could be a replacement for log stage it has a very powerful log layer and connects very system with others I have 2 minutes you have 9 minutes I only have 10 minutes okay I have the right time zone hopefully okay fluent could be a good alternative to log stage or if you don't want to use elastic search as a data store if you use something else because fluindy is supporting multiple backends fluent is also a good point to start an advantage over log stage could be that it has full reliability means it has a file and memory based buffer system where you can also replicate it to multiple fluindy services which log stage doesn't have so it definitely could be a good man in the middle replacement user experience probably the last area which covered all the others I don't know who is doing end to end monitoring in some fashion okay we'll not have a lot of fans here I guess it's not super popular means really figuring out how your browser works not a lot of people doing it because it's a lot of work perhaps I would say for a typical ops guy it's not so much fun to be with a front end and fill in different variables but it's really really helpful so end to end monitoring gives you another experience that you don't have a disappointed customer and it's really often the case that technically everything is right but the user experience is shitty anyway means not talking about a bad interface that's another story but just your interface doesn't work like expected and therefore for some services for some I don't know if you're talking about a web shop going through your shopping experience adding a product to the cart doing a checkout and checking if the invoice is correct could make sense there were these tools out there web and check and out it they were cool a long time ago means they are really not actively development web and check the last release was 2006 still people using it and out it is also a little bit outdated also the current windows versions are not supported there are two end to end user tools I would like to mention here one is SAKULE SAKULE is a combination of a tool named SAHI in one of SQLIX one is for web testing and one is for FET client testing and another is for NUGAS compatible systems means NUGAS I think are central and up to all that stuff most of the checks can just launch a docker container it's very much isolated which I'm not really sure that some of the SAHI features are enterprise only you have to check out what you need here and another interesting product I think nobody knows here I guess is a product named Alivix it's developed by an Italian company It's a complete solution monitoring web user experience, monitoring Fed Client. They have an IDE where you can really create test cases on end-to-end basis. And they have a full audit trail notification system as well, send you a screenshot for example if something is wrong, definitely you should have a look. So if end-to-end monitoring, if you would like to check your user experience, if you would like to see if your Citrix is working, they are also able to work with mainframe terminal if the tier thing, I would say mainframe and DevOps. I don't know, it could work in some way. Anyway, if you have it, they can do it. They can also work with Java applets. They can do pretty much everything. It's really cool. So the conclusion, so if you go out of the talk, come back next week and say, did you learn anything? So what now? So your boss comes on Monday. Perhaps he's not looking so bad, I see, or I don't know. I'm sorry, there's no best tool. And that was probably not the goal of the talk to say choose that one. I think it depends on what you need. Means they're kind of two different approaches. There's a monolithic way where you have tools that do a lot of out-of-the-box, like Subbix and OpenNMS, they have everything in there. You install the agents, you have graphing, locking everything. And if it's enough for you, then okay, because it's easy to set up, you don't have to deal with all the external components. Also, perhaps your main focus is not tech. So if tech is important part, but not all the guys are in tech and you just won't like to have a monitoring, then it could be a good choice because you get something easily. If you need more, if you need to scale it, if you want also play with the newest fancy hot shit on the market, then a modular approach could be better. Means if you're a tech driven company, if your ops are really up to date and would like to play with a new way, combining different tools together is the best way. So I prefer a modular approach on a tool chain because one thing is that sometimes also the monolithic approach is good enough, but the problem is at some point you see a new thing is coming up again. So there's a new metric system you would like to play with. And then going from a monolithic approach to a modular approach, and say I would like to replace my integrated graphing solution with graphite whatever, it's hard. Best is take your favorite tool, set up kind of a real life use case. So don't test your local Linux box. So perhaps in addition, but play with the use case you have in the company, play with the integrations. So try to hook different tools together, choose your favorite. Sometimes you are not able to make the best decision. Like in life you can go through every argument but sometimes it's just flip a coin. If you don't know, just start with one and figure out if you like it. And if you like it and if it does everything you need, then it's perfect for you. So I'm hopefully in time. Thank you very much for listening. Are there questions or are there time, is there time for questions? Five minutes. Are there questions? Are you awake? Kind of. I didn't get the question, I'm sorry. Hello. Hello. I was just curious why you didn't mention pager duty in this. Because of open source. Open source. Pager duty is cool. Like others, VictorOps and all these tools, but it's an external alerting service. VictorOps is also cool because they're using iSinger, I know. No, it's an external service. Like a lot of other cool things in the software as a service market, of course. Like Eurelix or CloudWatch or Datadog, Librado. But this is for me focused on open source. Possible on premise tools and therefore it's not part of it. Of course I know that all these alerting external notifications, SMS voice is a big part of a monitoring tool chain. You can do it on your own with various solutions, but definitely pager duty is a good choice as well. More questions? There's somebody. This question is not related to the tech exactly, but as I used both the open source monitoring landscape as well as the proprietary stuff. So in your opinion, how far ahead is something like Splunk pager duty compared to the ELK stack and everything in terms of for a scaling tech driven startups? Yeah, that's a good question. So if you name Splunk, Splunk is awesome if you can afford it.  So if you would like to store a lot of data, it's really hard to afford it. So if you say money doesn't, I don't care. I would say go with, it's super easy. You can call somebody and scream at him. It doesn't work. So Splunk is really, really good. But since you pay storage wise, you pay on the data you store, I think for me it's more alternative. What I see, a lot of customers go away from Splunk because, not because the tool is that, because they're not able to afford it. It's really only a cost reason why people go away from Splunk. Pardon? In terms of the features, definitely Splunk is much easier to configure. So I would say Greylog is more the Splunk approach than Elastic because all the configuration from the input and output is able to dump via web interface. Also enterprise integration to also open source tool would provide enterprise add-ons like Puppet or Ansible works pretty good with Splunk. Splunk is fast and Splunk is also capable to probably work with every input, output necessary. So it's definitely a good tool. So I'm not into Splunk, so I cannot install it. I can work with it. I know in general what it is and what it is able to do. But I would say it's really just a money reason. Otherwise a good tool. So like I said, if you can afford it, congratulations, it's a good one. I'm not meaning it bad. So if somebody from Splunk is here, I'm open for discussion. You never know who's in the audience. More questions? Yeah, there's just time for only one more question. Sorry? I've discussed about many open source tools. So my question is if we want to have an activity tracing, let's say all my use cases in the y-axis. I'm talking about a visualization and a particular graph where all my use cases are in y-axis and all my microservice one, two, three in the x-axis. So if I want to do something like a color-coded activity tracing for each of my use case, which tool is the most suitable one? Color-coded on what space? It's on the performance or? No, no, no. Activity tracing. Suppose if I have like eight microservices and there are eight microservices as one for UI and then one for the database, the service that's talking to the database and then to my IoT devices. So I want to activity trace my use case. Suppose let's say I'm switching on my device. So starting from my switch to the application, I want to activity trace it. So I want to visualize a graph like that. So which of the tools that we discussed now which will be more suitable to visualize that kind of a graph? I would say none of these tools. So if you would really go into application, New Relic is very powerful. Means it has some requirements because you have to replace PHP or JVM with their stuff but they are really able to go into it. There's an open source. I can tell you later, it's not part of the presentation but there's also an open source alternative to New Relic and we really can see what's going into your application and the combination of the user case coming from a service, what's happening in the database. I talk to you later because I have something but in general I would say none of these tools are good for it. Visualization could work in all these time series databases but they at the end just show a result of your previous work. I have something I can show you later. Okay, I'm here both today and also like I mentioned before, I think I came on Saturday. Thank you very much and enjoy the conference. Hi, can everybody hear me? I think I can hear myself. I am Zainab Bawa, I run Hasgeek and I am the producer for RoofCon. So thank you all for being here today.  I'm not gonna hold up Aditya who has other important appointments to do quickly after his talk but I will take about five to seven minutes. Very quickly, there have been two changes in the schedule today since Krum who was to be speaking about the anatomy of alert is not here because of health reasons and so Aditya Patwari, editor of RoofCon is here today filling in for him. The talk that Aditya will deliver today is on deployment strategies with Kubernetes. Sorry but surprises are always the case in life. We have one other surprise and I'm again sorry about it but it was completely unintentional. There is an OTR, an off the track record session on microservices this afternoon from 2.30 to 3.30. Unfortunately Anand has missed his flight for the second time and so this is rescheduled tomorrow at 3.45. The change is reflected on the website. I'm sorry but yeah, life is full of surprises. Very quickly I wanna make a very quick introduction to RoofCon and then want to call upon Satish Mohan to talk very briefly about DevCon and why we're doing it here. RoofCon is in its fifth edition as Todd pointed out this time. We have 500 plus participants who've registered across both states so it's gonna be a very big audience and I recommend that you make the most of this conference by doing a bunch of things. Please include, please attend the talks in both tracks. There is an SELINUX tutorial that will happen this afternoon. The Ansible tutorial is already on. Again, those who couldn't make it to the RSVP early, we hope that there will be more partnerships with Red Hat and others to do Ansible tutorials for the rest of the year.  And of course the others who are gonna be around across both ways. Participate in the off the record sessions. Off the record sessions are not panel discussions, they are conversations. We put together facilitators to help you to have conversations around microservices, AWS cost optimizations, art of software architecture, et cetera. OTR sessions will happen across in the porch opposite the auditorium. They will be in Pagoda, so please watch out and be there. There are two happening today. The microservices one is rescheduled for tomorrow but there's one on MySQL which Lig and Colin will be leading. And then another on AWS optimizations and hacks, which will be led by the Bangalow AWS user group and Jeevan Dongre. On that note, I also want to thank four communities who have actively contributed to this year, the Bangalow AWS user group for leading the OTR session. The internet freedom foundation for helping us push the digital rights agenda and to educate the audience on questions of privacy, network management, and internet shutdowns. Anurag Bhatia who will be speaking tomorrow from the Raipan CC activist project. We'd like to thank the IFF for helping us put this agenda together. Also want to thank the SCNUX community members who are here. There's a tutorial today and then a talk tomorrow by Tushan Bharvani. I think securing our systems is becoming a lot more important in today's day and time. So I hope you take the most out of this session and hopefully have more secure systems. And also want to thank all the communities associated with Red Hat. And I want to thank the PBSD Foundation for coming here and putting the agenda together for educating the audience about PBSD. So thank you so much for these communities. And on that note, I do want to mention that GroupCon this year has turned into a platform where it's no longer just about GroupCon. It's about all these communities coming together and GroupCon becoming the space for everybody to come and interact with each other. Given that, I'd like to invite Satish Mohan to very quickly tell us about DevCon and what is the future roadmap. And with that note, we will take on with Aditya. After Aditya session, we will have a very quick demo on the Contact Point app. You have these QR codes on your badges. We will quickly show you how to scan them and make the most of it because you can exchange contact details. And last but not the least, before Satish takes over, I hope you have a copy of GroupCon's cheat sheet. If you don't have one, please pick it up from the help desk. This has all the important information that you require including the Wi-Fi, SSD and the password. And other things including food tokens and how the systems work. And there's a further party this evening. So there'll be transport from here to the party venue. So definitely be there at 620 downstairs. There'll be various tempos going from here to the party venue. On that note, Satish, thanks for having DevCon here and very quickly about DevCon. Thank you, Sainath. Definitely, everybody noticed that the party... Yeah, is this? Okay. So yeah, so it's great to be here. So before I get into the DevCon, I think the moment I came into this conference, I met a lot of old friends and I see a lot of new faces that kind of... I can validate Sainath's point that I get, this has become a platform. This GroupCon, it's not just a conference, it's a platform where we talk about not just infrastructure pieces, but it's also about the application design patterns and the deployment patterns. So a brief on DevCon. This is a conference we started way back in 2009 in Burma, and it has more of a conference that focused on infrastructure technology. So all the community members who have been working with Fedora, just to come together and kind of discuss about the roadmaps and also talk about the features and the progress they make and also take suggestions from other community developers in terms of how these pieces will fit in together. So that was kind of the starting point for DevCon. And it has evolved since then, it has evolved into a platform again, which focus on application designs and designs and deployment patterns, and it brings in a lot of topics. So we just saw from the first keynote that even one particular segment in terms of monitoring has so many projects and with the new design patterns, every layer we add into has got so many different designs, so many different modules coming in there. So our objective with this conference is to make sure that we create a forum where the developers and the users can meet and discuss and that can cut short the time that we're going to deploy an application. So it's more in terms of like, learn from each other. So even as developers who is developing technologies, also would like to get the feedback saying that what next, they should prioritize or what they should bring in or what course correction they need to make in a particular stack. So that's the real objective. And when we had this thought of bringing this to India, we were looking for partners in terms of where, how should we host it and where should we start this? And then Hasgeek and Linux Foundation has been great partners and we found the root point is the great starting point to have this conference. So we're looking for a lot of conversations. So there's a lot of workshop and talks being designed and for the next few days. And a lot of people has put in efforts, not just the speakers, but also a lot of community members in helping us figuring out what are the important topics to pick up and one of the things the root cause audience would like to hear about us. But most importantly, the other objective we have with this initiative is also to increase contributions from this region, contribution to the existing projects or starting new projects as an outcome of the conversations we have in this forum. So that's the larger objective we have this conference on. So a lot of topics to cover in the next few days. So I hope you guys will have fun and I'm not taking too much time in your coffee break. So enjoy the coffee and then on to the next session. Thank you. All right. Our next speaker is a native of Bangalore, local guy. He is an expert in DevOps and Systems Engineering. He runs a consulting company here in Bangalore. So please welcome Aditya Patawari with the deployment strategies for Kubernetes. All right. Good morning. We have quite a sleepy audience and actually do help in sleeping. So one of the things that I really liked about this, okay. So one of the things that I really liked about this root corp is that there's a rope line there. I'm not really sure why it is a big rope bundle. I don't know what they're planning to do with it. But I thought that it's nice to point it out. All right. So I'm going to talk about Kubernetes today. And we are going to look at deploying deployment strategies, some very common deployment strategies. A little bit about me. I am a systems engineer and DevOps since 2011. And I led systems engineering team at BrowserStack before this. Currently I am running my own system engineering and DevOps consultancy called DevOps Nexus. I am a contributor to open source projects, including Kubernetes and Fedora project. And I've been an author, a published author and speaker at various conferences, including root corp, FOSTOM, Flock and FOSAsia. That being said, I actually lied when I told you that I'm going to talk to you about deployment strategies. That was my ploy to get into the schedule because they were not letting me inside. So I thought that if I put deployment, they'll allow me. So that was a very clever way to fool the editors. Right, so what I'm going to do is I'm going to talk about certain concepts of Kubernetes that will help you in deployment. They'll help you in designing your own deployment strategies. So, and by understanding these concepts like labels and labels and schedules and all those things, I want to learn deployment as a bike product. Right, before we go ahead, we need to have some sort of initial setup. I will basically need Kubernetes cluster setup and I need an image with which I will demonstrate how this will work. Now, I'm going to use the nginx image because that's pretty much very common thing to use. I request you not to follow the demo right now because that will screw the internet and consequently my demo will not work that nice. So please don't try to download the nginx image right away. So with that, I would probably want to introduce you to labels. How many of you here are familiar with Kubernetes? Or at least have heard what Kubernetes is, right? How many of you have used any cloud provider out there? Any one of them, right? Quite a few. In any cloud provider, there's a way to identify resources and the most common way to identify resources is to use something called label or tag depending upon what your cloud provider is calling them, right? So Kubernetes has a similar thing known as labels. Basically you pick any artifact like a pod or a deployment or a service and then you assign a label to it. And then subsequently you can identify that particular resource using that label. Now we are going to use this to our advantage. So basically when we want to do anything of the sort of deployment or routing traffic or whatever, we need to make sure that we have right labels in place. So I'm going to use quite a few labels here. Most common ones I've written down, like we have a label for environment, for application, for service. The color label is something which I will talk a bit more about later, but yeah, that's another label. Second thing I want you to realize is to imagine how your traffic flows from a client's machine to the service that you want. So we started at the very beginning. Somebody sends a gate request or something like that and it hits the load balancer, right? Now what happens to the traffic at the load balancer? Load balancer basically acts as a central brain. It tries to understand that where should I direct this traffic to, right? And the load balancer or the router or the service that is supposed to find out the resource for which the request is meant and then send the traffic there and then that resource or that particular server is supposed to take care of the processing of the request. Now this entire thing is quite standard. We have all have been doing this. If you have worked with any of the cloud provider, be it AWS, GCP, DigitalOcean, all of them have their own load balancers and or if you have used HAProxy, all of them work on almost the same principle. Now that we know we have this key concept in mind that we can actually tweak the routing of our traffic. Let's now start with the deployment strategy that we want, okay? So this is the pretty much, this is pretty much standard. So before I go ahead, let me bring up the initial cluster, which I'm gonna use, right? So I have two machines with me. This one, if you notice, this is the Kubernetes master. On that side is Kubernetes node, okay? I'll keep on toggling between two, but I'll make sure that the one which I'm writing to is maximized. However, you can always see that master is written here and node is written there. Kubernetes, if those who are not familiar, it's a very standard client server architecture. There is a master which passes, which against which you pass commands and there's a node which actually processes the commands in the sense that it is the one which is responsible to run the containers. Just to get a hang of it, let's see, I have one node ready with me, all right? Now, if you look at the initial setup, we have a two node setup, one master and one node, and I'm gonna use nginx 179 for this demo. So I'm going to create a deployment and a service. A deployment basically will create the Docker containers onto the node, all right? And a service is responsible to route the traffic from the user to one of these two containers that will be created. Let's quickly look at the things that we have here. I have a initial deployment with me, right? So I'm basically starting two replicas of nginx 179. Now, interesting thing to note here is that I have applied certain labels to it. There's an app label, there's an environment label, and there's a color label. Please ignore the color label for now, we'll come to it later. But most interesting label here is the environment in the app right now. So let's go ahead and create containers. And just to make sure that we don't have any residual thing running here, if I do a Docker PS, yeah, there's nothing running here. So I'll just create the initial deployment, it's created. Let me also show you the service that we have here. So this is my service. Now, if you realize that we actually tagged our resource, I'm going to use the same tag, environment production. And what this service is going to do for me is that it's going to receive the traffic on port 80 and it'll direct it to any resource that has this particular tag to it, right? So all the pods that I have, all the containers that I have with this particular production and blue tag, they're going to receive the traffic. I'll just create it quickly. When I, once I create the service, I get a endpoint. Basically just going to copy this and write. So I'm getting served with NGINX 179 here. This is my initial setup. Right now I've not done any deployments. This is my, you know, the first stage. Now going back to presentation, the most common thing that you do when you start playing with any application or when you want to do an upgrade is to test, right? Before deployment, you always test. And one of the most common ways to test is to do a canary deployment. Now canary deployment means that you basically route a very small portion of your traffic to a canary. I mean, very small portion of a traffic, production traffic to something which you want to update to. So what we are going to do is we are going to route a part of traffic. Now, can anyone suggest a way to do that? Like we have discussed that there are ways to route our traffic using labels, right? So can anyone here tell me a very good way to probably route the traffic easily, but not the entire traffic. Only a portion of the traffic to a different machine or to a different container. Any ideas? Is it too early in the morning? Okay, too early in the morning I guess then. Okay, so what I'm going to do is I'm going to create another deployment just like I created NGINX 179 deployment. But I'm going to make sure that I use the same tags. I use the same labels. What will happen because of that is the selector which my services use. If the label is same, then it's going to route a proportionate amount of traffic to the new set of machine as well, right? So let's say if you have a cluster of probably 10 or 100 nodes, if you just want a canary of one node, then have a deployment, a single deployment, a deployment of single node where you can basically, is it visible? Should I, is the font small? Good morning. I increased the form before coming but I didn't realize that it'll be still smaller. Is it visible now? I'll move it up a bit. Is it visible now? All right. So now what I'm trying to do is I'm basically going to run a canary deployment with one replica. Now for me it's 33% of my infrastructure because I already have two. I'll add one more, that's 33%. But if you want to do a real canary in production kind of set up, then probably make it less than 5%. Um, I'm going to update the image. I was using nginx 179. Now I'm going to make it 191. Thing you need to notice here is that I have added a few tags here, like type is added, but remember the selectors that the service had? Service had two selectors and that was environment production and color blue. Service does not care what part it's sending traffic to as long as the selectors and labels match. So you should take advantage of that concept and basically route a part of your traffic here. Before I do that, wait. Let me just show you. I'm just going to fire about a thousand requests to see what happens, right? It's all nginx 179. Now I'm going to create the canary. Right. So my canary is created. It'll take about a second to get created. Uh, if I fire the request now, I should actually see certain occurrences of 191, right? Can you observe that? In fact, we can probably go a bit ahead. It's going to fire the request in background and put a count that 328, which is approximately 33% of what we were doing, right? So this will give you a very good way to do canary deployments and test out the release before actually putting it production wide, right? So this is what we do. Uh, we just realized that services realize where the labels are and then send the traffic across. Now, once you're done with this, the next step is rolling deployment, basically deploying to the entire cluster. And that is slightly easier because Kubernetes supports rolling deployment out of box. So you don't need to play a lot around with labels and all. So I'm just going to quickly show you that. Um, but to show you that, to show you the rolling deployment concept, I need to upgrade the size of my cluster because right now we just have two or three machines running, uh, which might not be very helpful. So I'm just going to increase the size of my cluster to six. Right, now I have a lot of machines running. And what I'm going to do next is just set the image to the new one that I want. All right. As soon as I do that, the rollout will happen and the rollout is like this. So if you look at it, not everything is replaced right now, right? Only four of them were replaced. Fifth one got replaced. Now it's trying to terminate some of the older ones. So it's, it basically rolls to ensure that your, your users will not see a downtime. Not entire cluster will be taken out in one shot. It'll be rolled. Um, so while your users will not see a downtime, you have to understand that there might be some latency issues there. So when you actually do this, keep that in mind that your users will not see a downtime, but might see slightly degraded performance. So best is to either increase your cluster size like I did before doing rolling deployment or do it at a time when you know that the customers are, you know, you have very few customers around. So a rolling deployment will not actually hurt their experience. Rolling deployment actually has a lot more things around, which I quickly want you to show. For example, if you want to check out the revision history. Now there is an open bug because of which this is none. Uh, but what you can do here is that you can actually go ahead and check the revision history by revision number. If you do that, you'll actually see that there was a previous version with 179 deployed. Okay. And if you think that, you know, things are not working your way and you want to roll over, you want to, uh, go to a previous version, then that's easy as well because what you need to do is just do a rollout undo and that's going to put you on a previous version. Uh, so instead of getting 191 here, you will get a 179 back. Okay. So that's basically rolling deployment. Now. Lastly, I want to talk to you about blue green deployment. This is, uh, this is a different case which doesn't fall into rolling. Uh, what you do in blue green is that you maintain two sets of separate clusters. One is blue. One is green. That's why I use the labels color if you notice. Uh, what happens with them is that you say that my traffic is going to go to blue, but as soon as you are, you want to update, you set a new cluster called green and when you're satisfied, your tests are done on green. You basically route your entire traffic to green. Now, this is also very helpful if you are into immutable architectures where you don't want to, uh, you know, change what has been done once. So in that case, you probably create a new cluster, do your tests on that and just start the traffic. So what I'm going to do is I'm going to right now we are on, uh, 179. So let me just going to delete the canary. You don't need it anymore. Uh, once I delete the canary, I am going to create a green deployment. Now green is basically one nine one. So if you look here, whenever you hit this, you're going to get one seven nine because that is what the default is right now. That's what we're that's the blue and that's where our traffic is going. So now to get our traffic to the greens green cluster, what do we need to do? Any ideas? Basically, we need to edit our services instead of the instead of selector blue, we'll use the selector green and that should do the trick. Now Kubernetes give you facility to edit a live artifact. So now I'm going to do just that. I'm going to edit it live, which means that I'm just going to go here. These are the selectors that we have been using color environment. I'm going to set it to green now and as soon as I save this, my traffic will start going to green. Now, if I go here, I should see one nine one. Yeah. All right. So my traffic is now going to the green cluster. I can take down the blue cluster as in when the pending requests are served. I can do that. So with that being said, that's all I have on clustering and labels and Kubernetes. Do you guys have any questions? No. Yeah. Yeah. While changing the from blue to green as we are humans, what if you make a mistake? Is there a way to handle in Kubernetes? No. If you make a mistake, then your traffic will go here. It's generally a bad idea to make them. Okay. What I did here manually is probably something that you should not do manually. This is done for the demo, but ideally you should probably fire commands. So for example, if you notice when I fired, when I changed the set image to one nine one while doing rollout deploy, I could have done that using edit also. I could have changed it live and it would have if I would have made a mistake, it would have failed. So using a continuous integration solution like probably Jenkins or something and having those commands in place rather than firing it manually. That usually helps. And I do not recommend you to while I'm saying that it's possible doing it live. I do not certainly recommend doing it like this. This is a demo and this is just a way to showcase the features. But yes, ideally you should use a continuous integration or a system like Jenkins to make these changes or a deployment suit to make this that slightly peculiar because okay, what container is basically it's the same process running in a different name space and with its own isolation. It doesn't really, that's actually an interesting use case. I did not realize that that happened, but have you tried using Java restrictors like XMX and all those things? So Java is not honoring that as well? Okay. Yeah, that has to be looked at. I'm not entirely sure. My name is Satish. Suppose actually we have actually deployed here. Yes. Suppose if we deployed actually microservices and multiple containers instead of actually rooting actually these microservices they want to talk internally, how we can actually make sure that it's not hitting the load balancer so that containers can talk each other in the Kubernetes world? You're saying that you want containers to talk to each other without involving any sort of service or load balancer? Yes. I would probably not recommend that. I will tell you how, but I would not recommend that. I would want you to use something like Kubernetes services because that is helpful in case one of your containers decide to fail on you. That being said, if you really want to do that, pick any overlay network that you like. Flannel is very useful in that because I mean you have to pick an overlay as well as you have to pick a DNS add-on because if you pick just the overlay network, then you will have to memorize IP addresses. Your applications need to know the IP addresses which is not always advisable because IP addresses change. If you pick overlay along with the DNS add-on, then you just need to know the name and the container should be able to hit each other directly. Ideally it should. I'm not sure how your system is set up, but ideally, see load balancer, something like service is actually very lightweight. It's mostly by default it's based on IP tables. So basically you're just routing the traffic using IP tables and it normally works very well with very minimum overhead. I have never seen a delay of even like 2-3 milliseconds. Something that doesn't even add a couple of milliseconds is I think a worthy option if it increases the reliability of your application. I can... Yeah. Okay, cool. It does. Okay, so blue-green means that you actually... I mean, if you talk about the normal, the by-book definition blue-green, then you have to have enough capacity that you can run two clusters parallelly. Now, I know for most of the organization, it's a waste of money. For me, it's a waste of money. If you are on a provider like AWS or Google Cloud or Digital Ocean or something like that, then it's usually not a problem because you basically just pay extra for like NR or something like that, whatever they bill you for. And that's not a big cost. But if you are with a data center provider, if you have your own physical hardware that you're managing, then it becomes a cost issue. And yes, then it will be a problem. Rolling blue-green or a hybrid of rolling and blue-green, that helps. But technically speaking, that's not exactly blue-green because at some point in time, your capacity will be compromised when you are basically in the transition state. Compromise in the sense that it will be reduced when you are in the process of rolling out. Either you have to reduce the capacity or you have to make sure that both of them... that the users are getting served from both of them simultaneously. So you have to handle either that situation or reduce capacity situation. It's your pick. What do you want to handle? Last question. You are very... Oh, okay. Right, right. Right. Yes. Yes, yes. Hmm. It can handle it to some extent. If you are hitting your CPU, then Kubernetes can scale automatically. There are auto scalers available which work out of the box inbuilt in Kubernetes. That's one way. I personally have found it to be limited because it does not cater to all the parameters that I want to auto scale it on. So I sometime back for a client, I ended up writing a custom solution which will hook to Griffite and then scale... because scaling is just a command by the way there are Kubernetes APIs. So you can use the API, REST API as well and hit it directly instead of using command line. So what I would probably recommend you if you are just looking for very basic scaling, just CPU based scaling, then you have Kubernetes auto scaling group. You can look at that. If you want something more advanced, if you have more parameters to consider, then you'll have to get a little bit of hands-on with the code and API. It's not too difficult. Just have your data shop to Griffite or whatever a graphing engine you have where you realize that these minutes are being received and so on. And based on that, you can call the Kubernetes API to scale up and down whenever you want. So usually it takes a little time to understand but coding it is not very difficult. You can do that. If you are on data centers, then I mean you are not on a cloud provider.  Okay. The reason why cloud provider came in picture and gain popularity was because of this because scaling with the hosting product is very difficult. And I don't think Kubernetes or any other tool will scale hardware for you for hardware. You have to do a capacity plan. No, it will direct the request. Your application will time out. Is the standard thing it forget Kubernetes or forget any containerization or anything like that. If you have a process running and if you bombard it with if you have a web server running, if you bombard it with a request at some point of time, it's going to time out for you or for whatever customers. It might be a serving a partial partial number of records, but obviously for certain customers that will time out. That's not a Kubernetes thing. That's your application. That's your application. All right. So lights are on means that I'm sure off from the stage. Okay. Thank you everyone. Before we break for our morning and beverage break, which is coming up just next, I'd like to take a moment to talk about something that we may not think about very often in our jobs. We're always very focused on using the computer and technology and DevOps as an occupation very stressful. We sometimes don't stop to think about our own physical health as well. And has geek has earlier this year had a conference called kilter where we're thinking about physical health and fitness. And today as well, we're offering some activities for people who may want to step outside of their brain and more into their body. This afternoon at 3.30 in the lawns outside, there will be yoga and slack lining for something completely different, turn around 180 degrees and try something else. So if that sounds interesting or if you need to move your body, please by all means have a look at that. And also this evening, we'll be having a talk at the banquet on sleep and its effects on the brain. So for anyone who's interested in knowing more about this, I encourage you to make sure you don't miss the talk. So thanks everyone. Enjoy your coffee and chai. Hello.  I have a couple of announcements from our sponsors. You will have a chance to win exciting prizes. And I'm sure that appeals to everyone. So from widest concepts, you can win exciting prizes from the DevOps quiz. So have a look for the widest concepts of both outside in the vendor area. In addition, there is a digital ocean spin and win contest in which you can win a one terabyte portable external hard drive. That's one trillion bytes. Is that right? Yes. So excitement. Also, there will be a flash talk session today at 5.20 p.m. If anyone would like to deliver a flash talk, it's not too late. You can sign up at the talk funnel online or you can look for the whiteboard in the seating area at the box office. So please, if anyone has an idea for a flash talk, they would like to deliver. Come one, come all, put your name on the board. And you can give the talk in the session at 5.20 p.m. today. I'd also like to remind everyone that in your bag is a feedback form for the conference. We would really love feedback. In fact, we use the feedback to make each conference better every year. So please fill in the feedback form. It will only take a few minutes and it really makes things much better for us. You can drop your completed feedback forms in the laundry bags placed outside the auditorium. Very easy. Just drop it off. Please only put the feedback forms, no coffee cups or anything else in the bags. Or don't put your laundry as well. All right. I'd like to introduce our next speaker who has come all the way from Rajasthan, the land of kings and colors. But she's not picky about that. She'd prefer to be known as she is from India. So let's see. What else can I tell you about Kruja? I've taken some notes here. She'll be giving a talk starting the automation segment of our talks today. So she's talking about a little bot. So we'll find out what that's all about. And she's constantly striving to learn everything she can and share her knowledge with others. So that's part of her mission today. So without further ado, I give you Pooja Shah. Well, I really want to thank you for joining me here. I am Pooja. I work with Mo Engage. A brief about me is not so brief. After years of trying to find out about myself, I figured out one word which explains me more is I'm an explorer. I have explored a lot of things. But I'm yet one of you who still feels that I have a lot to explore more. In work, I had been a developer, a QA, an automation engineer, and now trying to double with DevOps and take-offs term. In personal, I am a foodie and sleep lover, but loving yoga makes me in a situation when nobody can find out that these are my hobbies. Coming to the work, I am... Coming to the work, sorry. I work with Mo Engage and at work, I try to bridge the gaps between all of my teams and with that mindset, I lead a team QA and currently actively focusing on building automated systems which can improve the collaboration across the teams and help us build, shape the healthier product. Along with the work, I am an open-source lover. With that interest, whichever project I use for my work, I make sure to contribute back to it. My recent contributions can be found in Jenkins and automation-related plugins. Coming back to the talk today, I am going to introduce you to one of our special team members. Her name is Alice and I can see some places glowing. Sorry for giving you hopes up. She is a bot. Yes, so we are going to go on a small, crisp ride to meet Alice, starting from Y, Alice to Bert to what all it can do and how a part of it, the ingredients and recipes, how we cook it, followed by the demo and QA at the end. So are you guys with me so far? Alice needs more voices. Yes, we are going to start with Y. I take you to one of our typical release days conversations. So just try to recall your release days to understand it better. It's nine o'clock in the morning. Ten o'clock you have decided to release it and all of a sudden a road blocker comes. So here is one of our team members named Karna who is coming running. Hey, Draupati, something is broken and you are about to release. Can you just figure it out? And here is Draupati, who is one of the QA member who was super sure last night that it was working and she's like, no, it cannot be. And she panics and says that, okay, let's try to talk to Beam who was the developer of that feature. Beam, on the other hand, has the general notion because, hey, I God swear, I did not touch this from a year now. How can it be me? Right?  Try to find the buggy commit and let's revert that commit and move ahead. Now, Ajna has to come and do the dharma and the dharma is go to the GitHub and try to find out all full requests and find the commit which was buggy. It is going to take time. A lot of full requests have been merged now. Seeing this, Vishma Mitama, who is the developer, Vishma Mitama, who is the developer, gets the clear confusion like, oh, no, we cannot release now because it's going to be peak hours for customers. So let's just postpone the release and you know now, this raster being held place, they have to say, no, we cannot postpone, we have planned and we have committed the lines. What do you do? I want to know how many of you feel the same pain which I just described? Hands are raising, thinking that bosses are not sitting around. Yeah, yes. Same thing happened and if that happens, what do you feel end up with seeing this situation? What do you call this? Blame game. We don't like to use negative words but we say that some kind of conflict wherein people are not able to move ahead because of so and so whatever reason is and when that happens to you, what do you want to do immediately? And you feel, no, there should be some way. I also felt that and that's where we figured out what all problem we are seeing are just the symptoms. The problems, the root causes are beneath and we wanted to really work on the black area which we see here, the black area but the problems which we were not able to figure out till then and that's where I take you, from now I'll take you to the problems one by one and the solutions which Alice brings in but before that let's understand how a typical code flow happens in our organization to give you a better understanding. So here is our developers who write code in their own branches and pushes to GitHub and from there there is a pull request which goes to the developer branch from there there is pull request to our test branch we call it QA in our context and from there we decide that this is the code freeze and QA team will be testing only on this so that no confusion happens and when everything goes right here we move it to master and we take the release from here. So what we figured out is the root causes were within this area this area which we called it as a sensitive area and all the branches within we call it sensitive branches with their test and master so we figured out if we could put some kind of way to monitor this we could resolve those problems. So to look at this problem again I went back the developer inside me told that he was right he did not touch from in here now should he leave his current work and check that yes we don't want to lose the productivity and as a QA I feel the same whoever bug it is I just want a solution to take it ahead and same as the DevOps and finally the automation engine inside me comes out in all this and say that hey this communication problem is also repetitive in nature it's happening every release can we do automation around it can we do automation of monitoring this sensitive branches I take you from here the how part what are the problems and how it solves it. So for example the very first problem the last moment panic attack comes because of we are somehow taking the unreviewed commits inside the system and that's where Alice says that hey do not let them go inside. So at present scenario GitHub has launched the approval and review review feature of it but when we started it was not there so we started with this and still it has its own meaning I'll take you to that in demo section again. So whenever such thing happens whenever somebody merges the code without the code review that Alice does is it auto reverts it immediately informing people. So we are using for communication we are using Slack and for code collaboration we are using GitHub so we Alice knows and parses and finds out what is wrong and reverts back. Second problem no quick record so we saw that that was Arjun who has to go through all the PRs and find out and it was him all the time. So could we make it more understood by every member in the team and that's where we decided we'll have a permanent channel for each repository we have and Alice keeps recording for all the events happening in the sensitive branches. So for us merge was the sensitive event and the branches were QA, Deo and master so Alice does this. So looking at this we can come to know if let's say I'm a QA engineer and I have to approach if this feature was working two hours back and it doesn't work anymore now I do not need to just alert channel hey everyone check this. I have understanding okay these two request probably I can check and who is the author I can actually discuss with them and it eases up everybody's task and in that sense so whoever has committed within that timeline they can see and quickly fix this. Next problem no danger boat. So mistakes are prone to happen accident can happen if you don't put the danger boats and that's where Alice said hey be proactive and put the danger boats. For example this feature we introduced for DevOps especially so we found that there was some machine level DB level changes which people were doing and missing for getting to inform DevOps team and that's where we decided okay how to not repeat it and one of the example is this. So wherever some sensitive files getting touched by someone in one of those sensitive branches Alice comes and rewards informs respect to DevOps team hey this is being changed so if DevOps has to check that and whatever they want to do they can do it afterwards. One of those example is this also. So we said that from we will take release only from the QA branch not from the any other branch. So we Alice we in the same Alice does is what wherever somebody creates a pull request from some other branch random branch to master Alice autocrosses it immediately. So we are saving our time of mistaken emerging wrong PR and then reverting the code again. Lack of awareness so this is one of the one where Alice got more feathers in its hat is Alice says that hey people have tendency to forget how to keep reminding them and that's where it says keep me posted. So Alice posts for multiple things for example Alice the moment you as a developer merge some code into some special branch Alice reverse hey you have merged this and so and so in this now be nice and just mention it the release notes. So it helps QA team to derive the bandwidth when they can release and others also have the clarity of the things. Next is this so whenever we are about to release we want to inform all this is auto alert Alice Alice and say we are about to release especially to tech leads if something is missing or something they want to add up just they can they are aware. This is one of very interesting things I myself has done this mistake a lot of times. We took release live without J's update and we all know what happens so it was for five minutes our UI was not afflicted and people are trying to find out what went wrong was just the forgetfulness. So we decided okay when there was somebody merges to the release or master branch Alice immediately pinged hey did you fill this checklist if this checklist is not filled they won't take the release live or whoever the release engineer is no automatic guide. So we were around 20 engineers when I joined Moingate and we exponentially increased to around 40 engineers and telling everybody educating everybody about the process was becoming tedious task for me and that's where I saw a full potential of including automated guide. So bringing the talking bot so I'll come to the take inside how we are doing. So you just ask Alice about the system it knows everything about that system from static replies to dynamic to giving it a task for example hey Alice give me release notes it's reverse hey Alice how do I take my patch live it reverse the instructions what do you need to do hey Alice dynamic hey Alice start my machine it goes and triggers whoever is responsible to start the machine hey Alice get me the branch name so if I'm a product manager or somebody else who do not want to go into all details of SSH to machine and finding out what code is being applied that's very handy for that person Alice reports back now coming to the take behind it's no magic it was all here but what it took me to do it was connect the start there were github APIs, slag APIs, report APIs and Jenkins so I really want to appreciate Jenkins community completely because all the ideas I got from there to automate things and how we did this so we have github repository and each repository has a webhook mechanism from github wherein whenever you merge a pull request there is an event triggered and this event triggers Alice so there is a full payload of there is a full payload of pull request which Alice parses and says does the particular action, the business logic business logic can be pushing back to Slack business logic can be talking to talking bot, you bot and Jenkins so these all are interconnected to do whatever whoever is best at doing what task they are just interconnected with the main source called Alice so let's have a look quickly have a look at demo so for simplicity I have kept one is it visible? now somehow not increasing the laptop no it was a joke I had a demo recorded so coming back so I have one commit ready and I say this I create a commit root con live from root con 17 and I say create pull request the moment I do Alice does multiple tasks do you see immediately the guideline has come the respective person has to do now people will have conflict this we can do in the contribution probably with GitHub only but I'll come to it also why we needed this because for different branches we wanted to do different checklist the moment you see this there is a Slack channel for simplicity I have created a Slack channel to show you so here is a repo code it has got the entry hey this is being in this repo this is being opened by this etc etc and if you see if you see this sorry so if you see this I have created a change in the file which has a rule of if the file is changed product plus one is essential and whichever that rule says that if this is being merged without the product approval this should be either rejected or being informed so I merge this commit let's see what happens in the channel you see it says that hey this is merged without the product plus one in one go I got this detail and there is one more entry in the repo code which we saw the common place to track back it gives the entry okay this is being merged so now Alice has all the trace back so if a person wants to know what were being merged they will come to this channel and release channel is especially to inform to take actions if it has to be reverted back or automatically revert back so you can set the rules or deselect the rules in Alice for that purpose now commit to talking about let's talk to Alice I can ask Alice I can invite Alice as a user in my Slack channel I can say that no it's not Alice it's not Alice I can say hey Alice that's life so if it knows it rewards if it doesn't know it say I don't know and hey Alice you can say what is branch on my one more for my machine it goes to that machine this is a dynamic reply it goes to that machine and it reverts back and if you feel that this is too much noise in channel you can directly talk to Alice you can directly ask Alice hey of hey Alice when is the next release these are the dynamic feeder data which one of the QA teams probably gets feed or the release team can feed and you can go more in-depth and say hey and what are the release items it goes and finds out whatever is being feeded either static or in a file wherever automatically you can write from the merge event you can do whatever you wish to so and if you're getting bored you can just ask hey get me a cup of coffee so things like that so you the idea is you can do you can do static dynamic or sort of things coming back to the talk yes so we saw that it can solve multiple problems so we can control and say that hey Alice do this so I saw DevOps engineers especially having a trouble like my morning 5 o'clock or 4 o'clock they have to wake up and log into laptop and deploy some script on some machine run some scripts this can be very handy we can just feed in okay go to this machine and do that you can connect the Jenkins or other CIE things with it and at the same time you also want to ensure that nobody else can do it in Alice you can do those authentication you can say that hey do only if it's me and you can also say hey Alice be more intelligent for example I have seen DevOps team suffering this like not suffering having this question again is my release deployed on that particular server or what are the all release on that particular server you can just have this kind of setup critically so whenever release is going from whatever branch that push event you can save the data from there whenever machine sorry whichever machine the release is going DevOps have that their own way of scripting finding out all the server details and that can be pushed in one file which Alice can keep reverting each release and this is one of the interesting thing which we did because I did not want to be a grandmother so I didn't want to tell everybody every time that this is a code phrase we are not going to take more items out of this except for things we did we fitted in Alice whenever code phrase happens from test branch it automatically reverts it has all the entries what is being merged during that period and it gives the quick details people can see okay if my code there is getting tested and QA teams can plan their bandwidth future so what do you think what can be the future apart from this whatever is written it remains right there is no single word you can code whatever you want to but what immediate future I am seeing is the continuous integration the reason is simple because continuous integration requires all is the the code commit data and Alice already have that all it needs to do is run some checks and move it to the next staging server so that is what I am working on probably will be there code right soon it is coming soon and some people are like you are going to be kidding yes I am kidding it is not coming soon it is already there we have open sourced it and you can try it and if you like star it, fork it contribute back to it I feel I would love to know more stories around what problems Alice solved for you and with that I want to give credit to a lot of people mostly the problems at work and going to the coming to the APIs and the photo credits also I want to give to the unsplash Google NTP sorry to summarize this we started with solving one problem that was people were merging the unreviewed comments we started with one problem and gradually we learn in the process that Alice could do much more behind beyond that now I hope you have idea about solving your own specific problems in a more automated way and an idea how the future Alice will look like with that I appreciate your time thank you Q&A I can take few questions hi alright so I saw your checklist for developers before the merge is done right I know the developers mindset is I want it to be merged right now just click the box and merge it now how long have you implemented this checklist how effective has it been yes so the checklist we started using for our own the first checklist which I shown in the slides if you remember that was for the release checklist that's for sure we are taking care of it because we already did that mistake right so for release checklist it's not a problem everybody follows it because then releasing will go wrong now developer checklist developer checklist yes it is hard to implement but it's like it's like there are people who actually do not want to miss things so for them it is very beneficial and for whom who after seeing also do not want to do that's where continuous integration is coming in so we do not allow the we do not enable the merge button itself until those things are done was I able to answer this actually the thing is we have one bot called bot so we have integrated with the Jenkins but the thing is my slack user and the Jenkins user might be different so I will control the credential it's simple so I created a Jenkins what do you want to solve with that what I want to control is like we have production deploy to Jenkins so I want to lock we are triggering the build so slack user might be the different one and the same user might have the different username for the Jenkins so how can you think of that so the answer is yes or no you want to do that or not so for us if I can create same name as Alice nobody gets to know about it but what is the slack user yeah internal authentication when you want to do for that users whatever user you have created you can do for that there is no need to make everybody everywhere Alice you just can name Alice internally for that specific user token you can authenticate okay only this user token should be allowed to do perform particular task but then you will not able to lock that whoever is triggering the build suppose I have a bot anyone can send the message to the bot so I have to lock it in the Alice not in the Jenkins so what if I want to do it at the Jenkins level Jenkins level so everything whatever message so Jenkins triggers a call sorry Alice trigger a call to Jenkins to perform certain task and in Jenkins in post build we have written reverse back to Slack so we know that who did what it's never open ended it's never open ended just do this in Jenkins we have always written a post Slack call to reverse back in saying that hey I did this on this particular server so people get to know it's not like somebody personally talks to Alice and do something no so that will be our handling can you help me a little more to understand this question what do you mean by phases here okay okay I get it I get it so so with HuBot one thing is when you interact via Alice through HuBot HuBot has a scripting it's called coffee script it's just like a JavaScript and reach in all you need to write is more of a regular expressions there so probably it's like the more rich regular expressions you write it becomes more natural language there so people do not need to mug up or remember commands they will just say plain English I want to start this machine then Alice will understand okay which command to hit against it people will come with plain completely plain English you just need to say in whoever joins new in you just need to say hey just talk to Alice they will figure out so that's how I have improved in writing my own regex I used to give to people so that's where it came up with more natural style of understanding okay so we're cutting down questions speakers will be available outside you can talk to them later and we'll have a QA session as well thank you all thank you for timing I'm all around here I would love to chat if you have something in mind and some ideas more you have to implement in Alice come talk to me thank you thank you Pooja I'd like to just take this moment to remind everyone and especially the people who came in late from the coffee break please do remember to fill in your feedback forms we rely on the feedback from the participants to make the conferences better each year and you can put your completed feedback form in the bags outside the auditorium for collection I would like to encourage everyone to do a quick stretch move your body just a little bit so we can stay awake take a deep breath roll your head around and then I'm going to turn over to our dear friend Karthik who's going to do a demonstration of a wonderful technology called contact point this is you may have noticed these little codes on your badge there's a secret hidden inside there hi everybody okay so this is a quick two minute demo so this is a really small app on the Google paster it's not on iOS yet we are open to contributors for this project if you want so it's a simple app that right now just gives you the ability to share contacts that are printed on your badge so so I'll just take you through a quick demo okay so just a quick run through for those of you who haven't downloaded this yet it's got a simple overview of all the things that are happening during the conference so if you're having any announcements sponsor posts and anything else regarding the flash talks we will have them as little announcement boxes here as you can see one of them is the core of conduct for now and you can also go and view the schedule on the schedule tab so all the talks here are so if the badges don't have the updated schedule like the two talks that are switched this one will always be the first one right so for today and tomorrow across all three tracks and the interesting thing is this contacts tab right so if you go to this and then hit the floating action button the plus one here you will have to log in first do that yeah so once you've done that when you hit the plus button it should just open up the scanner this is as simple as scanning the tool yeah so the idea is that you guys will be able to scan each other's contacts and this doesn't save it to your contact list instead you click add and you go back you should see a list of contacts in this contact tab right and you can hit the little export button on top right and this will generate a VCF that you can then share on email or whatever else and you can also tap on the contact if you want and then add it to your phone right and the other thing I want to show you is we also have a Slack team so you guys would have received an email I think yesterday and on Monday if you had bought tickets before that regarding the Slack team so I really encourage all of you to join so I can show this to you on the so if you guys don't have an invite on the first tab is this button called discussion so if you are not part of the Slack team you can just say no send me an invite and then it opens a little pop up where you can give your email and then get invited yeah so this is what the Slack team is like we have a lot of conversations we're just trying to experiment with having an online thing with people at the conference itself and try to see if you guys will break off into smaller groups and have discussions so yeah if you guys have any feedback on this I'd love to hear it I'll be sitting either here or near the registration desk outside over today and tomorrow so if you guys have any feedback on things we can improve in this please let me know and like I said all of this is open source on so you guys are welcome to at this URL you guys are welcome to send patches and come it because our idea is that this is an open app that other conferences can hopefully also adopt in you okay thank you Karthik I'd like to remind everyone let's see a couple of things oh the two sponsors I mentioned with the lucky draws it's not only those two it turns out a number of the sponsors outside at the vendor booths are having lucky draws and quizzes so you can win all sorts of exciting prizes so have a visit to the vendor stalls I'd also like to remind you guys that Hasgeek is doing some workshops this weekend there's three different workshops I believe and there's still a few seats remaining so if you're interested in attending the workshop it's not too late have a look on the rootconf website and have a look at the workshops that are available okay our next speaker is coming from Delhi and he is going to be talking about some more monitoring back to the monitoring part of our talk and we're going to be doing actually a Q&A session with our monitoring authors later on in the session so this is Manan Balala he's coming from Delhi again he has worked both with not-for-profit companies and for-profit companies so he's seen it from both sides today's talk he'll be giving a little bit of experience from his for-profit work on tooling and monitoring for performance critical applications so please welcome Manan Balala to the stage hi everyone just wave if you can hear me at the back awesome alright so thank you for staying back hope you got enough coffee and energizers to keep you going until lunch so this talk is about monitoring my name is Manan and that thing below is the Twitter handle I'd love some feedback on this talk I'm trying to constantly improve it so please if you have any feedback for this talk, write it there and that'll be great so yeah this talk is about monitoring but it's not about monitoring in any generic context it's in a very specific context the context of a large online retail shop website and we'll see how that changes the equation we'll see what sort of a monitoring tool we'll need when it comes to an e-commerce website have you begun regarding I hope so yeah it's about monitoring in an e-commerce context we'll also look at trying to conceptualize the entire monitoring system into phases and we'll see we'll try to divide it into phases phase sort of warrants its own tool there are certain tools out there which cater to more than one phase but hopefully after this talk if you see a tool you'll easily be able to map which phase of monitoring setup does it sort of go in right so first a brief about auto auto is a large online e-commerce retail company it's based out of Germany it's the second largest e-commerce company in Germany Amazon basically surpassed auto in 2014 before which auto used to be the biggest a little bit of history began post world war 2 when this guy decided that he wanted to sell shoes and he took pictures of shoes pasted them on a catalogue photocopied 300 copies hand delivered the catalogue to the local areas and the orders came in he delivered them over time the guy built enough loyalty that today auto is the largest mail order company in the world the second biggest e-commerce company in Germany operates in more than 20 countries the net worth of the auto family is about 18.4 billion and of which about 4 billion comes from retail a big chunk comes from their real estate business as well that is the auto premises this is where I was working with them and so that campus is about 205,000 square meters if you who aren't able to see how big that is it's roughly 35 football fields big like two football fields put together so it's pretty big but when we talk about scale at auto that's definitely not what we're talking about we generally talk about their online shop auto.de which records about a million unique visitors every day and about two orders happening every second and given those statistics which are public statistics we could sort of extrapolate and say that if it takes these screens to sort of place your order that by the way is the home page of auto if you say it's fairly fairly simplistic if you compare it with the flipkart and snap deals that's because they appeal to a very different persona than what flipkart or snap deal sort of appeals to so they like to keep things simple and that's basically their selling point but basically if it takes these screens you pick a product you add it to cart you later on you proceed to check out takes about eight screens to get your order placed and if one in thirty customers sort of convert we can say it's about 480 page impressions happening every second we saw that so auto has their own big billion days and they call boom days translated to English basically in German it's something else but on these days we saw roughly 1,000 page impressions every second by that statistic I'm not really trying to sell that this is large traffic of course there are companies with much larger traffic and facing much more visitor inflow as compared to auto but the thing that makes it unique is so when it comes to e-commerce the thing that makes that unique is because we're talking money there is no other context in which it is more easier to relate downtime with direct losses so we had we had these numbers being thrown at us that okay for a one minute downtime you just lost us 100,000 euros or something like this so it's very easy to sort of connect the loss value so essentially we don't want to do anything that affects the money we're making right so we want to maximize that we want to sort of offer an optimal solution which keeps the inflow going we want to continue building more features there are great features coming in everyone trying to get one up over the other but we don't want to build any features that don't tie back to the original point we want to keep building features which add more value which in turn basically bring in more money we need to make better decisions and we figured out that whatever features we're building can give us enough insights to basically decide how to go about fixing a bug for example to say there's something broke in production and someone coming to you over the top of their head they just say hey this is a massive failure we need to fix this now unless it's backed by real data unless you know how much of your users are affected by it it's really hard to judge what sort of priority to associate with it so it's very important that the monitoring setup we have equips us with an understanding of how big of an impact something has lastly we really want to verify this assumption for this I'm going to take an example let's say we were a little late to the game and we decided that we now need to personalize our e-commerce shop and we said that if we personalize one user out of three we basically expect a revenue increment of x we make this claim and using this claim we're able to sell the feature to the business and we basically are able to begin development so on so forth we develop the feature roll it out and it's great everybody's happy except we never go back and verify whether the claim we made is actually are we performing at that optimal level so are we able to personalize one and third of users and without that it's really just not making any sense so we're developers it's not our job to make up numbers we need to base our numbers on hard facts and that's what our monitoring tool should give us so what do I need from my monitoring setup I need what every other monitoring setup does I mean there's no discounting the fact that you need basic database monitoring we need to know if our queries are not performing well if basically maybe we need to add an index maybe some queries are slow maybe we need to shard and our monitoring system needs to tell us this much we need to monitor standard server metrics every monitoring tool does that we need this too what's our throughput currently how many requests are we serving what's open connection pool count and whatever right and we want to be alerted upon exceptions so this is a no brainer we need all of this what we also realize is that what is really important to us is also to measure the state of the system and while this is a a vague statement to make what this essentially means is we want to grasp how the system is performing not just an infrastructure point of view for example your website may be performing very optimally at 1 a.m. at night but there are no users there so essentially you need to know what's happening right so it becomes important to also track these other metrics and for this we turned over we said that okay let's just ask the product team let's just ask the business teams how they would measure the state of the system and and so this whole all of this agile jargon what it's done is it's broken through those walls now you can communicate and they were equally shocked they were like okay thank you for coming to us yeah we'd like to define our e-commerce system using orders per second and users who are on the website we're like okay these are important metrics we need to keep a track of them and so we realized there's essentially two sides to monitoring there's monitoring of infrastructure which ensures that you're offering a stable and optimal performance and then there's the business side of things which is key performance indicators lastly our monitoring tool should definitely allow us to narrow down when something is broken to narrow down on the source of the bottleneck narrow down on the source of the bug and the thing that helps us here is continuous delivery for example with continuously deploying your code onto production what we've essentially achieved is the marginal difference between the last deployment in this one this change has become smaller and smaller so it's become much easier to track down what exactly broke something and it's it should ideally become that much easier to fix it so our monitoring tool should leverage that right lastly tying into the last point we want to validate whatever business assumptions we set out to achieve are we achieving them or not and that brings me to logging so logging is great right we developers we love logging they point out the exact source of the problem exact line number down to the character and they tell you exactly why the error happened what sort of error happened after the error happened and that's the inherent problem with logging it's inherently reactive when I get a log based alert I sort of feel like I'm on this planet which is about to explode and somehow I need to figure out something that fixes the situation I mean I'm not saying the logs are not good the logs are great right if it weren't for logs I would probably be having beer on the planet like I wouldn't be doing anything on the planet but at least now I know that there's something wrong but I mean what about the time leading up to the point when it actually broke could I have known that this is going to happen so that's why I feel that perhaps if you're building a monitoring system today you don't want to base it on logging you want to base it on something more explicit so one more problem with logging is the signal to noise ratio with the distributed systems microservices the number of systems have grown their logging has grown of course you have great tools splunks kibana elk stack all good very easy to track your logs but for a newcomer to the system it's very hard to gauge in a glance what's happening in the system so even with all these tools the signal to noise ratio with logs is just terrible so I like to define this if logging is about scattered incidents that happen in the system accidents that happen in the system we figured that metrics explicitly collected metrics about the system help you in understanding the state of the system better and they allow you to gain valuable insights but the biggest advantage of explicitly collected metrics is that they give you insights at any time they don't necessarily give you anything after something's broken but leading up to that point they're excellent sources so we went forward with basing our entire monitoring system on metrics and proceeding with that so this is where we actually get to our monitoring system which can be divided into four phases our first phase is the collect phase where you essentially gather the data and I'd just like to begin with the disclaimer we want to gather as much data as we want but only if it is going to give us any useful insight it's very easy to collect all the data out there except you'll be running out of disk and secondly you'll just be increasing your clutter so try and be reasonable about it right so when I spoke about metrics I was talking about this metrics library it's a library written by Kodheil and I think he presented it at a meetup and there's a talk about it called metrics metrics everywhere I strongly urge each one of you to watch that talk if you haven't already it's an excellent introduction to the library and why the library essentially makes sense so metrics is a library that you plug into your system and it runs with your code you instrument it in code and what you're able to achieve is to collect data while your code is running in production and that's its biggest selling point so ours was a closure project but the metrics port exists in there's a Java there's a Scala port and I believe there's similar solutions available in most languages but yeah we figured that it was as easy as just including the dependency and then there were three parts to it so first you have to create a registry a registry is nothing but a container for your system so if you have a microservices based architecture each microservice can have its own metrics in its own container next you just create what sort of tool are you going to use from the metrics toolkit it gives you five tools we'll be discussing each of them in just a little brief but essentially in this case we're using a counter and a counter basically is good when you control when something is increased, decreased, opened or closed so a counter serves a good purpose here and finally it's all about just instrumenting it in your code just increment the counter where you see something opening or whatever but essentially you know your code better than any third-party tool so that's why this makes sense this is much more explicit than a third-party tool saying let me do all your monitoring for you so the first tool is counters as I said it's great if you control when something is opened or closed so particularly good when you want to check connection pools okay how many connections right now open it, increase close it, decrease gauges are great if you're relying on a third-party or an external source you want to gauge its value at every moment and you see how its value is changing for example if you're relying on a certain disk space you're using Redis you want to keep track of how the Redis memory is changing over time you can use gauges meters are great for tracking the rate of something so for example if you want to track how many requests you're receiving per second meters are a great tool while these were good use cases for these tools we found even better use cases with them tracking actual business metrics for example something like counters can be used to track the number of users you're personalizing and it's as easy as if I'm giving any personal recommendations at all just increment the counter and later on you can visualize how many users are you personalizing are you personalizing more than you were just a while ago are you not similarly for gauges if you have like a cache of products with which you do a lookup and you base certain recommendations on that and you see those recommendations underperforming or overperforming you can directly correlate okay so the gauge tells me that the cache product count was too little or too much so that kind of gives you an insight into why a thing is happening similarly meters can be used to track how many orders are being placed every second and the next ones histograms histograms are particularly good if you want to measure the count of something or the frequency of something and how that changes across time so you can check okay if you're offering a response how does the response size change for example something goes down maybe you don't have I don't know database your query gets messed up your response size just reduces drastically and you can easily tell but we'll come to a better use case of this which is how many of my customers what quantile of my customers are getting five personal recommendations or less so essentially histograms allow you to measure quantiles what is the main number of recommendations that you're serving which is an average is the median how many recommendations are 99 percentile of your users getting so that's what so it gives you a lot of insight timers are great timers essentially have a meter built right in and are able to measure how long it took at that level at that rate for example you can measure page rendering time and you can correlate it with how many requests I was getting similarly a great insight is okay it took me 80 milliseconds to give recommendations to my 99 percent of my users when I had 200 requests per second but the moment my requests became 800 per second I suddenly started taking 200 milliseconds so there's a problem that the timer is pointing out to you and we can take action on it again implementing them in code is fairly straightforward it's depending on your language it's always just one command one line of code or something so now that we've collected all this data we want to make sense of it first we want to store it somewhere and the choice of storage solutions has been increasing luckily all of these metrics are time series based data so what makes sense while storing them is a time series database of course a great number of solutions were being discussed today morning and one of those solutions is graphite and others in fluxdb mostly all of them sort of have an architecture similar to this one you'll have some sort of a back end storage where they would be using some sort of database maybe a sqlite or maybe a different file system altogether they would have an api which is your queryable interface you may use sqlite, you may have an ugly string that graphite provides and then they all have a layer in between which is your true metric storage which takes the pain of compressing or also helps in queries of course with metrics they arrive at a very tremendous rate they're just a lot of metrics that you gather so it's not optimal to write them all to disk so that's why usually the solutions have a cache built in where the data goes first and is a regularly flush to disk and when reading you essentially fetch from the cache and from the disk so you maintain a certain recency just some of the characteristics of a storage tool would be it should have a good query dsl in this case I think influx is a good job they picked sqlite which everybody understands graphite not so much because they pick just pure strings and you have to apply mathematical functions query editor is pretty prone to mistakes and everything so yeah well take your pick it should give you the ability to apply functions so essentially you want to apply mathematical functions because the same metric may give you different information based on what function you're applying and for example a counter may give you a current count of open http requests but if you check the slope of that talking statistics here but essentially you check the acceleration the rate at which the counter is growing you can easily check request per second just increase so essentially I'm facing some traffic now more than before the solution needs to be scalable again the metrics are coming at a tremendous rate so whatever solution you have should be scalable graphite says it works in distributed I didn't see it working so much but maybe there are solutions that works better than that the results should be up to date very important otherwise your graphs would be just completely silly right and this is a great note to read the influx DB how their journey has been first they started with using level db to store their metrics and then they proceeded with a different data structure all together today they use time structural mercheries it's a great blog post and there's a link down there hopefully you'll get it when we share the slides or you could just check it on the influx website now that we've gathered all this data the next phase is to visualize them so that we can finally draw some insights out of the data that I've gathered and for this it's very important that I'm able to drill down into why certain things happen so any visualization tool that you use should allow you the capability of drilling down it should be interactive I should be able to read what a value of something was at that very instant what happened at that point so it should have a good query query language and editor which is not prone to as many errors so essentially it would be good if you use easier syntax and you should be able to correlate changes in metrics to certain events that happen I think I think this was also discussed today morning when we discovered our annotations feature in Grafana similar features may be offered by other tools as well so given that we need all of this present to you the oscillator so this is a tool we wrote it's a d3 based tool it's written in closure of course you may like the language you may not like the language but you sure can like the tool it's very easy to implement a chart it's very declarative and the charts are highly interactive thanks to d3 which of course is a great library but the advantages that you get is you can define your charts in basic just data so for example you can have a simple map defining what your chart is called what your page is called what charts it has so you define tiles and you may have multiple charts on one page we'll see a demo soon but essentially each of these tiles each of these charts you can choose what kind of chart you want to show for example in this case we're showing my user request I'm pointing it to a graphite target in this case because that's what we were using when we wrote this presentation but essentially what we also open-sourced was a graphite DSL so it's a bunch of functions a bunch of closure functions that allow you to work with graphite strings without you having to actually write those strings so this minimizes the chance of making an error right so I have a small demo so right so this is so it's a random generator that I have and it's generating one request or the other and I'm sort of plotting the count of these requests if you check the chart is fairly interactive at any given point in time you can correlate you can sort of compare what the value was of one particular type of request and what the value was of another type of request moreover you can drill down into the timelines essentially you can see what was happening in the past one hour what's been happening let's say in the past 24 hours well it was an application running on my local system which is why you see the breaks hopefully your production system does not suffer through the same but yeah so essentially you can drill down into timelines you can also have summarization so how has the trend behaved if you summarize over a 10 minute scale so it seems to be going down a bit you can also have on the same oscillator you can have all your environments so essentially you can have your development environment you can have your pre-production or what you define what your environments are so you can have as many as you want right and then you can kind of hide one only see one you can have as you like you don't just have line charts you have all sorts of charts you can have bar charts, pie charts if you're into that maybe stacked charts makes more sense another another great feature is again annotations so these are events that significantly change something in your application and may lead to variations in these metrics so it makes a lot of sense to have them on the same graph as the metrics visualization for example those dots down there kind of point to what happened at that instant that the graph changed and so it becomes easier to correlate okay there was a code change there was a deployment there was a provisioning you can easily correlate why the change happened based on that event right so I had a few learnings based on this sort of flow the first was to choose the right metric and the first thing when someone talks about monitoring is hey let's just plot the CPU load and left out of that conversation because I honestly did not know what goes into calculating the CPU load and so I googled and I figured out that the formula to computer CPU load is actually very complex and you don't want your metrics to be as complex as that I googled further and I saw okay what spikes CPU load essentially what makes a CPU load spike and I figured out that there's this post that says if you're parsing text often times your CPU load will spike which is odd if my application requires to parse text I will be parsing text what is a CPU load plot giving me so essentially choose simpler metrics metrics that point to exactly what's happening for example disk utilization, user logins simpler metrics again the second learning was that plotting a mean value of something may be absurd at times for example we had a contract with t