he herd party that we'll give them recommendations they can show them and we had a contract we'll serve in under 50 milliseconds and when we actually saw it on our graph we were serving in under 20 milliseconds and we were happy we said okay so that's pretty cool we were well below 50% of the threshold so that should be okay until they came back to us as they said they're running into circuit breakers and this is you're often going above the limit and that's when we realized what we were seeing on the chart was actually mean an average value we plotted against the max which is the max value in every summarization interval for example the max value in every minute we see a graph something like this so essentially we were easily peaking over 100 milliseconds every minute and that's why the team was kind of running into circuit breakers so just choose how granular you want to be just don't pick mean because it just sounds cool so yeah choose wisely it's important to choose the right tool for the job histograms are particularly good if you want to measure the size of something the frequency of something how that changes right and the last phase of the monitoring setup is alerting because we all want to go home and sleep at some point in time so essentially we don't want to keep adding we don't want to just keep glancing at the chart and when it came to alerting again we being the team that we were we didn't like any of the solutions out there and we wrote one ourselves and this tool is called xray it's essentially again a tool written in closure in which you can essentially define your condition for the failure of the alert in a simple function so you can have a function saying if this happens if it returns true it's all cool if it returns false, alert me and then you can define what sort of alerting strategy you have do you want one alert every five minutes do you want if it's continuously breaking or do you want to be alerted on each instance of the condition breaking you can also define what sort of alert you want to receive you can have slack integration because it's all code right and that's what the advantage you get from having an explicitly written down solution so you could have slack integration any sort of emails or text messages I don't think you want to go for that though but yeah for example in this we're simply logging and you can change whatever like you can pretty flexible in the end xray also gives you a nice little dashboard gives you three level views this is the middle level and I've had to blur out the names because it was a live system I can't share that but essentially you have all your environments and how the checks are performing on each of those environments and you could basically drill down into the ones that are read and check what exactly happened, what was the expected value, what is the value we got you also have a higher level chart which sort of aggregates all of this and gives you whether the whole system is behaving okay or not so yeah we thought it was extremely high importance early morning in our stand-ups the first thing we used to do was decide what's going to happen in the rest of the day based on these alerts what is the value of the alerts we are getting we added these to the definition of done of our story so any new feature that you are adding would not be complete until you add the relevant alerts or change existing alerts as required we wanted to make it as visible as possible in our case we were able to provision a nice little monitor and slack notifications to ensure but at the same time you don't want to spam anybody so try and choose the right alerting strategy the entire solution kind of looks like this you capture your metrics you aggregate them store them in a nice time series database visualize them using some sort of a graphing tool and finally you alert and you choose an alerting library for this there's plenty of solutions that offer all these four phases tick stack and whatnot there's enough solutions you can take your pick for each of those phases just last note on continuous delivery as I mentioned since the marginal change between deployments has become so small it's become increasingly important to find out what broke something for example there was this one instance where we noticed one graph just shift drastically and what we realized was we had a deployment but since the last deployment before this there had been only 10 commits that went live so if it's just 10 commits it's very easy to kind of find out rollback 10 commits apply one commit at a time see which commit broke so it's a matter of practice as well no monitoring tool is going to allow you to really do wonders without you following some sort of etiquette and I think it's important to follow the etiquette of continuous delivery especially in a retail shop where you constantly want to release features you don't want to hold back the teams and a note on software that claims to monitor everything for you it just seems like they know your system much better than you know your system so I don't know why you want to go why would you want to go for something like this especially because some stack that we saw there are like 100 alternatives to that all of them free why would you ever want to choose something that doesn't even know your code so the only way I can think of they could do monitoring is by hooking into individual function calls it could be too expensive both performance wise as well as monetary because they charge an exorbitant piece and so really I mean a great monitoring setup is available for free really no need to go for such solutions so these are the references that I talked about the first one is a great talk again if you haven't seen it metrics metrics everywhere the oscillator is available at that link it's on github and the xray the alerting library is available at the next link I urge you all to follow the auto dev blog where they publish excellent things about how they're changing the architecture they're one company that's keeping up with the times and essentially doing great with all of that that's all I had thank you does anybody have any questions oh yeah so we fell into the trap as well honestly we faced enough black for that so initially when we built the solution we were spamming everyone in fact we were spamming more than just our team even like the organization so we faced that trouble essentially we left the choice fairly flexible because you may want to categorize based on the priority of the alerts since it's all in code it was a little easier for us because we could essentially see okay this particular alert if something happens I do want to be alerted and this particular alert don't on the other hand you could also have summarizations okay I want to be alerted if this thing happens continuously for 5 minutes if it's a one-off I don't want to be alerted but really I don't have a perfect answer for that it's really a grey area but we yeah we fell into the trap as well I'd say we really did not we really did not focus on building a product that was different from Grafana we think both are great products honestly at the time that we built the oscillator Grafana wasn't as mature as it is now that being said we would probably have still done it anyway we love writing explicit charts in code we like to have more granular control with the language that we are using I guess those could be the reasons and writing something yourself although frowned upon but for larger enterprises sometimes makes sense because it offers you the flexibility that no prepackaged tool could offer you hello yes hi so from the talk it looks like the matrix the library which you are using ideas to infer the lot of details from the code rather than from a log file and push it to a system so what happens is like in a web application at an end point you interact with like 10 other different system and collate the result and finally present to a user as a result what happens is like you have two places to infer the information for an example my web application can infer how much does a recommendation engine took the time so I can put what to say like the point at that web application level code or at the inference engine and later you can push to the system so which style did you follow and is there any reason to choose putting everything at a web application level or at the final system level we had matrix on both we were collecting matrix from both essentially the rule of thumb that we followed what Kodohil also mentioned in his talk is every microservice may have maybe should be capturing their own matrix and you may have up to 30-40 metrics from each microservice reasonably sized microservice of course we often go above board but yeah I don't know we just I guess it's one of those separation of concerns thing the web application did capture these metrics for a third party that was not written by us because we did not have control over them and we did want to measure how that third party is behaving so that we could communicate effectively but yeah we went ahead with the approach of just having every service have its own metrics and container kind of thing. Thank you. Hello. Hi. This is about collecting data on only useful data part so how do you choose to collect useful data I understand the intent behind that statement but there's also this fear of missing out and have you run into any questions where you actually missed out on some useful data that might have led to say a new metric. Of course obviously but yeah I guess what I'm trying to say is yes you should capture nearly like everything that you think could be plotted on a chart but definitely not something like okay what's the CPU load at the time right that's why I kind of tied it to the end of the presentation as well capture metrics that are simple that really offer you some insight into what's going on into your system. Of course we missed at times capturing useful insights at times it was hard to even keep the charts updated because there may have been a change in the feature after which the metric also should have changed accordingly but did not test helped us in that situation I guess that's not your question but yeah again that's a great area we ran into that trouble I guess you can aspire to be as vigilant as possible when you're building a new feature anything that's simple enough to be captured gives you some insight to be plotted on a chart just do it but yeah there's no black and white. I had a follow up question to that so when you said that you captured business metrics as well as infrastructure metrics would you say that looking at the business metrics daily is kind of the only important thing and if there's something wrong with the business metrics then you go into the infrastructure metrics not otherwise why would you say so so essentially something like if I'm using a redis I want to keep track of him how how much redis is filled or for example have I expired too many keys to suddenly am I done sorry okay so yeah so I guess we were keeping track of any of these charts but by the same logic if suppose you have redis at suppose 90% fill capacity but your business is going as usual there's no problem right no you may run out of capacity then your business will not be doing good that the trigger should be the business related metrics that's kind of what no so essentially the entire intention is to not let the business metrics be affected right there may be business metrics that tell you that or there may be infrastructure metrics telling you that so okay people who want to ask questions can stay back we'll have speakers Manan, Pooja and Bernd to answer your questions on this stage the people who are not really interested for the Q&A session can proceed for lunch so we're taking more questions but combined for all three speakers Pooja and Bernd could you please move to stage where does it happen sorry what solution okay okay well so the first part of your question was how do we get developers to be motivated to maintain the code that they've instrumented just for capturing metrics so essentially this is a culture change right in this case we were the driving team we were the team that went to the business and saying we think this makes more sense I guess with Agile the entire point is that everybody is kind of contributing and you don't sort of work in silos but yes I do understand that there are teams and there will always be teams which have a clear division and that makes sense for them but in this case it was not so we chose to be explicit and maintain the code we took that call that we will be happy to do this and the benefit we got was flexibility the benefit we got was we don't want to actually monitor something that we don't really care to monitor so I hope that's some sort of an answer I'm happy to have this discussion afterwards as well hello I'm Pawan here from Goldman Sachs sorry I can't hear you I'm Pawan from Goldman Sachs so this is a specific question to process monitoring so I just want to know what was done in auto or like in your space for critical process monitoring because say a process is down your order you know order rate could go down anytime and keep a host is down can you put the microphone did you get the question to some extent I think he's asking about critical process monitoring ok sir so yeah in certain cases there's a host outage and you have a barrage of alerts of process being down and you have to be with that so was there any intelligent solution that was adopted I guess it's not really related to this talk but we essentially implemented some like circuit breakers so that even if some third party or some critical process is down we still fail gracefully but yeah I mean some would argue that you know if you're critical process it's such a critical process how graceful can you be to be honest I was not able to get the question I'm sorry critical process goes down some sort of outage so how do you how do you handle that so your entire orders order processing is down what sort of thing can you do to sort of manage circumvent the entire problem so in my opinion it's also that process is part of an application means you can simply monitor if the process exists but it doesn't help but at the end it should have an effect on an upcoming service and then getting to the root cause you figure out means if you think about the first talk going from the top down thinking about your business as an impact and then seeing what application is responsible then perhaps a process is not working means also that it could be fine in the process list doesn't mean that the process itself is working means process monitoring itself could be could be easy means it's a process alive but how many of them are allowed to be alive how much memory is consuming it's probably hard to answer in a single sentence but it could be a complex topic it is I'm certainly thinking so okay I'll repeat so you're saying that it could be expensive to put so if I understand your question correctly what he's asking is so instrumenting these metrics in code what if the code itself is unstable in some way and so saying that if the code is unstable the metrics would not be gathered in terms of you know you're collecting metrics imagine a web application like yours you're collecting metrics through a request response life cycle right that's going to be expensive again you would rather sort of push it asynchronously to another system okay so so metrics does not push events directly it does do batches and it won't push every event so then your infrastructure is probably it's very stateful right because oh yeah so yeah so there's a chance that if your application goes down you lose certain metrics yes that is true and the strategy that you could go for or you could choose to do one metric at a time which would not work well I'm sure because metrics come at a very high rate and it's better to actually batch them the second question on that is that every time you want to collect newer metrics like you have data let's assume for a moment you have a lot of data you want to collect newer metrics you're going to instrument your code every single time to sort of collect newer metrics as opposed to me sort of extraneously collecting that in some fashion right I mean just a question yeah so we sort of it as a part of developing the feature we did not think of it as an external I want to collect metrics from a feature that is already there so we said every feature that gets developed has the metric collection part in place and that is the response way that is part of the story part of the feature so we did not I was just trying to understand philosophically what was the thought process behind doing this which is doing that we just favored flexibility so essentially we don't want to collect certain metrics we choose not to collect them there's an aspect of testing but I guess it could be frowned upon but we tested what metrics we are gathering as well and you could do that because they're in code well let me I guess I did not say it's not important I said that you won't be able to get meaningful right if you see a change in CPU usage it's yes it's important and well I'm just saying that if your CPU load drops what is that really indication of so as I understand CPU load is a function of not one thing there's multiple things that can bring down CPU load bring up CPU load so it going up or down may be caused by five things and you're not pointed to any of those five things so that's why we chose to go for simpler metrics which point to only one thing going up or down but certainly not not important it's important it's just not easily intuitively trackable so essentially we have systems in production and we try and maintain them such that they don't go up to failure so without having them go up to failure how do I know what sort of capacity I can get one and is any system available which would help me identify bottlenecks in my whole architecture because not everything scales linearly so maybe right now I can see that everything can go to X but one of the whole stack will fail much before that so is there some way I can have a dynamic bottleneck identification so starting with the last question again are you talking about being more services building an application in a combination means having a rule for them means that a service consists of others or are you talking about the user story question we had before yeah essentially even a very simple web application would have starting from the load balancer to the application code to the caches to the databases okay one good way to do it if you use in a matrix database so forever influx or graphite that you put graphana on top and then add different sources from different services you can create a graphana panel and then add probably let's say the load from your database server the load from your application server and the load from the web server and then you can see how a request impacts your infrastructure from the three whole stack simple idea but then you can bring all metrics together at the same time which is an advantage of graphite means that you combine different things and also adding to his answer is then also perhaps load monitoring makes sense load monitoring process monitoring with the correlation of an annotation for example to see okay we changed something with puppet and then my load is increasing then it's available information because I have a different load profile but without any information it doesn't help you because you can see okay no we have more load but what now shut down reboot with the annotation to see that some code change so you see a commit message for example depending on your continuous delivery workflow you immediately see what's happened to a specific part of the infrastructure means you can see that a user request at 10 o'clock was going well and 11 o'clock after I don't know deploying some new software you see an increase of load at the application server after a request in combining these metrics on a specific time you can see that some performance profile change and that will I think could be a way to solve it okay so follow up question what about you know any way of finding the current bottlenecks say the whole structure keeps dynamically changing and the code keeps changing maybe I'm currently making one database call and then I want to making three different database calls so in the whole development cycle can I have an automated system which would tell me in the current scenario this is the peak load you are able to provide with your current setup it's kind of a base lining that the monitoring system learns what the load typically is at 5 o'clock in the morning I say I just want the peak capacity of my system okay means you would like to know what the maximum number of user request is not on a cloud yeah that depends on so many factors so it depends if your system scales in a linear way for example or if garbage collection kills you in some way so it scales and then it breaks down so it depends on your I would say it depends really on the development language and so many effects also sometimes databases scale very very good in a linear way and at some point they are not able to write down the IOP to really promise you can do a specific amount means testing helps of course there are thousands of testing frameworks which probably only know 2% of them but I think I would say it's not possible to say that infrastructure can handle 3 or 4 times the user you have right now in the future except you exactly know what every system involved in that process has a linear scaling aspect therefore I don't have an answer for that perhaps you have sorry yeah I think that's as good of an answer as I can also Hello, oh thanks Manat, great talk, thanks for that and quick question I didn't know about metrics and I'm going to look it up and I keep hearing about tools like Zipkin and all that do they play in the same space are they comparable and maybe burned you can also weigh in it right because I'm confused about the whole aspect about you know you have monitoring application life cycle monitoring or something so just break that down for me thanks so I'll start so I don't perhaps one can help with Zipkin if you mention I am honestly not aware but there are alternates to metrics so metrics follows the more push events sort of flow on the other hand there's tools like Prometheus which have a so essentially your Prometheus server sort of knows where your application is running with all servers and they collect full data from them but I don't know about that particular tool that you were mentioning I'm not aware of it sorry I think these are a class of tools that allow you to trace a transaction across multiple systems you know it's more of a Zipkin? Zipkin yes well metrics is more for one one instance of yeah I'll ask a question to Pooja quickly it's probably not related to monitoring but if I understood the whole bot approach that you talked about correctly right you had to rely on regex a lot to understand what the user has typed in right had you by any chance looked at you know natural language processing that are things like api.a that might make it easier is that planned that's the question thanks it's my question even I had thought initially that we should have some kind of algorithm of natural language processing but the initial idea was not to go in much details of that the initial idea was to have a control over the code which was messing up things and that's where we focused on solving that problem first that isn't planned but not immediate plans because there are hardly certain questions which are repeated to being asked for example when is my release what is there in the release which machine actually replies there are hardly totally countable maybe so that's overhead to do it at the moment and that's where my priority is continuous integration first wherein we can actually stop the pull request merge button itself to go ahead so that's the immediate plan probably NLP is what is very interesting for me also so probably I'll have a look at it thank you hello Manan you said don't use any arbitrary measure like me so in so I'll just divert here a bit whenever you're doing machine learning so you build models use a couple of algorithms before you figure out which one is best suiting your needs or purposes so if you look at most of the things that we are doing is collecting data and applying certain statistical analysis on that data so do we have some heuristic method in which you apply this statistical model first see if it works for you if it is actually getting the patterns out of the data that you want so is there like step 1 use statistical model not this use statistical model B so is there some kind of flow chart or something like that if you could share with us if you are using it and any of you could answer this question I'll try to so essentially there's a stream of DevOps called anomaly detection which is heavily making use of knowledge of statistics and applying models is a fairly straightforward use case there isn't much derivation that's happening here essentially just yeah so I'm guessing in this case I am not aware of a lot of statistical models who say that are applied at least not standard ones of course people might be coming up with ad hoc statistics on top of something but yeah but there's a stream of which is catching a lot of traction of course it's anomaly detection where you kind of learn from how your particular one statistic had behaved and you kind of make assumptions about whether to alert or not for example when it happens again because you can now correlate that this would happen at this time of this month of this day kind of something like this that's as best as I can answer maybe one of you can I have no answer but I know somebody who knows it and his name is Avishi Ashalom and you can find a lot of talks on YouTube and I think he studied math whatever and he's giving crazy talks about math and metrics I can give you the name afterwards I don't know how to write it I will have to look it up Avishi Ashalom or something I give it to you they'll share it with you yeah we're on Twitter whatever and his talks are really a good point to start about thinking stats and metrics and understanding it very well perhaps a good point to start okay I'd just like to remind everyone that we're happy to take more questions if there are but this is eating into the lunch hour the lunch finishes at 2pm and the talks resume here at 2.05pm sharp and we're gonna try not to be late so just to remind you now that was it oh food break I'm out I saw your face like you're getting nervous story of my life is your mic on it's hard to okay can you put the mic a little so you have 100 nodes out of 50 no I got the question so you want to scale so sure it's possible I think that is what providers do they look at the traffic and accordingly or to scale of course I have no experience doing this and wasn't the subject for say my talk but I'm sure so essentially the tools that we wrote made use of monitoring data and alerted or not alerted based on that so instead of alerting you could trigger something else I'm assuming that should be possible but maybe one of you can I'm sorry that your mic is off now it is like this okay so my question is more about distributed monitoring so I have a system of tools which logically comprises one tool right so my tool talks to a lot of other systems before it can serve a response it's a general distributed architecture and these systems interact in different protocols so it's not like HTTP sometimes it may be RPC it may be TCP so I cannot really weave the HTTP headers and see how the data is flowing so right now the solutions are being implemented to correlate how a request is served by different systems at a given point of time is to use overlapping groups, group names so we are targeting systems with a certain name say this is product P1 this is P1's active MQ this is P1's Redis probably this is P1's database this is P1's MySQL we plot a graph or we want to correlate a graph and see how all the systems related to P1 are behaving so in that case we are actually bound to take care of giving correct tags at every point of time so is there a way wherein I can trace a request that this request flows from system A to B to C to D with different protocols and then have a holistic view without using tags somehow so that it doesn't have any manual dependency so systems like APM also do it for me but they are more like only one platform based say only Java based so if I go from Java to a different or go like a console or probably a Redis which is written in different language so systems get like they are like blacked out and it's also very costly to put APM into like every system to monitor it so what do you think it's like grouping, overlapping group with the only solution or can we do something else about this I don't think there is a tool for that I don't know the way you can do it it means enrich your code base and probably assign kind of a session ID or a random number or a serial number I know Netflix is doing it that way tracking their SMS and recommendation engine that they gave a talk on Elastic on about that how they trace a specific request in the infrastructure which means assigning a serial number and then to every lock entry they produce manually or out of the system the system outside which takes all these serial numbers and puts them together, leaves them together but the problem with this approach is that when we have different protocols to talk say for example if serial number is into my HTTP header and now from HTTP I'm going to a TCP so I'm losing that serial number because I do not know how to propagate the serial number along with a different protocol all together so it's like assume a JMS system I have an active MQ and it has its own protocol I don't know I think if you don't enrich your protocol how do you transfer the session ID if you don't enrich your data payload which is transferred with that kind of a marker it's hard to identify I have no better idea perhaps you you seem to have already done a bit of research on this but yeah I would love to pick your brain on this later I have no more solution on this that's it guys I'll show you lunch ok thanks everyone for your questions we'll close the session and enjoy lunch now thanks see you again at 205 sharp the features required by your application are met by your operating system sometimes that works out sometimes it doesn't if you decouple your application from your operating system you have more flexibility so everybody understands it right? so about me my name is George Clifford Williams you can email me at GCW at 8ions.com the G in my name stands for George I load the name George so you can call me that if you like I probably won't respond I go by Cliff live in Chicago married, no kids, two dogs I love spoiling my nieces and nephews I need practically agnostic means that when it's time to discuss the merits of something I will gladly say what I think is the best solution but when it comes to actually getting work done it's not time for deciding anymore so it's just time to get work done in my day job I spend a lot of time consulting with clients to put things in the cloud to help them develop better CI CD pipelines I do all the things that get labeled as DevOps and I try to automate as much as possible so what I talk about here is all from the perspective of someone who digs deeply into delivering packages to the cloud so understanding the problem if you are a developer or if you support developers by pushing out applications typically someone's developing code and they find an operating system to deploy it on everything works out perfectly and there are never any problems in Dreamland this view of your application is not really tied to the real world a slightly more realistic view of it is that you have a kernel that your application will need services from on top of that you have a library of some sort that interfaces with system calls you have your user land utilities and then packages that get installed and then finally your application code gets deployed this by the way applies to POSIXY type systems Linux, Solaris, FreeBSD, the sort if you are on something that is not POSIXY then sorry, I say it doesn't really apply to you so this is the way it really looks as compared to the kind of loose idea of how your application deployment might work so if you get that you set in, you write your code you deploy it on your application you deploy it on your operating system what happens when you need to upgrade so if you upgrade your operating system let's say can someone name an operating system or distribution that they use just for one out FreeBSD so let's say you deploy your application on FreeBSD and then you need to do an upgrade because you could upgrade your base operating system hopefully everything goes well but if it doesn't your application may break in interesting ways and when that happens you might have to rework your code or you might have to back out the upgrade now, why would it break there could be conflicts with versions of libraries there could be a security fix that forces a change of some tool to be deprecated which is pretty common so what if your application isn't affected by an upgrade but is affected by needing something different maybe a tool that conflicts with another tool or a newer version of something that's already on your system there's a chance that you could use a private repository on FreeBSD systems it's pretty easy to use the packaging system to say go fetch this set of packages from another place similarly on Red Hat or Ubuntu you can point to other repositories add that into the mix of the repositories that you use to deploy your application and hopefully everything will be fine if it isn't you could download and compile something yourself and the enterprise frequently, Java is a requirement that people need to download because there's specific versions that come with most enterprise operating systems or distributions and then your application may require a different version so downloading and compiling yourself it actually used to be the common way that people would deploy applications now it's rare alternatively if your operating system has a maintainer for a particular package you can reach out to them and say hey I need a new version of this hopefully they'll get back to you sometimes they don't if you're needing something that conflicts for example this is SSL there are many SSL libraries out there everybody is starting to be more security conscious with the way that they develop applications so TLS and SSL are pretty common your application may require a different version of a TLS library than what comes with your operating system and that can cause problems so one way to deal with that is with the troup jail or containers linux containers jails, solaris zones or environment management something like virtual inv on python or if you're using a java app setting the java home the kind of thing that lets you set a narrow focus for your application to find the tool that it needs alternatively you can just wait and hope that whatever tool that you need becomes the prevalent tool for your operating system a consideration in all of this is how you're actually deploying your code not uncommon for people to write their code tar it up, ship it over the wire with SSH or something or put it on like a gift server go out to machine, pull it down and consider that a deployment also some people package their apps in the native format for their operating system so in previous view it would be a port or a dev on a debian and ubuntu systems you could package it in the run time for a language that you're building in so if it's a java script app maybe it's an npm package or a ruby gem a luarop pycomic etc you can also do a make file maven and ant are really just anti make systems so you can specify where files should go ship that along with your tar ball sorry, ship that along with your tar ball and hope for the best then there are kind of smarter tools for doing all of the above puppetchef, antable salt civilly, blade logic they all do configuration management basically take a set of files from here, put them over there and it's better than doing things by hand but when it comes to deploying applications it's not the best solution so the problem statement what I'm here to talk about and try to solve is that when you build your applications on top of the facilities provided by your operating system you could be locking yourself into an ecosystem that does not meet the needs of your application and or customers and the solution is to build your applications to be independent of the underlying operating system and its packages so what that would look like in practice is you have your kernel again, your library that interfaces with the system calls your userland utilities and alongside your system packages you have a set of packages that meet all the dependencies for your application and then your application goes on top of the dependencies and this can be duplicated so that you can have multiple instances of dependencies for your applications and deploy your applications on specific stacks just for those particular applications so some of the ways that this helps with application delivery release management etc is you get fully autonomous applications meaning that you can upgrade your operating system and packages without worrying about breaking dependencies for your applications this means that if you're on Ubuntu and you have to do an upgrade because of some security patch you don't have to worry about going back and reworking code to work with the new dependencies or with the new stack you can create multiple application silos that contain conflicting libraries and tools so if one application requires a particular version of a Libre SSL and something else requires open SSL you can install those right alongside each other and they will not conflict because of the isolation that you get from autonomous application setups your deployments can be standardized across multiple operating systems so a lot of the clients that I've worked with have a mandate to not have more than 50% of any one distributioner operating system be not to have any one distribution operating system be more than 50% of the deployment that means frequently working with Sless and Red Hat and FreeBSDE and Ubuntu and HPUX and Solaris and some Constellation all at once taking the approach of building your dependency separately from your operating system means that you can deploy an application on any operating system with one set of configuration and not have to worry about specialties and file paths conflicting you can isolate the exposure to security flaws in underlying libraries that goes back to again there's some update that needs to happen it can happen whether it's in your application or on the operating system so if your application uses a particular version of a library and you discover there's vulnerability you can fix that in your application and not worry about it causing a problem with your operating system the features of your application can develop at your pace not the pace of your operating system package maintainer so if you need new features you can go out and get those new features implemented in your code and not have to wait for the next version of your OS and you still have access to all of your system packages so what this approach really does is let you decide what the delineation is between the platform on your network and the application and its dependencies and you can swap out the application swap out the platform it really doesn't matter because they're completely orthogonal to each other one doesn't depend on the other so I think that sounds great let's presume that all of you are on board and you want to get started with that there are several frameworks that come out of the box with the ability to let you do this on a positive system positive again, I mean Linux HPUX, Solaris something like that there's package source which is part of the NetBSD project it's very similar to freeBSD ports or any of the other BSDs port systems there's open package which was started by a former freeBSD contributor named Ralph F. Englishall he used to be a security or a security officer and he was also a founder of OpenSSL there's the Nix package manager it's currently mostly Linux specific and driven by the Nix operating system I happen to prefer package source it has more than 1,700 packages available it includes all the big things that people need and most deployments Nginx, varnish databases like Postgres just about everything that you would need it offers you the choice of binary or source builds so systems like Gen2 FreeBSD, OpenBSD by default if you want to install a package frequently you navigate to some file path and then type make wait for a while, get your package you type make install and you're off and running if you don't have time for that you can do binary builds of your packages and pre-stage them and use something simple to install analogous to a yum install or it's really easy to set up the multiple prefixes are what allow you to set up the application silos a quick example is that I had a client who had several Ruby on Rails applications that they needed to install and many of them required different versions of Ruby to run they were all deployed on one machine using package source with a different prefix defined for every application that needed to be run a prefix is basically a path for the dependencies and you can have as many as you need it's a simple straightforward process to package your own applications so when I was talking before about how you might deploy your applications using a tar ball or a make file if you package your applications in the package source format you can have something like Solve Ansible Chef, Puppet, etc go out to a box and install your application as a package which also lets you be able to roll back seamlessly it's easy to fork the repository and add dependencies that you need so when you check out package source it comes with you know, 1700 plus packages but if there's something that you want that's not in there you can add it very easily and it has unprivileged operation meaning that you can deploy package source for a given user and let that user manage its own dependencies for an application stack and you can do that for multiple users and they'll never complete package source is also portable it gets its whole own slide for portability if anybody in here has run Show of Hands has anyone run more than two of these so three people so daily I tend to work with at least three of these and having package source set up with autonomous deploys for deployments means that that's national when it's time to okay so what does it look like this is what it takes to get going with a very basic installation of package source by that I mean you pull the source code down that's what the get clone does you then change into the directory that's created and then go into bootstrap you run a bootstrap script and then you're all set to go then you can cd into a application path package source development cache and then type make it fall clean what that will do is build the package it will install the package and then it will clean up all the files that were created as a result every single part of your application stack so the idea again is to decide what you need for a system to be on your network and then decide what you need for your application to be able to be installed and treat the two as completely separate and here the package source develop memcache when you do the install there it will install the prefix that's defined in the environment so if I needed memcache install five different times I could re-run this command five different times with a different prefix and have different separate installations that I could then run in parallel so can someone name a run time or language that they use to deploy applications to something like Python, JavaScript to use Kepestrano for actually doing the deployments okay are you deploying Ruby apps okay so do you ever have problems with gem versions so with a setup like this you would do the bootstrap you go in build Ruby then build Kepestrano for on another system build Kepestrano to do the actual deployments for those who don't know Kepestrano is simply a tool that goes out and it lets you use Ruby code to specify a set of things to happen on a remote server similar in some respects to tools like Ansible except that it's not it's imperative that you actually have to do every step very much like shell script so you would have your Ruby script or I'm sorry you build Ruby as your application depends on all of the gems that come along with it would also be part of that dependency bundle and you would have a definition in Kepestrano to say this is what's required for my applications including Ruby every single gem and then deploy that when you needed to upgrade you would simply redefine the versions of the packages required for that rerun your Kepestrano script and then you'd have an upgrade that doesn't care what version of Ruby is required by utility in your system I'm trying to think of a good review utility that's commonly installed but and I can't but in a situation like this your application is completely safe from any changes that happen with the operating system packages so I ended up running through my slides pretty quickly I will now take questions if anybody has any I just wanted to ask how does this stack up against me just pushing my application into a container like why will I not go with a Docker container put all my dependencies in it and ship it everywhere instead of using the one of the tools that you just mentioned great question and thank you for asking it so how does this compare to containers basically containers exist on well if you're talking about Docker containers or LXC type containers then only on one platform Linux there are different distributions you can have Red Hat or Blue Two Origin Two or whatever but it's still one platform the clients that I work with can't have just one platform so they need a solution that works on multiple things a solution that works on those and because I'm known as a strong advocate for BSE systems sometimes people think that I'm aligning other platforms and I'm not the fact of the matter is that containers frequently hang in ominous ways you don't have to worry about weird unrecoverable hangs with a system like this it's very lightweight forward if you need network level isolation then it gets a little bit tricky so containers may work for you there or if you're in an environment where you have access to more robust container systems like zones on Solaris or jails on FreeBSD then there's kind of an administrative overhead tradeoff like would you rather manage containers or manage like just the application dependencies so that's basically the difference thanks so my question is regarding let's say for example when I break my OS suppose the TLS package has been created but then my code and my web server and everything is all of a sudden breaking now if I use package source and install the previous version then what's the point of sort of upgrading or having the newest security update then there's sort of like is this more of a temporary fix that we do so that things are in production and then we try to make it more compatible to the newer TLS libraries or something like that I'm sorry I lost you at the end there you said is this more for example if the OS has come up with the new TLS library and if we use package source to have the previous TLS library which is more compatible with my web server for now then I can still have everything in production and nothing would break now in that case what's the point of upgrading the OS in the first case then is package source only for a temporary measure or what do you recommend okay so if I understand your question correctly you're saying that if I can handle all of my application dependencies without touching the operating system what's the point in ever upgrading the operating system well no I won't upgrade the operating system so that I can have the latest TLS package right so if I'm gonna go reverse compatible and install the previous TLS package using package source then what's the point then is this a temporary measure like we use package source as a temporary measure so that everything is working fine and once we resolve the issues then we just knock this off completely got it okay so package source in this case would not be a temporary measure it's the way you build your application back right so from whatever layer forward and that's something that's you know that needs to be decided by a dev or engineering team saying that okay we have java let's say it's a open JDK 1.8 everything from that point forward is part of our application step you you may need to do rollbacks but that's related to your application not to your operating system you would need to start thinking about this division between the platform and the app you know what needs to happen and one doesn't need to happen in the other and it's not that you are using one as a short term fix it's just the way that things are from that point forward does that answer your question yeah it sort of does but yeah again what happens is that each application has hundreds of dependencies let's say for example if I have a node application it has bcrypt and bcrypt relies on these browser packages which come along right and then if I've created the browser then everything starts to break so I don't have to just fix my code but I'll have to probably dig into my dependency that is the bcrypt library and then modify it so it's good to pick the right version of the package that it needs so yeah of course it answers the question but it's just it's a lot of work around as well so you're talking about the administrative overhead of having package source packages yes so that's a concern just as it is with containers right so if you have a containerized system which by the way I'm sorry I should have mentioned before with containers there's one additional consideration with containers in that many people get prepackaged containers and there were recent audits of many prepackaged containers and something like 70x percent of them had really egregious security flaws in them I mean not like an app had like a buffer overflow in it but like bad configuration that like a systems engineer would like never do but yes you then have at least two sets of packages to maintain right your operating systems packages and the packages related to your application but again it's really easy to manage package source for public source you have you have things coming from upstream from the NetBSD project you also have and there are many contributors to that so Joints that happens to run a huge cloud platform they're access contributors to the package source project so it gets a lot of updates you know like fairly quickly and if you need to go out and do something on your own it's really easy to do but yes you're right there is administrative overhead there I think it's worth the tradeoff it may not be if you're always the point of one platform I'm sorry I lost you I have to give oh resource isolation yeah package source does not do that it's simply a way for you to install dependencies for your application and anything that you need beyond that you have to have either a user-laid facility to do it like something that comes with daemon tools or run it that does that or use an operating system facility like capsicum or jails or zones or some other thing like that great question so the question to restate it is how do you deal with building multiple packages and in particular the conflicts that can happen with tools like SSL or others where your application may require one requires another and frequently they don't play together nicely and can package source handle that the short answer is yes absolutely very easily in 99% of the cases there are some edge cases where simply having file system and path isolation for your libraries or your applications isn't enough and in those cases you might need to go to a container solution like salaris zones or previous dj or linux containers but in most cases with something like SSL the version that's installed by your operating system is usually not an issue for anything that's an application installed separately from the packages for your operating system so if you install package source you install libre SSL or open SSL or boring SSL or whatever and you install your application in the same prefix it will use those libraries and not conflict at all with the system SSL libraries the thing goes for 99% of the other libraries out there there was a problem a while ago with Postgres but that's been resolved and I'm not aware of any others that exist so the question was does package source support reproducible builds and yes for most of the packages I'm not sure that they've gotten through all of them but reproducible builds has been like a goal for package source for a while they've made really good progress on it I know that on NetBSD and Lumos I think freeBSD more than half of them can compile with reproducible builds which brings me to building multiple packages in general if you're going to build a repository for your packages so that you can do a very simple install similar to an app install or a yum install there is a distributed package builder that comes along with package source actually there are three of them there's one that's official build that will build every application that you specify in an isolated environment and you can tweak the parameters for that so if you need reproducible builds that's where you would go and do it tell it the list of packages that you want to have built and it will kick off a distributed build for that even if distributed means local to one host using multiple cores and then it will produce an index file that your package manager can use the package manager is called pkgin and you can install from there without needing to compile every single time any other questions? thank you very much just a reminder Cliff is giving a workshop this Sunday what is it salt stack and Ansible I think there's still some seats open so if you're interested in attending please have a look on the rootconk website and you can sign up okay I have a little bit of an announcement about this evening there will be a party tonight I guess you could call it a party and there will be transportation provided from here to the party venue the first shuttle will leave at 6.10pm and they will leave every 20 minutes after that so people should gather around near the registration desk to queue for the transport so just so that you know you will need your badge to get into the party tonight and you have to be careful because you also need the same badge again the next day to get back in the conference so don't lose it oh also today at 3.30pm just another reminder there will be slacklining and yoga session in the lawn if anyone needs to shift gears away from brain and tech conference and into your body that's available as well okay our next speaker is from Kerala from a city that I'm not sure I can pronounce properly hmm Alapuja Alapuja okay close the Venice of the East so please welcome Ranjith Rajaram who will be talking about what should be process ID number one in a container a topic I'm sure that will generate some controversy so yes welcome Ranjith hi guys my name is Ranjith I work for that app and I basically do product support so my session title is actually what should be the bid one in a container so I hope after 15 minutes actually I actually don't leave you with more questions but some answers which you can take it forward and build on top of it okay so please raise your hands if you have used dog containers or you know if you are using it for production or if you have either tested it or start that the containers are in place most of you so this is what I have for you this is going to be very short I will be using the first few slides to apply my main point and then we will have a demo so I will be using a demo to actually show you what the problem what problem you could face and why it is important to have why it is important for you to take control on the bid one and so we will be spending a lot of time on the demo and we will be switching between the demo and the slide more often so for this session I will be making use of dog containers since that is the most popular right now so how do you control which process becomes the bid one in a container right so you actually do a docker run when you do a docker start how do you control which process becomes the bid one so in a docker file we actually have two different ways one is the entry point the other is the cmd I will just use only the cmd option so how do you decide which process becomes the bid one you actually mention the process which you want to become the bid one by mentioning it with the cmd director so there are three different ways how you can use this popular director so this is the most popular one where you actually mention the application in this cmd director so this is just a ps output from two different containers that are running on the system and if you look at it so now I have this is my first container where I am actually starting the bid one here you could see the bid one is actually the bid one and the second container is just a sample output where your bash is the bid one and the top is actually child of the bid one so now if with the help of the bid namespace multiple containers the process which is running in multiple containers can have the same bid number but when you look at from the outside of the container from a host perspective each process has a unique TID number so now let's take a look at the demo so for the demo I am going to use a simple Python application so this particular Python application has some known issues so this particular example is just taken to highlight the problem so I am going to use two containers for this so the only difference between these is for the container one I am calling my Python application directly and for the second container I am actually calling a shell script and within the shell script I am actually calling the same Python application I am just going to change which process is going to become the bid one right so now let's take a look at the demo and on the extreme right side you actually see some lines which are being printed out I will tell you why I have done so now and in the demo I will show you a different container so this is my first container where I am actually calling the Python application directly and so I am just going to start the particular container so on my second tab so this is the same system so on the second tab I am just going to start my second container where I am just calling the bash script first and then within the bash script I am actually calling the Python application so this is my first container and within the bash script I am actually calling the Python application so now let's try to access the particular application so we will use a simple curl command so now you can see that I am able to connect to the container and also I am getting an output thing which is the hello world so for the same I will use the same command same curl command for my second container also just that it will have a different IP because the application which is running inside the container is the same so you would see the same output so now let's take a look at the p.s. output let's see how it looks like from a container perspective so this is good because as I said before I am calling the bash script first and within the bash script I am actually calling the Python source so now let's go to the first container and do the same thing let's look at the p.s. output do you see a problem here this is the same application it is just that in the first container this is the first container I am calling that particular application directly in the second container I am using a bash script and then from the bash script I am calling the particular source but now whenever I try to access this particular application now I am just going to show you so do you see an increase in number of different processes so each time when I access my application I am getting a different process at the same time for the second container where bash is my pit one so when I try to access the particular container actually I am not seeing any problem here so now why is defunct process a problem so now let's take for example if you have multiple containers running on each system the same instance of the same container running on each system and at one point of time when the load of the system goes high you would have defunct process within each container and what really happens is now let's take a look at the p.s. output of the whole system when you take a look at the p.s. output of the whole system you would actually see the actual pit number which is used by the process which is running inside the container so here you can see this is the actual pit number that is consumed by the process which is running inside the container so if you have multiple containers running on the whole and they all have the same sort of a problem then you would end up consuming all the pit numbers which are running inside the container so by default by default on a well-opened system of a pit you would actually have 32,768 pit numbers so at any point of time you can only have 32 processors or 32,760 processors so if you have some container that is actually leaking there are often processors and if the processors are not being reaped you would end up in such a situation where you will not be able to log into the system as well because to log in a system it has to fork a new processor but since all the pit space is being used you will not be able to log into the system so the only way is to either re-put the system or if you have some other way to reduce the number of ports which are running on this processor or take care of the often processors so now if you run the same application on a bare-metal system or on a virtual machine you will not see this problem because on a bare-metal system the pit one is actually your system D or your init so they actually have a special feature where actually they take care of processors so they actually clear the often processors so this is problem number one now what is problem number two now let's take a look at it so now is bash the answer for the problem so the bash is not so when you have bash as the pit one you are not seeing any often processors so is bash the answer for the problem the answer is an s and a no because s because it has the feature to read often processors why it's a no look at it so what I'm going to do is I'm just going to stop this particular container this bash container so now what is going to happen is you would see this particular command getting paused for 10 seconds it is actually paused for 10 seconds so it is not hung it is just waiting so now it has come out so it is able to stop the container but it took 10 seconds so now what is happening in the background so docker stop actually first it senses stick term that is it is asking the container to do it but here it first senses stick term it waits for by default 10 seconds and then it actually senses stick skill so here so when the container get the stick skill it actually you know exit very quickly without closing all the stuff so if you have a DB sort of an application running inside the container and if you don't handle the stick signals properly you end up having a problem where your data is not consistent where the data from the cache is not flushed into the oil system so now how do you find what is going wrong here so let's say for example I'll start the container again let's find the process so the process ID is 3960 so if you look at the work file system for an PID you actually have a file on a status where it actually gives you a lot of information about the process so here we are just interested about stick catch so stick catch is actually in an hex value so we will have to convert it into a human readable way so for that we actually have a bash script so now let's take a look at what is happening here so now the bash script is actually registered only for two signals one is stick so one is stick int so stick int is nothing but a control C sort of stuff and then you would see why bash is able to reap the child process because it actually also leaps into sick child so since the bash has a particular feature so you are actually able to find it using a stick child so now is bash the answer for the problem is bash the answer for the problem so we actually know bash is able to reap the child process but it is not able to handle the signals properly so now do we have an hack so can we use bash as the answer for the question what should be the PID PID one in a container so the answer is yes you could use it but you would have to rely more on some sort of script so now let's see how we could do it how we could make bash even handle handle the stick signals properly so this is what I said before so here if you see so what we could do is we could actually put the actual application in the background and then register for that signal in the trap so if you register for trap and you actually wait for the signal then as int and term so the moment it gets the term it actually takes it and it passes the term signal to your child processor and then it exits so it will do a graceful exit so bash can be an answer for the problem but do we have a better answer like what should be the PID one in a container you have seen this so so as I said before if you run the same application on the whole system it is actually taken care because your PID one is actually a system it actually passes the signals properly it can also read the child processor so now what about having a system be or a enamel in it inside a container so now to solve these problems to with processor weeping as well as signal handling it is still there we actually have a couple of open source implementation known as a tining and dumping it so this particular tining or dump in it is so small that it will only deal with two things that is processing and signal handling so it comes around 120 kb so you could have this particular binary container directory and then you can actually call it using this particular way this is how you do it so you actually use it in your KMD directive either tining or dumping it and then your actual application so when you start that particular container so this is how the tree would be your dump in it becomes the PID one and your python application becomes the child of it so this particular enamel in it can actually handle two things often processing and handling of it so now there is a thing that is coming with docker 1.13 known as the init flag so now this particular flag is actually a daemon option as well as you can actually pass it with the docker 1 command so what it does is so the moment you start a container using docker init so now let's go to the demo so this system actually has docker 1 1.13.1 so I am just starting the container the only difference here is I am just adding this particular new flag I am actually adding the new flag known as init so let's see what happens here see what is happening is I am using the same container but since I have used the flag init so what the docker daemon does is it actually starts enamel init so that is what you see here the init and then it starts the python app application as a child of it so now this could be a very good answer for this problem like how do you so let's say if you are not the author of an application which you want to migrate to a container and you actually don't know how it behaves so this could be a very easy way to do it that is with docker 1.13 you can actually pass it as a daemon flag so that every container that starts will have the pit one as a pinwheel init all you could control it per container by starting it with docker run by using it with docker run so we actually saw the emel init then we actually saw the docker init then what about having system d inside the container so for some of us yes system d could be heavy but then there are some additional features which you get when you have system d inside a container so now with the latest version of docker you can actually have system d running inside a container without using the privileged mode so this is actually done with the help of OCI hoops so now what are the additional benefits so now we actually saw you can actually deal the problem with signal handling as well as tile vaping using pinwheel init but then why do you want to use system d so system d actually comes with some additional features which I think you should consider it for example let's say for example you are actually migrating an application that actually writes it log to the dev log or the particular log is captured by the payment which is running in the host system so if you are planning to migrate any application to a container so then what you would have to do is you actually end up writing the logs either to the std out or to the std ever and there are multiple ways also by using log4j you can actually send the logs to a remote system so it actually differs but when you actually moving an application from a bare metal to a container things should be easy for you so this is where the system d actually helps you out now let's take a look at it so I am just starting a container which is actually going to start the system d so now let's take a look at the so here you could see the pinwheel init is actually the system d so what is the beneficial benefit that you are going to get so with the help of OCI hooks the moment so the moment when you start your container and your container so if you see here so here you are actually telling it to be sbin init so if it is a fedora 25 or 10, 20, 7 sbin init is actually your system d so when you start a container the moment you start the container and if the docker daemon with the help of the OCI hooks it actually detects your pit one is actually your system d then it does some additional things for example it actually registers your container with the machine CTL it will take 10 seconds to each container so here you can see so right now I only have one container which is running and that particular container the pit one is actually used by a system d so now what I could do is I could check for the logs of the container by running this particular command using the force system itself so now whatever is the output that your application that can be captured here using the general CTL now for example let's say you actually use a very simple application so the logger what it does is actually writes the logs to the devlog and you can actually capture that on your force system so now if I run this journal CTL command again this actually is a very good benefit if you are just migrating your application if it was actually running very well in your bare metal system to a container so I will just open it up for questions please here you could do it but then what you are actually doing is actually migrating an application which works very well on your bare metal system to a container but now if you actually want to handle these things you actually end up writing more actually make more changes to your application so now let's say the same application is running on your bare metal system you actually need not do anything else so if you are not the author I am saying if you are not the author of the application it actually so as I said before I have used this particular yeah see so that is what I am saying if you have an application bug and if you let that particular application bug to run inside a container without you knowing what is happening and you would come to know about the problem only at the last minute when you have multiple containers running all those containers actually leaking for societies you would end up having this problem so now how do we handle these things in a better way so you could have a minimal in it as a pit one so that you actually have some a confirmed way that application bugs can be handled because what will happen was for example how I came across this myself one of my customers was using Tomcat within a container and all of a sudden just days before the actual production we actually noticed this problem so that particular application was running perfectly fine outside it wasn't creating any zombie even if you run this application outside because it wasn't creating any zombie because we never knew what was happening whether it's an application bug or not so this can be captured at the time of testing but unfortunately we actually found this at the last minute so we actually had a tweak around so that is how I came across this particular topic what should be the pit one in a container so your questions are very valid but then any other questions so we just say one minute so if it is just one container that's fine let's say if you are using hundreds of containers of the same instance at one point of time it becomes a problem right then it becomes so then you have to choose the right orchestration tool then you have to use container native storages so it is not that easy for you to use the horse mount option we have thousands of containers and this container can run on any nodes so you have to make sure that particular horse mount is there in all the nodes so instead of using horse mount then you would start looking at nfs or the container native popular cluster it becomes a little bit top problem but you can do it you can actually do a volume so from the horse system you actually send it to your symbol server that is your ALT stack or maybe you can also use the planks you can actually send the logs from the system D to the planks server to a remote server so when it comes to docker logging or container logging you wouldn't keep it on one particular horse because you actually don't know whether the horse will go down or not so you actually send the logs to a remote server so I think you are done so thank you for your time thank you very much I'd like to take just one moment to remind everyone about the feedback form that's in your conference bag please do take a moment to fill out the form it's very helpful for us to improve the conference every year you can put your completed forms in the bags just outside the auditorium and I'm now going to turn over to Zainabhava our fearless leader just a quick announcement the off the record session on my sequel at 3.45pm it's been moved to the dining area food court because the outside there's rain so please be there downstairs we'll also have ushers ushering people in but if you had food downstairs that's where the OTR session is before the next speaker goes up on stage I'd like to take about 45 minutes to quickly introduce the editors of rootconf and just walk people through the process by which we did editorial can I have Aditya I'll up here so this year okay so this year we have five editors for rootconf and I was the editorial chair which means I was just herding the cats Aditya Patwari who's been speaking at rootconf for I think several years now and it's been a great opportunity to work with him this year Philippips who's doing the second year as editorial he will most probably retire next year editorial for the third time we have three absentee editors and I hope they're watching us on video I have a lot of gratitude for Saurabh Hirani who has taken a lot of time and has taken rehearsals and sessions from Singapore and given very granular feedback to attendees Sezyagni Khanna who's not here who's another conference in the US right now who's also been a past speaker and Piyush Varma who also spoke last year who's an editorial also in the US editorial team of five people with myself we started curating talks from early February we opened up a call for proposals and received a total of 95 proposals including talks for DEF CON and the way we structured the editorial process we did a bunch of things differently we did a lot of public commenting on proposals where we had questions where things were not clear practically every proposal submitted draft slides and a two minute preview video for us to assess both the person's content as well as the presentation skills and before we confirmed any talk we did a rehearsal in order to double check for ourselves whether we were making the right decision or not so between March until I think last Sunday we were rehearsing with speakers practically every speaker who's gone up on stage here and who will speak at DEF CON tomorrow has gone through rehearsal and in some cases even for so apart from myself who's employed with Hasgeek Aditya, Trouble and the others three are full time employed in other places we've been doing constant calls late nights across continents and trying to coordinate all of this so we definitely would appreciate if you give us feedback on the talks if you think anything has not worked out right that could have been done better we always try to improve the editorial process and keep it transparent and be happy to talk to you if you'd like to propose to speak at any point in time at DEF CON, at Meetups and at conferences do reach out on the help desk I'd just like the audience to give a round of applause to both of them and to the absentee editors thank you very much I have a lot of gratitude and appreciation for Aditya and Trouble to do rehearsals and to be very brutal in feedback so don't run away Zainab so I should point out that Aditya and I and Yagnik and Piyush and Saurab we just we are gainfully unemployed and we just idle around we never show up for conference calls the only thing that makes this conference work is this voice in the background and on Trima and on email are you coming to a rehearsal have you read this all throughout the day quietly sitting on your shoulder like an external moral unit and this lady here who makes it work so don't applaud Zainab she makes this work complain at us praise Zainab well this was fun that's all I have I think it was a good learning opportunity best part is that now I can roam around because I already have a technical approach one more thing so I will be forced into editorial retirement next year because apparently I think you're only allowed two or three apparently I'm not yet allowed into editorial retirement but if you think you have the soul crushing ability to rehearse speakers and to deal with the voice of Zainab at all hours of the night I would encourage you to volunteer for next year's root confeditorial we'll break you in gently and then next year or the year after rather you'll be on your own so if you think this is something you want to do come and talk to us it's a lot better than being talked into it so volunteer now or be volunteered okay just a last quick note before I take center stage our editorial policy is there are two key things that we that we do very diligently one is past past speakers are eligible to become editors for future conferences very active community members are eligible to become editors for future conferences trouble has not spoken at any Hasky conference or at root conf but he was pulled into it because of his involvement with FOSDEM and a lot of other conferences and we felt that it was a good decision to bring him on and to streamline the process the second thing is nobody continues as an editor for more than three terms you can either do three continuous terms you can do one year take a break come back another year take another break come back another year or you can do three continuous years but you will be voted out after three years because the only tyrant is until I'm replaced I'm the only tyrant okay so thank you very much for your time and yeah do talk to us if you think that there's anything that we could do better we already have a blog post that Phillip has written and that's on the root conf blog about how we select the talks for root conf so do check it out we try to be as meticulous as possible if there have been slip ups you are free to fill your feedback form and comment others up front thank you very much alright our next speaker has come all the way from Mumbai and his claim to fame is that he may in fact be the nizam of Hyderabad you can ask him about this later I'm not sure what the claim to fame how that works but this is what he says so he'll be talking about provisioning for bill metal servers as soon as his microphone is working yes very good so I present to you Azhar Hussain right good afternoon everyone thanks for joining so afternoon sessions are special right because you've had your food you're settling in to the spot now and you know a perfect lullaby session is all you need so you know what I'm going to raise up so quick show of hands people related to IT here seriously right so people who have got half their wardrobe from free stuff that's surprisingly few I'm not going to take it back how about people working day after 12 o'clock right ladies and gentlemen I gave you the sys admins in the room so let's get started a little bit about me I'm an operations engineer at endurance so my day to day job involves working around a bunch of vps solutions bare metal servers and dedicated servers when you boot this system into pixie boot you're going to get a screen which gives you two particular options which is the default option and the other one would be sent to a 7 so the default option basically means it's going to boot using the local disk of the system of the server itself and in case you decide to go with the sent to a 7 option it's going to start installing the sent to a 7 ISO so razor does this thing slightly differently in the sense that it uses ipixie now ipixie extends pixie in a few ways the most important of which I would say is the ability to use HTTP directly instead of having to stick around with tftp so finally jumping into razor here before we jump into the depth of razor and figuring out how things proceed over there we'll just take a quick look of basic razor installation so basically you need a dhcp server you'll need a tftp server you need to set up your dhcp config according to the pixie configurations over there you'll need to set up something called the microkernel into your tftp serving directory and finally you'll need a bunch of razor server and client packages and it's going to use postgres as the backend store so most of these things are fairly simple and straightforward steps and at the same time you always have puppet modules available so that's going to automate the entire thing for you so at an age where we have so many existing tools available for network booting what was the point of creating another one and why should one start to even think about using razor like what is the point of razor so now the whole philosophy of razor is the need for wanting to be able to consume physical resources like it's not real so you want to consume your hardware servers, your bare metal servers like it is not real, you want to consume it like it's a virtual machine the way you have VMs in aws and open stack where you specify you know what give me a 4x4 machine with 4 gigs of RAM what hypervisor it's going to be on or you know what kind of hardware configuration is underneath similarly you don't really have to know what hardware specifications are you know what is the latest density RAM which you can plug into this particular system as long as that system has the set configuration give me a system which has 40 cores and something which has more than 30 gigs of RAM and I'm happy with it I don't really need to know whether it's node 1 or node 2 so you have another aspect over here which is the whole cattle versus pets philosophy where in a lot of tools treat their servers as pets you're supposed to configure them up with names beforehand like the way you do it in cobbler, you're supposed to specify you know what this is the system I'm going to be setting up for this particular use case and you specify the market and you specify the profile and only after that your system starts to boot as opposed to razor all you have to do is, given that you have a system, 3 of which have 4 gigs of RAM you need to say that give me a system which has 4 gigs of RAM and it's going to pick one of those systems and it's going to set it to boot so you have a bunch of components present in razor so which are quite necessary to understand what they are to figure out where things are proceeding further to start up with you have something called the nodes basically any hardware node register under the razor server for management is called the node next up you have is something called tags a tag is essentially a unique name coupled with a matching rule so let's say if you have a tag of web servers and your matching rule is something called you should have number of cores greater than 5 so any of the nodes which have more than 5 cores they're going to be attached with that tag of web servers next up you have is repositories so repositories is nothing but an object representation of your IOSO what your installation disk is next up is policies so policies is the glue in the middle this is what matches the tags to the repositories and this is what is going to install the actual operating system onto the server next you have brokers and hooks so strictly speaking brokers and hooks are not essential components to razor so what broker is essentially is since razor has a very clear line between the time the installation is done and the time when your config management needs to start off so you have the mediatory thing called brokers so broker is going to hand off once the OS has been installed to your puppet agents or your chef agent where it's going to start setting up the server according to your config management hooks is basically letting you allow to run arbitrary scripts at different points of time to give you an example of how hooks could be put into use so we have a lot of management present on our switch on our switch sites so typically you want your deployment to be done on practically private subnets over private VLANs right and finally when the system is going to be put into production it's going to go over public VLANs over public subnets so you can have a hook fire into your June OS or your Nexus switches at the end of the OS installation and it's going to change the VLANs present over there into the public VLANs and your public subnets are given that you have specified the IP address over there and on top of that it can apply whatever shapers and filters you want right so I might have mentioned about microkernel quite a bit back so what this is is it's essentially a CentOS 7 kernel loaded with puppet factor and the point of microkernel is any system which has been pixie booted and is brought into razors management is going to be loaded in with the microkernel so what the microkernel does now is it's the idling state of any node present in the razor system so the microkernel collects all of the facts from the hardware and it's going to dump it into the razor db now when the facts are present in the razor db the tags can have an async call run over there and match whatever nodes according to the matching rule right so if you're looking at the flow of razor here so any system which has been pixie booted is going to be loaded in with the microkernel the microkernel is going to go into the system it's going to collect the facts from the system it's going to collect your details about your physical hardware of the sort of what sort of disk do you have do you have SSDs or do you have hard disk drives whether you've got what kind of processors you've got and the counter processors you've got whether you've got hyperthreading enabled on them stuff like details about your network card your memory options so on and so forth and finally once all of this data has been collected there's going to be an async operation going on in the background which is going to check every single tag and its rule correspondingly with its nodes and if there is a match over there the nodes are going to be attached to the tag and there's a sequence of steps which is going to follow on as soon as the tag is attached if there's a policy attached to that tag the policy is going to be attached to that particular node and once the policy is attached the system is going to reboot now and this time it's going to be loaded in with the particular operating system which needs to be installed and after that you have a bunch of kickstart files and pre-seed files if you may which are going to be created on the fly and finally once the OS installation is done the brokers would be given control of the node and at the end of the complete installation there's going to be a flag which is going to be set which is basically telling the razor node that this system has been installed and you do not need to reinstall the operating system on this particular node in case the device has been set to pixie boot and rebooted again so I've got a fake live demo of this process here basically this is how all of your nodes would show up in the razor database over here so I'm basically firing a bunch of razor CLI commands to the razor server so you can see that there's a bunch of nodes already present in the system here with their MAC addresses and a couple of them have tags already present over there and corresponding policies as well so this is a brief description about one particular node so you can see that the flag has been set to false if you look deeper into that particular node you can take a look at the facts so this is the entire factor dump for that particular node so you get details about your processors whether it's a physical machine or not your architecture if there is an existing operating system installed on it what kind of operating system is installed further on you get details about your partitions as well whether if it's present in a rate configuration or not so moving on next we have the definition of a razor tag so all objects in razor can be put forward as a JSON object and directly imported so over here you can see that the actual check is the number of processors should be greater than 20 and the label has been put as demo so if you list all the tags present on this system you can see that we have one demo which is the number of processor count is greater than 20 it has one policy attached to it but it doesn't have any systems attached to it at this point because there's no systems present in this razor dv which has more than 20 processors if you look at the definition of a policy here you can see that the repo and the tasks have been assigned over here as well as the broker so we've assigned the no broker at this point right and you can have very generic stuff present over here as your root password and the max count would be the maximum number of nodes that you want to be installed into this particular policy so let's say you have a caching server policy and you don't want to have more than 5 servers present at any point of time so you can apply that a max count of 5 is there so finally you have a very important thing over here the tags array which is basically all the tags to which this particular policy needs to be attached to next up is a very basic definition of a repo which is basically the endpoint from which it's going to be collecting the ISO from for the first time eventually it's going to be cached in the system another interesting thing about Razer is it also manages to store all the IPMI credentials for each and every node the fact of storing IPMI credentials for all the nodes over here is that now Razer has control of IPMI tools and it provides a direct wrapper of using IPMI tools and this is beneficial because now you can directly extend Razer's API and call IPMI tool commands directly from there so basically if you have any power state information present in the system so basically you can specify in Razer that you do not want this particular system to be powered up you can specify that this particular system needs to be on a desired power state of off and finally you can see that this is the installed flag over here which says that it is false in case it was said to true this system would not be pixie booted at any point of time right so finally I'll be getting into what was the use case present here and why we decided to go ahead with Razer right so first of what we needed to do was we needed to install a whole bunch of operating systems and which wasn't specific to one particular flavor including windows next was the fact that we needed to provision systems based on rules where you can specify that you know what this is the configuration I want and give me a system of this kind and I don't need to figure out you know what what kind of servers do I have and then figure out in that list and then start spinning up servers from there next was a discovery and inventory management system so typically in a DC operations environment you end up having two systems you have a DC system something like rack monkey or device 42 where your DC ops team is generally going to be filling in data and let's say after a hardware upgrade it's the onus is on them to update it manually as opposed to Razer which inherently has a micro kernel present so the next time the system is rebooted the micro kernel is going to be loaded and any hardware changes that have happened over the time are going to be reflected directly we also had an API which Razer has an API which is completely restful and quite extensive which is also quite useful for our purpose and finally the complexity and ease of maintenance part of it so this particular project was a lot of operational heavy rather than having a lot of dev work on top of these boxes so typically we didn't need any config management system so we just needed to install a bare bones operating system on them and then give it off so here's a couple of alternatives we evaluated over here so we started off with cobbler because we've been using cobbler for a while around here so cobbler doesn't have a direct rule based engine by default and it doesn't even have a management system a discovery system of sorts so which is why cobbler is not big for this particular use case next up was open stack ironic so this is a typical example of the cost not being justified for the returns around here because this was a completely new DC and having figured out the intricacies and problems dealing with open stack clusters we decided not to go ahead with open stack ironic and finally there was foreman so foreman practically does everything which Razer does and a lot of other stuff as well so this was actually a split hair decision the fact that Razer can just do this particular part and it based completely around the rule based engine was something which was which tipped the scale in the term of Razer but if I think about it if we had a typical backend architecture where you have your web servers, you have your application servers and the whole lot over there in that case foreman would probably be a good option as opposed to Razer so in the end a bunch of these tools do a lot of things but they do these things specifically well so we needed something which was quite generic and basically throw we can throw anything at it and it should be able to install it yeah I guess that's about it so when you're using Razer did you work with I mean did you try provisioning a UFI machine I don't think so no so that you can only do it in legacy bios mode yeah now that I've tried it off but I'm not sure at this point I know it doesn't work so I just wanted to try it that's where foreman would actually work better alright thanks everyone I just wanted to let you know that if you have received a red lanyard and if you would prefer to have a white one there's now an additional supply of white lanyards so if there's a difference to you you'll be very happy to know this the red one is for when you don't want to be photographed okay so if you don't want to be photographed and you have a white lanyard then get a red one yep okay and in a reminder we're about to have our beverage break this is your last chance to submit your flash talk so put it on the white board out there and be ready to present at 5 20 p.m and that's it enjoy your break and we'll be back here at what time is it 4.35 p.m enjoy Mr. Philip Papps am I on? yes alright let's see if I can walk over any cable alright so this presentation is entitled 3DSD is not a Linux distribution and whatever you take away from this presentation you should be sued by the fact that it is not a Linux distribution am I feeding back? it is not a Linux distribution so you can relax, calm there's no anger nothing to be afraid of it is not a Linux distribution everything is fine and we've had a lot of Linux presentations at this conference I think there's a parallel track about it and I'm just here to tell you that you don't have to live this way it is in fact possible to find a unix operating system which does not make you angry comes with all sorts of nice features and does not present any surprises along the way so thank you all for the introduction my name is Philip Papps I'm wearing many different hats the hat I wear today is the director of the 3DSD foundation HAP the 3DSD foundation supports the 3DSD project in many interesting ways we have meals to hardware we are a charity registered in Boulder, Colorado in the US and we take money from companies and individuals who use 3DSD and feel that it needs to be improved and we spend this money on sponsoring conferences and organizing BSD conferences and arranging travel for BSD developers to conferences around the world we also spend our funds on hardware and improving support for 3DSD for different kinds of hardware and it's kind of platform so if you actually use 3DSD is anyone in this room using 3DSD in production or yes good the usual suspects anyone else ah good very good so at least a handful of you if any of you happen to have too much money lying around and you have fiscal incentives to get rid of it the 3DSD foundation will be happy to relieve you of this burden and use it to improve 3DSD so that was the 3DSD foundation so who so you know that was the organization who am I I'm a journal hacker I live in the mystical world below the system fall layer and I'm also a conference organizer repeat offender I apparently I cannot ascend the conference without somehow ending up organizing it or doing something I don't know how this happened it just happened I'm also a consultant I'm in the business of telling people that they're doing it wrong and getting paid for it but my my domain where I live I live mostly in device drivers I live in real-time operating systems I also have a troubled history with electronics particularly radios and I'm a professional paranoid so as we go through this presentation the professional paranoid and you know the hardware the simple hardware mind creeping in from time to time so bear that in mind also this presentation was originally written by George Middle Neal another 3DSD foundation director who has many similar properties in his background except for the electronics so on with the show what is this 3DSD thing you know is this Linux you know you type 3DSD into a search engine and you see this little devil guy smirking at you are we some weird satanic cult no we're not we produce an operating system a complete operating system unlike some other open-source operating systems we feel we take a more holistic view of what an operating system is we don't just produce a kernel we don't just produce a bunch of tools we don't just produce a bunch of libraries we actually produce an entire operating system with all the build-glue you need to build it yourself and to build your own release of it so if you're familiar with Linux you might have heard the word distribution once or twice and it makes you cry a Linux distribution is a collection of tools written by different groups of people with often conflicting interests and priorities and somehow it works 3DSD is a complete operating system produced by a team of people who feel that they are producing an operating system there are no conflicting interests in the 3DSD project the people who maintain the kernel are also the people who maintain the C library the people who maintain LS and staff all that sort of goodness which I understand lives above the system call layer we produce our operating system and we also produce the tools to build it and to maintain it if you look at how 3DSD will come into that in a moment you get everything you need basically if you have 3DSD you are ready to code as soon as you install it 3DSD is a physics-like operating system and at last count we support 24,000 third-party packages in our fourth tree or our package system so any piece of software you are familiar with on other physics-like operating systems and even Linux on 3DSD so you package install nginx and you have nginx just like you would expect it on another operating system and in addition to all this operating system tools and third-party glue we also document everything so I was smirking some of the presentations earlier this morning I think Manan in particular was looking up what is the CPU load average what does that mean and typed it into some search engine and we went on to stack exchange and many layers of searching later he found out that the answer was well nobody really knows what the CPU load average what that means the previous project feels that you should not have to go through this so we provide documentation so if you want to know what is the CPU load average you can do man-minusk uptime or some other tool that gives you the CPU load average and it will tell you in the run queue and it is determined by et cetera et cetera so we have documentation that will hopefully make your life easier and we are also an open-source community so the operating system does not stand by itself it's a community of like-minded individuals that produces the system and we hang out at conferences and we turn up in places so I'll tell you a bit about the community as well so okay who uses this 3D as the thing there are 5 people in this room or 6 people in this room great I don't know if your names are on here but at least I hope some of these names should be familiar to you I think particularly WhatsApp is quite popular in India I think it's a messaging application for phones they run on previousd entirely so all of their servers all of their stuff runs on previousd and they admit to running previousd another example of a company running previousd is NetApp if you have a lot of data does anyone use NetApp probably not willing to admit to it NetApp is a big company producing storage systems for people with a lot of data who care about data their on-tap operating system is basically a fork of previousd and they just track previousd Panasas and Dell well Dell EMC have similar sort of use cases but really it's just previousd so a lot of these names are in the storage industry so Panasas, Dell and NetApp wherever they are at the top that's all storage so that's one area where previousd is very popular another area where previousd is popular is networking so Juniper Network is the usual example Juniper makes routers and switches and other networking equipment other stuff that lives there and their Junos operating system is basically a fork of previousd the control plane of your Juniper router is basically a previousd machine which talks to the actual forwarding engines other examples Yahoo Mail runs on previousd it still does it's remarkable that it works and then Apple is another good example of someone using previousd Apple takes the previousd user space and parts of the kernel merges it with the mock VM system and mock ports and some other stuff add their own secret sauce and they call it Mac OS X I call Mac OS X so I call it pretty via the Unix for the desktop so that's you know oh and Netflix of course thank you Todd another company in the networking industry Netflix I think is responsible for about what 30 or more percent more than half the internet bit flowing across the internet on any given day all of those bits flow across previousd machines Netflix's caches are basically previousd machines tuned to deal with this massive storage in such a moving picture pushed into previousd machines so all of these are big companies so previousd is not this marginal thing which doesn't really exist and it's not been dying since 1998 or whenever slash off came up with that previousd is not really real and is being used by many companies you might be wondering why are they using this well you know why are they using this they are using it because previousd has a history of innovation and gradual innovation over longer periods of time so the previousd project will think something up and we'll develop it and then you know five years or ten years later we'll find it in other operating systems like you know Linux we produce great tools the previousd is more than the sum of its parts companies who just like the fact that they can take apart the previousd and use it in their application people in this room who shall remain nameless have shot parts of previousd into space people who shall remain nameless have used previousd parts of the previousd operating system on embedded networking devices the components of previousd just find their way everywhere and the tools are great and they stand alone well and previousd also has a very mature role so we'll historically we released when it was ready and that didn't really work very well so about ten years ago we started releasing software roughly every six months we say you know maybe it's time to roll a release so we go into GoTrees and we go into Flush and you know we polish we add fit and polish to our operating system and then we release it and we move on so we have a branch development model which I'll talk about a bit later and this model is easy for companies to follow so if you are the Juniper of the world or you are a Netflix of the world you want your upstream operating system easy to follow if you can talk this with say Linux where every couple of weeks someone throws a kernel over the wall and you know maybe it works, maybe it doesn't maybe it boots, maybe it doesn't and the kernel then goes into conflicts with the other release models of say the sea library which has the same release model you know maybe it's ready maybe it isn't and if you've ever built an embedded Linux operating system you know that you can know three versions of Lipsy the Linux kernel and I don't know the compiler will ever want to work together you don't have this problem in previous because all of us have the same goals we all work towards the same operating system we don't fight companies who use previous we also really appreciate our documentation and the fact that it's available in many languages George originally gave his talk in China and he said that hey the previous handbook our main corpus of documentation has been translated into Chinese and it's a very well maintained translation I took a look at the Indic languages and unfortunately the only official language of India which has a good translation the previous handbook happens to be English so if any of you like translating documentation you are encouraged to join the previous project I'll talk about that in a moment and you'll find useful the previous project also has a very business friendly license which I don't want to talk about for too long but if anyone has ever in the GPL you know that it is massive and full of scary traps and dragons the BSD license is about 300 words and it basically says here is some software use it if it works well for you great we'd appreciate a note but it doesn't feel any pressure our community is also something I like to advertise the previous project has many many mailing lists and when you join them you can usually find a warm welcome you send the patch and people will thank you for the patch and if the patch is wrong they'll shout about the patch they're not going to shout at you I'll talk a bit more about the community later on so these are all very compelling reasons to use the previous BSD so how did we get here George is in 3 minutes I'll talk a little bit more but the history of Linux is well known this guy in Finland had an itch to scratch that's why he's an operating system and that's what I've told a little bit the BSD history is a little bit more let's say measured or a little bit more lengthy at least so in the beginning dinosaurs roamed and chaos reigned that was the 1960s and then eventually people realized that we need operating systems no sign of BSD yet but someone came up with MULTEX everything is fine cruise is secure time sharing very good and then some people with a strange sense of humor said well we've got MULTEX we should have UNIX as well and then things got a little bit still no sign of but then after a year or two people started using the UNIX thing from their labs and they discovered hey this is some stuff but wouldn't it be nice if we have nice things so nice people at Berkeley started producing add-ons and patches to the UNIX operating system from their labs so people found useful like say X and VI because the UNIX editor was N can tell you that it's not fun X and VI are a lot more fun so that was one of the things that was on the Berkeley tool stage another thing on the Berkeley tool stage was a Pascal compiler it turns out it's not everyone likes writing T but a Pascal compiler was one of the Berkeley tools and other tools just came up and then in the early 70s someone had the bright idea that maybe these computers could talk to each other instead of people being angry at computers sincerely angry at computers so DARPA the defense advanced research project agency threw some money at the computer research group in Berkeley and said you know build the internet I don't think they said that but I only got three minutes for this slide so build the internet and out of the CSRG came things like TTP IP UDP and all of these protocols which we are now well we now know and love then in the 90s we had a bit of a pick up where the people at the lab said well you know this DSD stuff is very interesting but this is actually our stuff and it's the first one to be using it so there was a lawsuit that distracted people from DSD for a while and people started using the next system never mind eventually 3A6 DSD came up, we have DSDI and then we have finally DSD is free after all this time and it quickly turned out that DSD was not just one thing so we have DSD historically a bunch of people at Berkeley here's the Unix have fun with it or here are some add-on tools for Unix have fun with it when DSD was finally free at the early 90s it turns out that there were two or three competing views on how this DSD should be free and we came up with a couple of different DSD projects we have the NetBSD project which felt that DSD is very nice but the best way to use DSD is on everything from your toaster to your main claim to everything the previous DSD project felt well yeah that's nice we like portability but we actually would like to be fast so they disagreed with the NetBSD on where the platform works there's two different DSD project forms the NetBSD project going off to be portable and it's running on every printer in the world these days and we have the previous DSD project which runs everywhere else in 1986 the OpenBSD folks had a disagreements with the NetBSD people about priorities on security and portability so OpenBSD forced off and a few years later we also found your force forced off in less than two months the previous DSD was a powerful tool which takes the previous operating systems just like Juniper does or just like any of the other consumers the 3DSD does and they optimize this for running on a desktop Apple does the same thing and they optimize this for running on a desktop and there's some smaller DSD distributions all that aside I'm going to talk about them I'm going to talk about 3DS what do we do? We produce a whole system what do I mean with a whole system? I mean that we produce an operating system which comprises all of the device drivers all of the compilers and associated tools debugging tools as well I don't know if anyone has ever tried to debug Linux but it's just you can't certainly not as soon as it's installed try and hope we have editors in our base system and we have a packaging system that allows you to install anything that is not there so the point of what we actually produce as a 3DSD project is an entire operating system that is ready to code when the install is done obviously your definition of code will depend on what layer of the stack you operate on and our definition of code is you can write programs in C or C++ as soon as your install is done provided that your editor of choice is either X, PI or nano and that's ready to code when the install is done we also have a packaging system so if you prefer another editor like them or you like to live a life of pain and you use Emacs you can just package install these things and have them but the whole system approach basically means that what we deliver is known to work at least together with itself you never have this awkward situation where your compiler is not willing to produce anything that your assembler likes or your linker doesn't like the object spat out by your compiler or your debugger does not understand the particular variant that else that comes out of your linker those are all things I've encountered on Linux by the way and we produce this as a team we work together on this and we have one bug tracker for all of these components and this allows us to deliver a polish operating system that just signs very nicely so if you write a device driver for instance you don't need to go and write a tool for that device driver to dribble some bits you can just add it to if you think whereas in Linux every single network device driver has some user-based tools to fiddle a bit there's also ETH tools but there's at least four different variants of ETH tools so it's really easy we don't have this because we have this holistic approach we produce the whole stack and it's not because you're in device driver land that you are somehow forbidden from developing if you think sometimes I wish that were the case if you think it's not my favorite use code but there's nothing preventing me from going into if you think and adding something for my particular device provided of course that architectural constraints are met so I'll talk a bit about what parts of stuff we have in this whole operating system we have some file systems which are real selling points for previous we have two files we have a whole bunch of file systems only two of them are really relevant we have the UFS file system the traditional Unix file system which is essentially the same file system which is on Solaris or any other Unix because they are basically derived from our original UFS UFS is it's a rock solid Unix file system so it's predictable it's not necessarily always fast depending on workloads but some workloads are actually high performance and previously UFS is high performance on read and write under certain workloads our UFS also has snapshots I think Solaris has UFS also has snapshots now and we have journaled soft set data we have a terabyte of data on a UFS volume it doesn't take three weeks to be FSDK but if you have a terabyte of data really what you want is you want ZFS and ZFS was originally developed by Sun with open source that had an interesting history but it is the only file system you need if you have a lot of data and you care about your data ZFS is really the file system you want ZFS is a file system and the volume manager combined it has snapshots which are copy on write they take no time at all ZFS destroy is instance you can instantly destroy your data also unlike any other file system I've encountered ZFS is aware of the fact that your disk and it does not trust your disk in the background it snapshots all your data and it just checks that the disk is not lying to you because disks lie and every other file system just doesn't get this just assume that the disk is telling you the truth and that data does not get silently corrupted if you care about your data you really want ZFS I believe there is a ZFS for Linux as well I have never heard anyone using it successfully or I've never heard anyone who is using it not complain about it ZFS also works on Ilumos Sonos or Solaris or whatever they are called this week of course the ZFS is slowly becoming the reference well not the reference implementation but one of the more mature implementations of ZFS so if nothing else and you care about your data ZFS would be one good reason for you to go and install ZFS you still need backup but you might not need to restore them quite often ZFS also has a lot of security features there are a lot of talks about containers and container systems and namespaces all sorts of things and all of those are just cheap ripoffs of what previously called jails well cheap ripoffs done wrong as far as I'm concerned so if you want lightweight virtualization and you don't want to ask the question what should PID 1 be in my container maybe you should look at jails so for the record PID 1 is whatever the first process runs in the jails it catches big child then your zombies will be reaped so jails are lightweight virtualization basically it takes the charoute system call from previous z or from unix rather and it teaches it about networking so it just constrains the power of the root user of it beyond the file system so it's a charoute with an IP address or any number of IP addresses we also have mac and audit frameworks which you are in an industry that cares about security and cares about knowing what happens in an unforgeable sort of way mac and audits are things which should be familiar to you or should be something you care about previous z has distributed auditing so you can track every file system event every network event gets an audit stamp and that can be sent to another machine so the logs can't be tampered with mac can also constrain other things so I've just been reviewing a kernel module for real-time priority constraints so that you have your process you can use real-time priorities without having to be rooted on your system so mac can give you more privileges or remove privileges it's like sce linux but done right we also have capsicum in previous z we also have capsicum in previous z capsicum is a research technology from Cambridge which tries to sandbox individual applications it works on a capability system and a capability is a privilege you have which you can either shrink or delegate but you can't ever expand it so once you enter a capability sandbox you are constrained by that sandbox so use cases for this are things like decompressors or things like tcp dump where if you're running tcp dump and you're getting some protocol on the wire you would prefer not to have this protocol exploit bugs in your shell and suddenly run random code on your system capsicum allows the disector to run in a sandbox that does not have the privilege to such files tcp dump can still write to the terminal but the disector can only communicate with the tcp dump through a well-known or well-defined and predefined IPC mechanism with compressors if you have this compressed file which you downloaded from some dubious source and you start uncompressing it you don't want to bug in zlib or a bug in l.ma or whatever compressor flavor of the week you have you don't want that to suddenly go and blow up your system so that's some of our security features we also have good compiler technology we moved to LLVM and clang years ago so GCC is no longer part of our world so we have a lot of compiler in previewsd not just compilers but also debuggers I don't think anyone here cares about the t-toolchain so I'll move on Dtrace I think someone asked this morning a question about how do I know what my process is doing at any given time previewsd has Dtrace which is a dynamic tracing framework which also came from Solaris originally I think Todd gave a workshop on it here last year basically Dtrace makes the blue smoke to all this code going on and it shows you what goes on inside your process it tracks system calls it tracks library calls function boundaries you can just you see what your code is doing if you're trying to debug on another operating system you're basically you're going to be doing printf or you're in the debugger single stepping with previewsd you don't have to live this way we have Dtrace on Earth is my process doing Dtrace oh yeah it's doing that so Dtrace is another big selling point of previewsd and if you maintain complicated systems then Dtrace is definitely something to look at I believe the JVM has been thought about Dtrace a long time ago so that might be something to look at I ramped up earlier that previewsd in general has a networking history CTP and IP are Berkeley technologies previewsd in general is still the reference implementation of many networking technologies we have fluggable CTP stacks if you don't like the default CTP stacks it's another someone has recently distributed the BBR CTP stack I don't really know what it does I know Google is very fond of it we have racks we have the usual Cubic and new Reno CTP algorithms to have applications which run on CTP and you care about tuning congestion control in different ways previewsd make this really easy I think you can do this on Linux as well but every time I've missed the Linux CTP stack I want to describe we have three firewalls you'll find that only two are listed on this slide the maintainer of one of these firewalls is right there we have IPFW which is a very mature, very good firewall it's been around for ages the only thing that's wrong with it is that this syntax is not really always friendly and then we have PF which we imported from OpenBSD a long time ago it's got a much friendlier configuration syntax I have done a PF workshop and I've had people type a PF configuration file without looking at a single manual page by just guessing what the command line syntax would be or just guessing what the configuration file syntax would be one of those that compared us to IT tables does anyone know how to add a simple network address translation rule to IT tables without looking it up shut up minus J, master rate minus Q, jump through 6 hoops to hope for the best yeah we've also got something called DummyNep which is a very useful for people who test things that run on networks DummyNep can pretend to be wider or narrower pipes that you can introduce latency on your network and it can also do queuing so if you want quality of service, DummyNep can help you too but it's primarily a network testing tool that can introduce arbitrary latency on your network we have some other networking tools that I don't have enough time to talk about all of them in depth we also have virtualization things like virtual machines and things, we have that many of you use Unreal hardware Amazon and digital ocean and things that's great, that's wonderful 3DSD releases are available for VMware, for virtual box for QMU and for Hyper-V they will just work on cloud provider of your choice since I think 10 point something release we also have official Amazon AWS images so if you are an Amazon consumer and you would like 3DSD I think it takes you 5 minutes well it takes you 20 minutes to create an account and then it takes you 5 minutes to boot your first 3DSD machine roughly ish and those are official 3DSD images so you can just do the usual 3DSD tools 3DSD updates, etc to update your system we also have VHIVE which is a native virtualization tool I already mentioned JAIL which is lightweight virtualization VHIVE is the, well I would say the more like equivalent of KVM but it's a lot lighter weight it's a lot more consistent so VHIVE is all the multi level so if you want hardware system virtualization you should look at VHIVE I typed this slide earlier because I realized that I need to sell this thing to all these Linux victims we also have something called system call translation I call that Linux personality disorder if you have a Linux binary which for some reason you cannot compile with package source whatever this is 3DSD you can just run that Linux binary on 3DSD and in the vast majority of cases it will work fine and it might even run faster than on Linux so one of the few redeeming features of Linux is that it has a strict contract between the kernel and user space that is usually not violated and as long as the operating system presents a system call table that smells a bit like Linux a Linux binary will just run so 3DSD has a system call table that smells enough like Linux to convince Linux Elf binaries that oh yeah hey it's Linux I'll just run and that allows you to run a lot of binary only things that are inexplicably only delivered for Linux such as Oracle's database or various cab tools from people like mentor or eagle it also works well for you've lost the source code and you only have a Linux binary ok it still works as an easy sort of gateway drug into 3DSD and you might find that your binary runs faster on 3DSD than it is on Linux last time I thought it was some game I don't remember which game it was but it ran faster on 3DSD than it did on Linux despite being a Linux binary some newish features that are in the pipeline we scale pretty well already, 3DSD is a massively scalable operating system we scale to many cores but we have this 256-core ARM thunder machine sitting in a rack and it takes 40 hours to boot that's being worked on so scaling to more and more cores 200 cores, 500 cores, whatever a lot of memory that's all so it works up to reasonable limits but they're always room for improvement and that's being worked on as a priority NUMA there is support for NUMA in 3DSD it is ever improving we have ARM 64 support it runs, I've seen it work we also support a number of experimental network technologies like multi-pass TCP and data-centric TCP and again this congestion control system TCP algorithm from Google should you be running on bare metal as opposed to a virtual machine 3DSD also supports secure boots or known as UEFI so you can run on constrained firmware without any difficulties so that was what the 3DSD projec