430 mark, a couple of people coming in. Hi guys, awesome. Welcome. My name's Josh, or if you want to be really formal, Josh Calderimas. You can find me on GitHub or the Twitters. Or you can just call me Mr. Calderimas and find me either. This is me with Anthony. He's a sexy guy. I'm from the Travis team. But before we get started, it's the afternoon. You've just had some donuts and coffee.  You have everyone, everyone in the back with a laptop. Yep. Yep, everyone in that section. Yep, perfect. OK.  So it could be Rails, it could be an agent. You were using logging. I'd expect this. There's one person that sat down. That's a shame. I want you to stay standing if you're using New Relic in any shape or form. OK?  For example, libretto or graphite, statsd. Can we stand back up? No? Sorry. OK.  Boom goes a dynamite. Thank you, guys. OK. And the people that were standing up, you can just feel good. And maybe I'll give you a sticker there. Just feel good. OK, now you can sit. OK, so two small things before I get my proper talk started. I want you to meet the Travis team. This is technically our official outfit. We wear a onesie. You join the Travis company and we give you a onesie. I have a feeling my microphone's going to fall down. There we go. This is perfect for coding in. And you can buy them online for a good price of $120. We'll even put a Travis logo on the back. We used to wear shite shirts and awful moustaches. Don't tell Sven I said that. Beautiful moustaches. And we used to present together in them. But now this is a side note. Is there any one from the engineer out here? OK, well, thanks to the beauty of Ingeniard, they're actually sponsoring Piotr Sinaki, Drogas on Twitter. He did a lot of the Rails engines work and was working on Sproutcore. So he is now working full-time on Travis open source. So I just want to give a huge thanks to them because they're helping push Travis and the open source testing part of it forward. So thank you, Ingeniard, for being another talk. It does all the way up. And there's a summer version. So if you know of anything about Travis, Travis is a crowd-funded company. We started roughly in January putting up a crowdfunding page. We raised about 125,000 US. And from that, it was from about 780 individual donations from people in the audience here and from at home. 29 companies also joined in to help us get going. Everyone from Ingeniard to Heroku to Bendyworks to SoundCloud. And this was only a fraction of them. So these are the people that have helped made Travis possible. So we want to give a lot of love and thanks to anyone here who has donated, anyone who's donated or who's outside of the room as well. But a big thing for us, and we haven't actually released it officially, but thanks to the beautiful people at Sticker Meal, they have in the past created a whole lot of stickers for us. So if you donated, you would have got these beautiful stickers in the post. But thanks to David, who works at Sticker Meal and is a pretty dog, we now have an official mascot. We have a Travis. And we have a whole lot of Travis stickers. So if you want to Travis Sticker later, come and get me and I've got one for you. But back to the talk. So as I said, my name is Mr. Calderimas. I come from this very far out place called New Zealand. We're known for hobbits and orcs, cable cars, beautiful views. I'm from Wellington, to be exact. Beautiful nightlife, amazing coffee, and the most fantastic amusement rides ever. And I've also been on one of those. It's one of those small planes where you can see out the front of the cockpit.  I can see a hill. I can see a runway. And then even while it's landing, all you can see is a hill up to the last second. The last four years, I've been living in this beautiful place called Amsterdam, where it's known for its huge amount of bikes, its amazing architecture, its nightlife, its recreational activities. But now I'm technically a traveling nomad. I've got no official home. I'm in between San Francisco, New Zealand, San Francisco, Berlin. I travel with a ukulele. I've got a nice little coffee kit. And today I'm here to talk to you about fun things like logging and metrics and monitoring. It's part theory, it's part code, it's part internal tools, and it's part moustache. But you'll notice that the talk title in the conference planner was about the black box. And I had a huge issue trying to find a good talk title. Having a good talk title is what people look for. And I was at the American bookstore in the Netherlands. It was actually just before I was going to Sao Paulo. And I was like, you know, Josh, you don't know anything about Sao Paulo and you're going to Sao Paulo. And this was the day before going to Sao Paulo. So I had to run here to get myself a Brazil book so I actually knew what was going on. And I came across this book and it's not how good you are, it's how good you want to be. And it had these really nice little lessons in there. For example, fail, fail again, fail again, all of these things that were taught as programmers and when even starting a business. Like the person who doesn't make mistakes is unlikely to make anything. Or if you can't solve a problem, it's because you're playing by the rules. There are nice little lessons. So I thought to myself, since I'm doing a talk about metrics and what's going on inside your app, really it's not how good your app is, it's how good you want it to be. What I want you to take away is that your app, when you deploy it wherever it is, it's not just a black box. You need to know what's going on inside of it. Knowing what goes on inside of it gives you the power to optimize and to make things better. It's not about the worth, we're doing a million when requests a second. That is a very, very circle jerk kind of metric there. We want something, we need to know really what's going on inside it. And we've got a whole bunch of tools and libraries available to us. Everything from exception notifier to libato. We should be using these tools and even better, you should be building internal tools. Internal tools will allow you to mold the data and allow you to really abstract what you need. And on top of that, services are your friends. Don't build things when things are available for you to use. It doesn't matter if it costs 100 bucks a month. These services are worth their weight in gold. But anyhow, I come from this nice little crowd funded company called Travis and we run your tests for you. You might have seen a little status image in, you know, across GitHub. We run those tests. It's free, it's open source, it's full of fairies and unicorns. We've got another piece of infrastructure and we are for builds what RubyGems is for libraries. We have over 25,000 open source projects on us. We do 10,000 test runs daily. Over the last year and a half, we've done 700 and 7,000 build requests from GitHub totaling 1.9 million total test runs. And we support 11 different languages, everything from Ruby to PHP to Python to Go. I think there's eight different versions of Ruby, three different versions of PHP and Python. We've got it. If we don't have it, tell us and we'll add it. And if you see this, it's the sign that it's on Travis. It was started with fun. The whole idea was for instant and live and modern and hackable. And as any Rails app, it started kinda like this. It's just a simple Rails app that would run on Heroku. It would use, rescue, and it would talk to a dedicated box. We'd use Redis and Pusha and a nice, pretty database. And it's about getting things out there. This was simple and this did the job. It did the job when you had five projects, but not when you've got 25,000. So it kinda grew to this. This is a simplistic view of our architecture. We have TravisListener, which listens from GitHub, queues stuff to Rabbit. That then goes to Travis Hub, which will process the requests, send it to the database and via Pusha. The user will, of course, talk to Travis via Travis CI, which is the web part. And then all of the jobs go to Travis Worker, which uses virtual box. And I will just state that I want to stab virtual box in the eye, but we've got 10 of these boxes that we manage. And this works. This is how we have scaled. As a side note, we're splitting Travis Hub into three bits. So now, when you look at this, it's not four different bits. It's actually, you know, seven different bits. And on top of that, we've got other bits behind the scenes. We use everything from Redis to Rabbit to virtual box, JRuby and MRI. We're using an assortment of everything to make this job possible. And it's kind of built around a service-orientated architecture, but we kinda call it the mini SOA version because we are sharing services. And when we say services, I'm saying we share Redis' and databases between these little apps, you generally shouldn't be doing this. But anyhow, we'll get to this. Right now, we have a total of 10 deployable apps. And on top of that, we have four different environments. We have org staging and production, and we've got a whole separate setup for Pro just to keep everything a little bit more secure and also because org is sponsored, we don't want that to leech over to Pro. So we've got four different environments, totally 40 different deployments. I mean, that's a lot. So really measuring and monitoring is a challenge, but even more so, it's more important to get it right. We need to know what's going on inside of Travis so we can optimize and make it better for you, the user. Also, I used to play Canoe Polo. I'd like us to get to know each other. So let's talk about logging. So logging is important. Everyone here is doing it and they might not be knowing it, but you're doing logging. It's useful, but this is the standard Rails login. We've got, this is what, seven, eight different lines just for your production log. This is your default in production. What we really want is something simple. We want something concise and it should be easily readable. It should be one line. If you look at this, the Rails log, standard Rails log, we can actually break it down. We've got the request type, the URL, the format, the action, status, total duration, view, and database. Those are all there, but they're over eight different lines. Why not something like this? This is all we need, and even better, it's on one line. So this is what we use across our web applications and production. It keeps it nice and simple. It's keyword, we can search for it, and it means that we don't have super verbose logging going on. And you can find this available. It's from Matthias, one of the Travis guys. It's under roid rage log rage, and you can just add it to your Rails app and you should have better logging just for production. But what about non-web apps? That's not gonna work for a non-web app. It's very Rails specific. So when you're creating a Ruby agent or a Ruby tool, you want to think about how you use logging. Logging has everything from debug, info, and error.  It's not a huge thing to grasp. Really, you want to be logging all the things. Now if we think about Travis Hub just for a second here, this is one of our components. We're doing about 1,100 requests per minute using RabbitMQ and doing processing via threads and subscriptions. It has things like build finished, notifications, and it's doing background processing and threads. And this is what we have. You'll see that on the, or down here, these are all related log messages, and why they are is that you can see that the build idea is the same. So you need to think about context when you're logging and easy way to group them. If it be tagging, if it be putting an identifier or a UUID, what you want is any log message that happens in your processing needs to be easily grouped and attributed to a previous log message if that's applicable. And what you can see is it's a very simple kind of context. We've got the type, debug or info, the class, the method and any extra information, and the model and its ID. And for an error, it's just even simpler. Error, class, and error message.  But on Travis Worker, it's a little bit different. So that was the part that's actually processing the jobs, which is managing the virtual boxes. That's got five VMs per worker, which is effectively five jobs per worker at any one time. What the goal is, so every time you look at logging, you should have a specific goal in mind. And for our goal for logging, we wanted it to easily be able to see what was going on. And we needed it to be able to do it live because we don't want to disturb the application, we just want to tail. So in here, the context is the worker. And you'll see on the left-hand side, on the left-hand side, it says Ruby2. And then it's got each different class and then what's going on. So we can even see in a nice, clear, human-readable way about to run, about to perform, change directory, export, et cetera. We know what commands it's running as well. We can see it's doing a bundle install on the bottom. Being able to see that makes it much easier to just tap in and watch the build and make sure nothing's exploded. It's a much more verbose form of info compared to the Rails app, which is usually doing it on a singular request, but it's a different God. So you don't want to treat every app the same. And as I said, like everything's grouped appropriately, making it easier to read. And then all we need to do is draw a nice little tail, grep for Ruby2, and we can see everything going on with that worker. And even better, we can improve it further. We can consolidate all our log. For example, you could ask Sync to a single server. You could use something like grelog2, which is built on Ruby and Java, uses Elasticsearch, self-hosted, so you need to look after that. Or you can use something like paper trail, hosted and just pure awesome. And for us, now we've got all of our web stuff or our hub stuff going on in paper trail. Nice and easy to see. We've also got search at the bottom, and then you can go back in the time frame nice and easily. It's got excellent stuff like archiving notifications and web hooks when it detects a certain string. So now you don't have to build and alerting yourself. You can have it detect certain errors and then maybe notify you on campfire. And of course, we love Peter Trail. Oh, I also used to play Nick Bottom in the Midsummer's Night Dream. Thank you in high school. So exceptions, boom. So exceptions, you never want to check logs for errors. That is no fun and we're beyond that. When we're doing exception logging, we want to use an app, a service or a library for this. If you're not doing this already, there's a million available. And just to clarify really quickly, we've got saying as basic as exception notify a very old Rails plugin and would send you an email each time. You could do something like Sentry, which is self-installed or hosted. And this is built on Python, if I believe right. And it's very nice interface. And it's just simple. You've also got Irvit, which is a self-installed and written in Ruby. You can host that on Roku as well. For us, we use Haystack. So Haystack is built by the guys at GitHub. Sadly, it's not open source, but this is what you get. So this is an example of an internal tool that they've seen the need for. And the difference that you'll see here is that there's a nice pretty graph and visualizations and context to be able to see what's going on with your errors.  And of course, you get the normal breakdown of what's going wrong. So you can easily add extra bits to here and you can have the backtrace. A lot of people see errors as a to-do list. Me personally, I don't. I don't see errors as a to-do list. I see errors as a natural part of your app. Errors are going to happen. It's, you know, I'm not saying ignore them, but don't fret over them. They're not the end of the world. The best error exception notifier is your customer, your user. That's how I find it personally. An error goes wrong in Travis. We know something's bad when someone tweets saying, hey, sync isn't working. Respond, monitor Twitter and email, like support emails, Twitter. Oh, this is my little sister. So that was very unexpected even for me. I should actually start just littering them everywhere and I won't even know where. I'll just get someone else to do it next time. So exception monitoring is really important, but don't see it as the first thing that you need to do every morning. It is a good thing to get rid of them though. So do watch the big numbers. And my little sister's okay. So let's talk about metrics. Who here uses New Relic again? Just a show of hands. Awesome. Who here is using New Relic and not for a web application? One person, maybe two. Over there was two? Sold, okay. So this is what you'd normally see in New Relic. Pretty graphs, a nice little appdex score, throughput, awesome. It's really super powerful, but there's something that you might not notice here. This is not for a web application. New Relic is bendable around your agent apps, anything that's doing background processing, it might be background jobs, might be a whole separate app just during AMQP subscriptions. This here is all of our AMQP subscriptions and we can see which ones in particular are very slow or in this case, the most time consuming. And for us to do that, we just built a little rapid. It's a little bit of handy and better programming but we tell New Relic to attach its proxy and controller notification so we can then get a trace for that method.  You get all of the database breakups, all of the web external breakups as well. And then you also get to dive into each of those requests and see what's going on. It's really powerful, I will say though, it's not incredibly intuitive to add because they call it controller notification and people here will probably think of a web controller as well, a web controller, that's what I associate controller notification with. But for us, controller notification was used for an AMQP subscription which was a different way of hooking into it. Nonetheless, it's really powerful so you can get working with it straight away and because if you're using New Relic, sorry, a Herotu, you get the standard for free. You can just shove your app on Herotu and then you get all of this pretty stuff. So as I said, not just for New Relic, it's for all apps. I love popcorn. We're really getting to know each other here. This is great. So let's talk about metrics. So metrics are awesome, okay? Hands down, awesome. I would like to introduce you to Mattias, our Chief Metrics Officer and his pineapple. And when I was preparing this talk, I've given it a few times now and I was like, Mattias, how do I explain to the audience how awesome metrics are? And he said very abstractly, metrics are to a running app as unit tests are to your code. It's like, that's beautiful that even I have trouble understanding that. Tell me again, well, it represents a certain part of your app by measuring how long it takes and how often it occurs at a given point of time. It's as simple as that. I've even attributed it in 2012, just in case you ever reuse this. Effectively, metrics is attached to a method core. It's about how long it took. Well, it doesn't have to be attached to a method core but it's attached to something that's happening in your code, an event. And for us, we use it on the method cores a lot. So for example, there are four parts to metrics if I start there. You've got collecting, sending, aggregating, and visualizing. Those are the four main parts of metrics. And if we talk about collecting for a second, I want to introduce you to active support notifications. If you don't already know it, you're using it every single day in your Rails app. It's littered everywhere. It's even how logging works. There's no logger.print or logger.info. That happens via active support notifications. The greatest thing is it's a very simple publish and subscriber or observer kind of pattern.  You can subscribe to one event type or multiple using a redgex. It's really simple. You publish one-time events or you can yield a block and time it. Or you can just publish an event using instrument and send some extra info. I won't go fully into active support notifications, but I would highly recommend that you buy this book from Jose Valim. It is absolutely hands down amazing. We'll give you a very thorough insight into what's happening inside Rails and also that particular topic of notifications. So I'm just gonna refer to active support notifications as ASN for now, but let's look at how we could use it. That's all we need to do. We can just start littering our code with publishing and campfire and URLs and we can start passing information about this was called. We can go instrument, okay? So they're pretty much the same. Or we can instrument and it will do a timing as well. Nice and simple. We use it to publish three events in Travis. Now this is all over our code. We publish a receipt, completed and failed. You'll see a pattern similar to this. You'll send a message and we say, oh, someone fired off a campfire message or notification. We do some processing and then we publish that it completed successfully. Or if there's an error, we publish that it failed and we re-raise. It's nothing particularly special but because this is a whole lot of craft code, completed, sorry Josh, there we go. Because of the craft, we use a little bit of metaprogramming and now you just go instrument and send message. And all we're doing is emitting a whole lot of events. These events are completely metrics agnostic. So like step one is that you can even add this to your library or gem, but you can add this all over your code knowing that it's not gonna take up extra CPU, it's not gonna make your app slower, it's gonna just start emitting events and then you can start thinking about how do you want to use those events? It's very general purpose and it's all about the subscriber. This is the beauty, oh, I love German food. So let's think about sending and aggregating for a second. Stats D, that's no JS using UDP is one of the methods that we can use. We can start aggregating and sending or sending and have it aggregate. It uses simple UDP messages like this where you'd say this is one counter, increment by one please, or we can do a timing like 320 milliseconds. And you can even do this from your console. You can set up stats D and then just do this using Netcat, nice and simple. You've got a nice little Ruby gem for it. You can just set up a host in the port, increment, do some timings, boom, done. And it's only 155 lines of code which is a very small gem. It's got some tests in there, but this is what we use. We use a combination of Metrix D which is by Eric from PaferTrail. And it's based on Ruby and Event Machine. It's coupled together with the Metrix gem. Now the Metrix gem is very similar. Again, you're not doing anything mind-blowing here. We set up counters and we get to increment and we can do some timings. And then we flush that every X amount of seconds. So we flush it every 60 seconds. And then what Metrix, the gem does is it part aggregates, it aggregates within that process and then it sends it. And now what it does is it sends it to a reporter. So this is quite cool because now you can send it to different types of reporters. And in this case, we can send it to our logs and we can actually see what's happening in our logs for each of these Metrix. So let's subscribe to something here. What we're gonna do is we're gonna add a started app so we know how long it took. We'll continue to use this instrument. We're gonna also do the started and finished app on the completed. And then we subscribe. We just subscribe to anything using that namespace Travis at the top. And then if there is a finished app, then we send the time it took. And if not, we just mark it that this method was called. So it's nice and simple. This is all you need to subscribe your entire, you know, code base. You litter it with events.  And now what we do is we start the login. And this is what we get out. We get everything from, now you'll see that at the top, it's active record. And we've got a timer, we've got how many times it happened, the one minute rate, 15 minute rate, means, max, standard deviations, everything. This is just for one process. So now what you can do is if we think about it, this is what we've just set up. So we use PaferTrail. This is Travis in particular. Travis uses active support notifications. We use metrics. And then it goes to PaferTrail. So all of our metrics are going to PaferTrail. It's a very quick way to get started. And it means it's very visual as well. So it gives you a little bit of debug time. But it's the thing that comes after it that's really important. Because what we're doing now is we're emitting events. We're aggregating, we're sending awesome. Now we need to start visualizing. We need to start making use of these events. So there's Graphite. And Graphite is a self-hosted product.  And it gives you very thorough graphs. It's not the prettiest interface, but sometimes it isn't about the interface, even though it actually is. That's why I have a Mac. And then there's Liberato. Liberato we are in love with. We get pretty graphs like this that allows us to do absolutely everything creating our own dashboards. This here is actually a set of our errors. We have everything from HTTP errors on the top left to AMQP messages. We've got log entries and also notifications. And if we look at here, this gives us a breakdown of every type of notification we're sending. The top green bar, the big peak, so actually push notifications. So you can see that we're sending a crack load. And the cool thing is that you can embed these graphs into your campfire, including now live graphs. So you can actually share a graph with your team and say let's monitor it live together and start commenting. This is actually one of the graphs that we blogged about a while ago. What you'll see here is that we deployed some code and then boom, we had a lot of errors. And then we fixed the bug quickly and everything peaked off. We still had some high peaks. And then we fixed another error and they just all went away. So as you can see, you could use something like Erbit or Get Century or Haystack, but you can also use Libratto and your own metrics and monitoring as well. Even better is Teseo. It gives you a nice handy dashboard that links into your Libratto or Graphite metrics. Now actually the original fork of it or the original version of Teseo which is on a obscurity at the bottom there, the GitHub, that only works with Graphite at the moment. But if you use Matias's or Royed Rage's fork, this can also work with Libratto. So you get a nice little dashboard and it's also an easy dashboard that you can note your graphs. You'll see at the red one, AMQP ready. You can see that we've got 17,000, 18,000 messages. This is another error. And we can see nice and quick and simply that is a big warning sign. We need to do something about it. So if we put everything together in our holy circle of metrics, it goes into PaverTrail. We need a webhook to pull it out of PaverTrail and then it goes to Libratto. But we can do a lot better here. So we've got a lot of aggregating and sending. It's a little bit silly. So we can use the Metrix D reporter along with Metrix D and now we get something like this. Okay, we already cut one piece out of it. Or we can go even one more step and we can use the Libratto Metrix reporter. And then this is all you need. Libratto will do the aggregating, the sending and the visualizing for you. I used to scuba dive a previous. Fun fact. So the next part is like you're using all these tools. You're doing logging properly. You're doing exception tracking. You've got Metrix. But the third part to all of this is the internal tools. So I want to show you Travis Admin. It was a little app that we've built. It's open source as well. You know, all of Travis is open source and this is a app built on Sinatra along with Bootstrap by the beautiful Constantine Hazard. I call him Constantine House. I would urge you all to do that because I don't think he likes it that much. I love you, Constantine, in case you're watching. So this is part of the app. It's composed of three, four different bits. This one here allows us to view the service hooks set up for each of the repos. What we do is you would put in a user or the user and a repo on the right-hand side and it checks the database for their OAuth token and then we can check the hooks set up for that user. We can confirm that the hook is set up correctly. We can also disable pull requests if we need to or maybe we can enable some extra things to look out for. So you'll see pull requests just down here. We've also got the GH web console. GH is a gem created by Constantine which is a nice way of interacting with the GitHub API. It's different from OctoCat. This one uses the headers a lot to traverse and it's a bit more low level. It's very low level. But what we can do is we can put the user and the command that we're going for, like user repos, and we get a nice formatted JSON list along with the headers on the right-hand side so we can see what our rate limit is and what was the scope of that request. So it's a nice way of seeing what's going on with our GitHub request because we do a lot of them. But the crown of all of this is the event monitor. So as I showed before, this was our infrastructure. It's not even part of it. It's just half of our infrastructure. And the Daryl Nicholas Cage joined our team and we tasked him to create a way to unify all of our notifications. So if we have all of this information going on, how can we kind of see the path that a request takes? Comes in via the listener. It goes to Hub. It goes to the worker and then it comes back to Hub. How do we visualize that? So he built an app for us, built on EmberJS and using Redis PubSub and we've got something like this. Now you might not see it very easily but I'll turn on the live version in a second. But what we have on the left-hand side is an event list. All of the events that happened and the master event. So in this case, you'll see that a job finished and then we see all of the events that happened after that. That might be a push notification was sent an IRC notification was sent, database was updated. So we can see all of the sub-events and then we can see the payload attached to each of these. So now you can unify and actually see live what's going on. This just goes to Redis, it expires quickly and we can watch it live and then we can pause it if need be. It uses Travis Instrumentation, a custom thing that we built for doing all of these hooks along with active support notifications. But what I'll do, so this should be Travis happening right now. And wow, it's very quiet. I don't know, there we go. And you can see that it's updating accounts. We can kind of see the payloads that are going on. You can see the Mongo Ruby driver just started and all the stuff that started with that. So it allows us to see not what's going on in a logging kind of way, but what's going on in a rich JSON logging kind of way. We can kind of see all of the events that are happening after each other. Now right now, if you wanted to go use this, it's completely open source. If you want to use this, it's completely open source, but I will admit that there's a fair bit of code that is kind of bound to how we saw the problem. I'd love to have other people try this out and get involved and see how we can take this to the next step. It's, a lot of the code is located in Travis Support, which is one of our repos on GitHub. It's completely open source. I'd like to just finish off very quickly to say that the whole point of this talk is to not just deploy your app to Heroku or using Capistrano to a dedicated server, you need to know what's going on inside your app. Knowing what's going on inside your app will allow you to find those errors to make things faster and to find out what features your users are using. In fact, one of the best ways you can use metrics is start attaching it to features. Start seeing the paths that people use and what they're doing and how often something is used. You just spent two weeks working on a feature that no one uses, is that a good return on investment?   Actually look into what's going on. The knowledge will help you improve. Don't look at those big more numbers. Use services. All of these services are available and they're amazing. Build internal tools. Don't build them straight away, but over time figure out what is it that you need or your team needs to make it easier for you to administer the product. You don't need to build it in your current normal application. Build it as a side app. That's why we have Travis Admin. That stuff does not belong in Travis itself. It's just another tool that would read the database and all the external services. Services are your friends. I think I've said services are your friends enough. You can find us on IRC and on GitHub and on the Twitters. If you'd like a pretty sticker, come talk to me. You are awesome. Thank you. And any questions? There's two. I don't think we have microphones or any, just yell it out. We can get Travis through the council bill, if you want to say. Email support at travisci.com. We've got a blog post with our pricing and all of the plans. We are freely giving out the invites. We're just controlling that slowly while we get those pages up and running. Yeah, email us and we'll get you on that. That's why we built the little, I guess, the class declaration of like saying instrument this method. Because it's our way of pulling it out of the code. We don't have to have it sitting with the method underneath. You can actually eval that into the classes in a different area later on and then you can group them together. But we find that it's the least intrusive way. We're not actually worried about instrumenting too much. In fact, the more you instrument the better. It's about how you kind of use that statistics. Like the main things that you should probably look at like start easy is how long do maybe some of my notifiers take? If I'm posting to Twitter or Facebook, how long do they take? How many of them error out? Maybe you can look at features as well. It's a great way to find out what your users are doing within your system. If you have an amazing time tracking app and no one ever uses the schedule function, then maybe you need to think about how you rework that. It can help you with your A-B testing. Metrics is very abstract. It's not just about code, but it's about everything that can kind of go on within your app. But we generally feel that every time we add a feature now, we make sure that we add a metric so we can track what's going on with it as well. You can't, because you only have HTTP for Heroku, we can't run metrics D on there or stats D. We use, at the moment, the webhook with pay-to-trail, but we'll soon be changing over to using the Libratto notify or reporter. So now it will just be the app will use the metrics gem and then it will go straight to Libratto. Libratto is building and better alerting. It's kind of the one thing I didn't cover in this talk because it's a little bit hard to throw in everything, including the kitchen sink, is the holy thing that mixes everything, connects it all up, is alerting. So the final stage in all this is that when you've got all these metrics, you've got all the measurements and the monitoring, you need to be alerted when something hits peak. Now Tiseo takes it to a certain degree, you're getting nice colorings out of it, but what you really want is campfire or a pager duty or someone to come out and get you from the restaurant or you need a way to find out when something's going wrong. It's the first step. This step takes you to knowing what's going on outside the app. Then you need to kind of add the alerting to notify you when something's going a bit astray. No other questions? Oh, well, thank you very much. If you've got anything else, just come talk to me after. Thank you. Thank you.


QuickBooks Online. Save customization or memorize reports. Get ready to start moving on up with QuickBooks Online. We're going to be using the free QuickBooks Online test drive searching in our online search engine for QuickBooks Online test drive selecting the option that has Intuit.com and the URL Intuit being the owner of QuickBooks. We're going to be using the United States version and verify that we're not a robot. I'm not a robot, but I crank stuff out like a machine. Anyways, we're going to zoom in holding down control up on the scroll wheel currently at 125% on the zoom in and the cog drop down. We're in the accountant view as opposed to the business view. We'll try to toggle back and forth between the two views in order to get a feel for where things are located in each of them. Right click it on the tab up top to duplicate it so we can put reports in them as we do every time. Right click in the duplicated tab to duplicate it again as the tab to the right is thinking. We're going to tab to the middle reports on the left and open the balance sheet. And then as that's thinking tab to the right reports on the left open up the P to the L the profit to the loss the income statement. And then I'm going to close the hamburger otherwise known as the hand buggy and change the range from 010122 tab 123122 tab. Run it to refresh it tab to the left middle tab closing the hand buggy and changing that ranging from 010122 tab 123122 tab. Run it to refresh it. That's the setup process that we do every time we've been looking at the major balance sheet report. Remembering that the balance sheet and income statement are the primary financial statement reports all other reports for the most part, giving more information on one or multiple line items of these two major financial statement reports. Now we want to think about memorizing these reports and typically you would think about this in the context of having your reports sorted for internal use possibly in the way that would be most efficient for you as well as reports that will be used for external use, possibly as a bookkeeper to give those reports to the client or if you're working internally in a accounting office, the grouping of the reports that you might provide periodically say to a supervisor, I'll primarily be thinking in the context of a bookkeeper that's providing this information say to a client on a periodic basis. So in that case then as you adjust your reports, you might then want to customize those reports and group them into a category, save them and in the desktop version, by the way, it's called memorized. So in the online version, we have save customization. I don't know why they are differentiating those terminologies. I think memorized rules off the tongue a little bit better than save customization, but you get the idea so we can customize it and we can save the customization so that when I generate the report again in a future period, hopefully I can just change the date range all else formatted for us that will save us time. Now on the internal reports, oftentimes the default might be suitable for us, but maybe we want to change like the negative numbers to be bracketed or something like that, even on the internal reports. So we can then think about those things that we think are most efficient for that purpose. And then the external reports where I might do things, for example, taking the panties off. I like to do for external reporting oftentimes and then I might do some comparative analysis and whatnot with the external reports. So let's think about how we might group them then I'm going to do a little customization. So let's customize this and let's say that we're going to say here's the range. Let's say I'll keep the panties on for the internal report, but I still like to see the negative numbers as bracketed and read possibly. And then maybe on the headers and footers I'll say this is the balance sheet and I'll say this is balance sheet internal for internal use. I'm going to say internal balance sheet and maybe I can just remove the date time or maybe I like the date and time but not the report basis for the internal report because I think that's not really necessary and then let's run that. So not much change on the internal report except those negative numbers kind of pop out to us. So if I want to save that then I can then save that up top and say I'm going to save the customization and this time instead of just saving it into the customized area, I'm going to make its own group. So I'm going to say that this is going to be a group for an internal reports. Now if you have multiple people you might put it under your name as well. So if you have multiple users using the QuickBooks file logging in you might save the internal reports under your name and then have them different possibly than the external reports. So I'm going to add the group and so there is the group. So I think that's right and then we're going to say share with now do I want to share when you think about sharing the reports if you have multiple users in the QuickBooks system you might be sharing with other people and you can also think about sharing like the report structure if you have a particular report structure with into it if you would think that that might be something for other users could be useful. I'm not going to be sharing with anybody here. I'm just going to save that and then I'm going to go back to the first tab. I like to refresh the first tab now with this little refresh button so that I can then note that there's hopefully it has been updated in my report center on the left hand side. I'm going to go now to the customized reports and now I've got this item under the customized reports. Now sometimes it's a little glitchy with the reports. I went back over to the reports here and made sure to add this report so we added the group and now I can see it in the added group here. So I'm going to save it again and try to run it back over here back to the tab to the left and refresh it again and I should get a group this time that it's going to be in. So there's our group. So now I've got my internal reports in this grouping. Now if I want to make so that could be a useful tool. Now for external reports you might have different formatting for external reports that you're going to give to say clients. There's a couple of ways that you can deal with those external reports. We'll talk more about how to group them and put them together but you can save them this way and then try to attach them to an email or zip them together as a PDF to provide, give them to someone on a cloud drive or possibly you can use Excel which will look at that method to export them to Excel and that's a way that you can get them all on one PDF file with the help of a PDF printer. QuickBooks also has this managed reports area over here which can give you some kind of nice formatting for the reports. It's a little bit restricted in some ways but that's another method that you can try to make your reports look a little bit more professional when providing them to clients. We'll talk more about that in a future presentation. For now let's just think about how you might group all your reports because remember on the balance sheet reports we talked about the fact that we could have a whole bunch of different balance sheet reports that we might want to group together and provide to people at the end of the month. So how can we make that system as easy as possible? Maybe we make a group that would be month-in reports, maybe we have a different group for year-end reports or quarter-end reports and then maybe we even number them so that every month I can just basically go into those reports, change the date range and then provide them to the client in a way that is efficient and nice and clean and easy. So let's go back on over here and say okay maybe the first report we give is that summary balance sheet. So I'm going to go into the plus button here, go to the reports and I'm going to say standard report. Let's go into that summary balance sheet which is right here balance sheet summary and I'm going to close the hamburger and do our customization. So on this report I'm going to say that we want to remove the pennies, the negative numbers and red and then on the footer I'm going to get rid of the date time report basis and it's called a balance sheet summary. Maybe I prefer it to be called summary balance sheet possibly. Let's try that just to change it and so if I save that there's the report. I can change the end date to 123122 and run it. So now if I want to save this as an external report that maybe I'm going to generate every month then when I save to a customization I'm going to add a group. Actually let's do it this way add a group and the group is going to be month end reports.  I'm going to add the group and then I'm going to select the group month end reports. So it's a little bit you got to make sure to add it so that it's in there and then select the group. And then I'm also going to number it just so it won't be an alphabetical order in that group because I would like this one to be printed first or the first thing that we represent. So one way I can do that is to number it.  So now we've got these month end reports. Let's just make a one or two more just to see how that would work. If I go then back on over here and let's go to our normal balance sheet. Now I'm going to go to the reports. I'm going to go to the standard reports normal balance sheet. And if we made like a comparative balance sheet let's just do a quick comparative balance sheet or let's just do the quarters and take this to 1231 222 and run it. So now we've got our quarter by quarter. I can customize it and say I want without the sense brackets red and then I'm going to call it now a comparative balance sheet or balance sheet balance sheet by quarter something like that. And then get rid of the date time report basis save it. And then if I memorize this I can I can say save the memorization. I'm going to put this into my month end reports. I'm going to name it the second report imagining that this is the second report that we want to batch together. And then I can save that information. And then at the end of each month I can go into I'm going to go to the first tab refresh it and go down to my reports. And then custom reports. Now I've got my month end reports in this nice little dropdown and at the end of the month possibly then I can just I could just go in here and print these out in order. And then that will help me to group them possibly to batch them possibly with an email possibly with a zipped file possibly with a cloud drive. Possibly we can then put them on one PDF file with the help of Excel and a PDF printer which will test out in a future presentation. So that's the general idea that I just want to get across here. You might have those internal reports that have formatting that's a little bit different than the defaults. And then you might have the external reports which are often going to have a little bit different formatting. And then the question is how are you going to be grouping those external reports. You might have month end reports. You might have slightly different reports for the quarter end because now you've got a full quarter that you can run quarter reports. And you might have different reports say for a year end because now you've got a whole year's worth of data which could have a more complex batch of reports dealing with the year information. So that's the general idea of that. We'll talk more about how to possibly group these reports and present them in a future presentation. But for now let's open up the cog and let's switch to the business view. And let's just look at where we've been located under the business view. So the reports of course are in the business overview and the reports. And then we know that the reports that's basically the only place that we have been going to right. So that's right here. So that's where the reports are under the business view.


Is there something about your game that you think you could really make a difference in the NBA stand out with a Just a personal trade of yours. Yeah, I mean, I'm adaptable Like we said, I can be like water I can figure it out in any situation They call that a five-tool player in baseball, so you're saying you could do pretty much anything that's asked What would you consider to be the most important part of draft night? Is it your outfit? Is it the handshake with the commissioner? It's got to be solid is there at the interview after is there anything you're looking forward to the most that I got a nail


Congratulations on the very last time. Thank you. I'm excited to be here. As a matter of fact, more than me, my son is very excited that I'm coming to this club, the Real Madrid Club, the biggest club in the world, you know.  Whenever I enter this room, there's a huge big TV and there's always matches going on. I'm very, very excited. I'm looking forward to meeting all the players and, you know, visiting the fantastic stadium.


coming. My name is Avi Kiviti, some of you may be familiar with my previous project KVM which I was the maintainer for the last, for seven years, but now I'm doing a new project OSV which Gabor and I will present. And for those of you who doesn't know me or for those that do, I'm Glauber working together with Avi on OSV and previously working on the past few years with containers. So I'll let Avi start. Hey, thanks. So this is the team that is doing OSV at Claudius.  That can change. And it's completely frozen. There it is. Okay, that's not expected. The first one was fine. Yeah. Okay, so this is going to be a little awkward, but probably resolution difference. I don't know. Yeah, we'll manage. Okay, so let's look at the slide to come up. Is there any other PDF here that I might have in Fedora? I don't know. Okay. So let's look at the story so far in operating system evolution. First people coded directly to the hardware. There was no operating systems back in the good old days. But then they found out there was a lot of common code to application and common tasks they needed to perform. So people wrote operating systems for those things.  So they invented hypervisors way back in the 70s. And again, applications still had a lot of common code, and so they abstracted that into runtimes. A good example of a run-manage runtime is the Java virtual machine, the JVM, and also things like application servers. So there is a pattern here where people keep adding more and more layers in order to abstract common functionality. And this is how it looks like. The hardware on top of that almost always to date these days is very prevalent. The hypervisor in clouds. Then you have the operating system presenting an API. And more often than not, there is some runtime, the JVM running on top of that, and then the application server and the app. And then we have to switch to a slide and it takes five minutes. This one is fast. No, it doesn't move. It's also cutting here. I'm sorry about that. Maybe if you take it out of presentation mode, it's going to get better. Maybe. Okay, so let's focus on three of the layers in the stack, the hypervisors, the OS and the JVM. And what we see is that they all have some duplicate functionality. So each of them is trying to protect the layer below it from the layer above it. For example, the hypervisor protects the hardware from an abstract. They have the hardware from the operating system and it presents a unified set of hardware in KVM, that's Virio Block and Virionette. The operating system does the same thing. It abstracts the hardware via device drivers and presents it as files or processes. And the JVM, again, does the same thing. It takes the various operating system APIs and abstracts them again into the JVM APIs, which are then used by the application. And the same thing happens in reverse, where you have protection. Each layer does the protection. And this is all duplicated. And we all know what duplication means. It means enlarged footprint and inefficiency and more things to manage. So that would all be okay, as Avi said, if it came for free. And what we can realize, when we look at the job that the operating system is doing, so here you can meet more or less what kind of tasks each of those layers are doing, turns out that the, historically speaking, the operating system is not really good at isolation. It can isolate processes well enough in terms of security, in terms of accesses to resources, but when it boils down to actual multiplexing the resources in terms of resource usage, it doesn't go that well. So there has been attempts in the Linux kernel to change that through C groups and the whole container thing, which was what I was working on. But the historical fact that virtualization, that the operating system is traditionally not very good with isolating things, is that what led to the wide adoption of hypervisors. So that's one of the reasons. If you look at how things are, it makes sense for like back then, in the day when it had like VMware start getting virtualization to get serious, that is in the x86 market. I mean, virtualization itself was invented by remsas II, already working for IBM to build the pyramids. But if you look at the x86 world, it comes with VMware pretty much. And what you get is that you get a virtual server. So if you're going to isolate a workload, then each virtual machine isolates whatever is inside the virtual machine pretty well. When we have the cloud things change even further, because now there is another pressure which is pretty much the flexibility that you want from cloud appointments. So we get Amazon, for example, which is an extremely large cloud provider.  They're talking about the compute nodes. So you just have something that compute, and it's getting more and more commoditized. You just get on some of those compute nodes, and when you need them, you bring them, when you don't need them, you send them back. And things have been truly, have been turning truly, truly massive. Your workload is usually not entirely tight together. So for maximum flexibility, so this is the kind of things you have in cloud. I mean, a bunch of buzzwords, but it's basically many kinds of way, many types of way in which you can deploy things in the cloud. But traditionally speaking, there is an extra pressure to have things completely isolated. So you don't really want many applications running on the same virtual machine. Because aside from the fact that the operating system inside won't isolate them that well, you have, you would like your workload to scale independently. So if one part of that scales, the other one does not necessarily so. So you just bring up a machine. And really, if you look at how those things are being deployed, you really have commodity tiles being deployed. So you don't really think anymore. People don't really update their softwares in their virtual machines. So the OS is basically immutable. If you want to run a new version of the operating system, you basically spend out a new VM. And it's very important from my point of view to point out that this is not really a proposal that we're making. We're not coming here to say this is how people should run cloud and VMs. This is pretty much, we look into the cloud and we look at the Amazon workloads, for example, how they're doing their deployments. And that is how it is working. So not only there is duplication and functionality, but that duplication is pretty much useless. You're not using all that isolation. When you go look at how does the typical virtual server looks like in the cloud. So you have the traditional cloud infrastructure and each of those virtual machines, they run a virtual server, Linux most of the time. Linux supports, it's a multi user operating system, but most of the time you're not using it because of things that are isolated among VMs. You can run many applications, but turns out that in practice, you're usually running just one of them. And that's pretty much it. I mean, so you get a lot of complexity for no reason. So the whole idea that we get for OSV is that we should really try to diminish this complexity by making things behave in a way there is more natural giving boundaries that we have today. So we truly believe that less is more. And that's why we're not giving shirts this year, but we're having boxer shorts for you. Right. So because less is more. So during the talk or even after the talk, come talk to us and collect your boxer shorts. And don't forget less is more. Okay, so so what are we trying to do? We're trying to build the very best operating system for running cloud workloads. And that means the easiest to manage providing the best performance being the best overall. And I like to just say that we are often asked, is it a new distribution of Linux? Is it a modification? No, this is something that is written from scratch, something that we've written that cloud use and are now open sourcing. And it's completely new. So let's look at how OSV is built. So this is basically a core sort of kernel that runs in the same address space as the application. It runs just one application. So there is just one process instead of a wide range of process which have to be managed and switched around. There is no concept of user space and kernel space. There is just one space. So there are no privilege level transitions which are which cost time and also require validation of parameters. So that's on what we change. What we don't change are the APIs that that's application sees. So there is no need to port your application to OSV, it will just work, but it will be faster and it will be easier to manage. So these are some of the changes that we are making. In terms of memory, we are using huge pages to reduce memory access latency. We have sharing between the JVM heap and the system memory so that you don't need to configure each of them separately, but it dynamically changes according to requirements. We have improvements to IO and since we're in a single address space, that's very easy to do. We are making scheduling improvements. The operating system knows that it is running on virtual hardware and it is optimized to run on top of a hypervisor. So we use this knowledge to improve the scheduling algorithms. Since it's smaller and simpler, then it's already tuned for the cloud workloads. There are no thousands of configuration files and knobs that you have to twiddle to get the exact performance. It comes pre-tuned. And we're also doing improved communication between threads. So if you look at that, we're very confident that by writing it from scratch, we can get things done better because we have designed decisions that a traditional operating system that runs many kinds of things cannot take and we do because we're basically writing everything over. The very fact that we have pretty much no drivers help us out with this task because there is no 10,000 network cards to support. There is only Zen, KVM, VM, or pretty much. And there is very unlikely that we're going to have 10 hypervisors in the very future. I mean, we can have an extra one who knows but not too many. So right now, up to now, I've told you things that we've been doing that everybody does and we believe we do better due to the decisions we made. But there are also nice opportunities to be done once you get rid of the user space, kernel space separation. I'm going to detail some of them now. So when you get a network package, for example, what you get, as you can see here, is packet processing is quite serial in the way it behaves. So you get an interrupt coming and it tells you that a package is available. So while an interrupt handler, you allocate SKBuff, for example, in the Linux world and then you keep processing that until it finally gets in the application. And one of the reasons it is like this is that somewhere there, like you have a very clear kernel user space boundary that we do not have. If you look at the, and this is the traditional socket model, right? If you look at Van Jacobson's proposal, you cannot see all of it but most of it, it's well enough to understand. Van Jacobson has been speaking about that since 2006 and that's not something that it's impossible to do with Linux. It's something that is more natural to do with our model, which he calls network channels. So instead of sockets, we're moving towards network channels. So what network channels does is that in interrupt handling, you actually don't do anything. You don't even allocate memory. You just get the package and relate to the correct location. So whatever the network car put the package is fine. And then you process that very, very close to the application. So you're already processing this. So first of all, outside interrupt context. And also in the same CPU, in the same cache line, very, very close to where the data is going to be consumed. The results are actually quite good. Those are not our results. Those are experimental results from Van Jacobson on Linux. So it's actually very easy to do that in Linux experimentally. But in practice, it is a little bit harder. You need somehow to get your network stock running in the same context as your application to get as close as possible. So one way of doing this is to moving more of the stack into the space. But the other way of doing this is what we're doing, move more, I mean, move the application into kernel space. But what matters is, once you're out of the boundary, we can try, we can do those kind of things. Also as Avi said, but I'm going to detail Java machine virtual, the Java virtual machine memory ballooning. So if you're familiar with the ballooning drivers in KVM, for example, that's pretty much the same thing. People who run the JVM usually have to statically set the size of the heap and say, yeah, you need to know in advance how much memory you believe you're going to need. And what we do is that we just allocate all the memory to the Java virtual machine, and we have a balloon driver inside. This is still actually work in progress, but it's currently being done. So it's not for the future or mid future. When you need memory, that balloon drivers get memory back to you. And when the Java virtual machines need that memory back, instead of garbage collection, for example, we can try to ask the operating system first, can you give me back some of that memory? So it's the traditional ballooning driver, same thing between guest and host. And this is also made not only possible, but a lot easier by the fact that you have no trust issues here. So the operating system and the Java virtual machine are actually all operating tandem. There is nothing to protect against. Okay, let's look at the timeline of what we've accomplished since we started. We started around December and quite quickly we have a Java Hello Award running. We have SMP support quite soon after that. And then we enable the various features, support for networking and storage. We're using a ZFS, our file system, and we gradually optimize the networking stack to get really good performance. And we're not running real-world applications like Cassandra and MemcashD and we're getting very good results out of both of them. And we're very happy with where we are and we're not continuing this, optimizing more, enabling more functionality, and testing more applications. Part of the reason we've been able to proceed so quickly, first of all it's a smaller application domain. We're not trying to do every possible workload, like Linux does. We're focusing on a smaller set of workloads. And the other reason is we're using modern techniques. For example, we're using the C++ languages, our implementation language, and not C, which reduces the amount of coding we have to do. We're able to reuse more libraries, more data structures, and this way it reduces our development and debugging overhead. So the status right now, we're running Java, but not only Java code, pretty much any JVM-based language like Ruby, which runs on the JRuby project, and the newer cooler languages like Scala, Ruby, Clojure, they all run on OSV. In addition, we also run selected C applications. We're not supporting the entire POSIX API, so we can't just run any C application. But we do support a large enough subset that important applications can be ported with relative ease to OSV, and we run Memcache D, and others are also possible. In certain benchmarks, we already get very good results. So right now it's not tremendously better than Linux, but they're sufficiently better to show the promise, and we plan to improve it even further. In micro benchmarks, it's very easy to show the potential because you have things like a context switch, which are relatively expensive on Linux. They run very, very fast on OSV. Boot time, which is particularly important in the cloud because you scale by spinning up newer instances on demand. So boot time is very short, which allows it to respond very quickly to loads that become greater and lower in time. It runs the same thing. Of course, you can just pull the plug. It's a lot easier. We're using the ZFS file system, which is a great choice for the cloud because it's self-healing, so it reduces the administration overhead. And like we mentioned before, we're using huge pages. So why is OSV open source? So first of all, we all come from an open source background and we love open source, but it's more than that today. To be credible in the infrastructure space, you have to be open source. Otherwise, no one is looking at you. And we're looking for a cooperation. We're looking for developers who are interested in the kernel space and doing an operating system from scratch. We're looking for people who are interested in a management stack versus the operating system and people who know the DevOps workflow, how you can deploy your application dynamically, push a new version every day, or sometimes even multiple times per day. And we would love to cooperate with the open source community on that. We already have multiple contributors to OSV, including from two companies and also individuals. We have a very friendly BSD style license and no contributor license agreement, so just sign off and that's it. And we really would love to cooperate with these people. In terms of architecture ports, so we started with x86, of course. We're only 64 bits and no 32-bit port. Out of familiarity, we started with KVM and it's running very fast on that. There is also a Zen port that runs on Amazon and you can use our image on Amazon and it just works. It's still not as fast as it could be. We're still optimizing it, but it does run. And we plan to do a VMware port eventually for the enterprise customers who are mostly on VMware. We also want to do an ARM port since ARM is such a hot topic in virtualization and in data centers. And if you have another architecture that you want to see supported, then patches are very welcome. All right, so this is what we do. Not everything that we told you is completely ready, but it's everything at least working progress. I would like to tell you a little bit more about things that we plan to do that shows the potential that we have. So those are all still ideas, but those are ideas that make sense and they're actually implemented in other scenarios in other circumstances that we just believe we're actually capable of implementing them in a nice and supportable fashion. Most of them revolve around changing the JVM somehow. So this is one of the main differences that I see, particularly between what we're doing and the containers approach, is that we now have your application having access to all the facilities of the hardware. Because, I mean, KVM or Zen or whatever the hypervisor is protecting you, your application, the JVM in that case, and it is the JVM only to provide a single entry point. So you don't have to change your application and every change is behind it. But the JVM now can see the hardware. You can see everything and there are many interesting things that can spend out of it. The one I'm going to detail is this. I mean, Garbage Collection is by far one of the most annoying things of the Java world. I mean, it's nice when you code, but it's not really nice when it just stops your machine to Garbage Collect and you can't really do anything. One of the techniques that the Java Virtual Machine already employs is to keep a dirty bitmap or any kind of dirty tracking, really, to know if a particular region of memory has changed since the last Garbage Collection run. And if it hasn't changed, there is no reason to go through it again. You can just reuse information from the previous run. Now, the JVM does that, but it spends a lot of time by tracking those memory because it needs to be tracked in the just-in-time compiler level. I mean, every memory access needs to be audited for that. But it doesn't have to be this way if we give JVM access to the page tables. So, if you can use the hardware dirty bits, for example, whenever you have a change, you're just going to see that represented in the dirty bits by the hardware. So, you don't really need to do anything in software. We can proceed a lot, a lot faster in that. So, in this case, it's not really only the Garbage Collection that goes faster, but the whole execution can actually proceed faster. Okay. Let's do a little technical deep dive into some of the features of OSV. So, the first feature is not really a feature, but it's an attribute of how we develop it. And it's using C++ for a kernel, which is pretty rare. But let's look at two pieces of code. On the left side, we have a typical kernel code that locks a particular data structure with two locks that are nested and does some operation and has a few checks. And as most kernel developers will be familiar with, there is a lot of codes that checks for errors and does the appropriate rollbacks. And it becomes very complicated and very long. So, this is a very typical code. And on the right side, you can see how it transforms when you use scoped locking. You no longer need to do all those rollbacks manually, all the error handling. It just works. You want to return something, then you return the value. And the language already takes care of unlocking everything in the correct order in its correct place. Something else we do is we have a strong focus on performance and tracing. So, the code here shows how you can declare with just one line a trace point and then use it in just one other line. And on the screenshot below, we have an OSV shell command that allows you to enable trace points dynamically. So, usually a trace point is compiled to a single knob instruction. But when you enable it, it compiles the code that dumps all the parameters to a log file or is able to a statistics counter. And we can count statistics or generate call stacks. And then we can use that to analyze application performance and understand it and all of that without changing the code since those trace points are placed statically into the system. Another feature, this is a more advanced feature which will be a little in the future, are data plan applications. So, a certain class of application has performance requirements that are very high and they don't want to go through the entire stack. So, for this kind of application, we enable direct access to the virtual ring buffer. And there's an application that can sit on top of the ring and just consume packets as they come in, do the transformations you need to do and send output data very, very quickly. And this way you can obtain very high performance and very low latency. Of course, these are applications that need to be written specifically for that API. But there are some classes of applications that will benefit greatly from it. Now, OSV can save a lot of resources as we've told you. We believe we're faster, we're more performant in many ways, and we're reducing the application. But CPU and memory are not the only expensive resources you have in your organization. I mean, another very expensive resource are people. It's not that we want to fire people, we want to give people vacations. So, what you typically have is it's not that we're proposing to get rid of people. I mean, it's in the same thing. It's all about the application. So, when you look at the whole stack, you have someone managing the hypervisor to offer you some cloud offering of any kind.  And that's a sysadmin. And it's not that sysadmin is in general should not exist, is that sysadmin is misplaced. I mean, you shouldn't be there. It's just cost. If you look at what people are doing in the cloud today, you always have an operating system as your guest. So, you need to configure your operating system. You need to set up the users. You need to set up the configuration file. You need to tune it to the particular use case that you have. All of that needs to be done. It's either done manually by a human being or it's automated away. But even this automated away process is error prone and it's particularly tricky because if you make an error in the automation, of course, you're just going to replicate that. So, automating away the problem, it's a way of looking into this. So, the OSV way is slightly different. Instead of automating away, you just get rid of the problem completely. So, there is no tuning to be done. There is no tuning to be done. Like as Avi said, we're only doing one thing. So, you run virtual machines and only one application. There is no choose between workloads. That's your workload and end of story. There is no state to be saved. I mean, there is no configuration files. There is no way to do that. Basically, you need a new VM.  There is no patching. I mean, there is especially patching. I mean, of course, we can release new versions, but we're not doing something which you're going to try to update your operating system. In particular, if you're using any kind of Linux distribution for that, you have unrelated packages that are just there because the system needs it to run. And you need to update them eventually. You need to follow the security advisors and all that. We're all doing a single binary thing. And again, we just expect that to be a lot easier from the management point of view. The actual management of the runtime of the virtual machine during the lifetime. We do have management software. So, we have a REST API and you can connect either Shell or SSH or web browser is that the example we're showing, which is basically aimed at deploying things at OSV. So, it's in our interest to make this as easy as possible. You basically, if you have a Java application, for example, you drop the jar in there and you're good to go. It's basically a way to install things. You get things installed in OSV and once you prepare the image, there is no maintenance to be done. That's the goal. So, and again, I mean, we're aiming this to be completely stateless. And this is something that fits the cloud model very, very well. You don't ever have to keep state of things. If you need a new VM, you just come up with a new VM and that's already in the state that you need. We're planning on integrating all of that with Chef, Puppet and all the tooling, as Avi said, of the DevOps workflow. So, that's not anymore in the kernel, but it's a very important thing, a very important aspect of what we're doing. And it stuck again. So, while the next slide loads, you have an application. So, what do you need to run on OSV? If you're running Java application, you need to breathe and you're done. There's nothing else. So, that's one of the reasons we're being so focused on the JVM. So, it's not that you're, we're not JVM exclusive, but focusing on the JVM allows us to allow you to run your application without any single change. You can just drop your jar and you're good to go. Because any eventual modifications that we may have to make to the JVM are made to the JVM itself. Your application is just fine. Even if you use JNI, and that's why. Because we also try to support the Linux POSIX API. We cannot support all of it. So, some of it is obvious. For example, if you run only a single address space, it's impossible to fork. I mean, threading is fine. We do have pthread create and pthread destroy and all kinds of pthread callings. But fork, xx, something that you cannot do. And we need position independent code. So, at the very least, you need to recompile your application. And to instead of generate an elf binary, you generate a shared library. So, there is some effort. That's not for free. That's why we focus on Java. Because if you have a runtime to isolate it from that, then it's a lot easier. But we don't expect that to be a lot harder either. So, please. So, the question is, I need to repeat that, sorry. So, the question is that we need to make a position independent. Sorry. Why? Yeah. That is because when you run, when you traditionally run an elf binary, it expects the entry point to be at a definite location. And it's very, it's fairly easy to guarantee because they've just created the virtual mapping anyway, right? So, I can just, that's free.  But since we only have one virtual mapping, again, we don't ever switch virtual mappings. We don't ever flush the TLB explicitly. That's why our context switches are so much faster, by the way. So, we cannot guarantee that the address in which your program needs to be loaded is available. So, and that's pretty much what position independent code does. Position independent code is code that you can load anywhere. So, chair libraries because, and that's pretty much the reason. So, some applications may not go well with OSV. We're not too much concerned about that because, again, we are very focused. So, if you're, if OSV cannot run your application, we're fine. So, the question is if either the first restriction that it must be a single process application is temporary restriction. And the answer is no. It's part of our architecture. But again, single process does not mean single processing. I mean, we support threads just fine. We just cannot have different processes. And what does that mean is that we cannot isolate things using memory mappings. So, when you fork, for example, you generate a completely different memory mapping. And that we cannot do because we only have one address space. Now, that address space can be running as many threads as you want because they share the address space. The process in which you do isolation through memory mappings that we cannot do. That's part of the OSV architecture. That's why you cannot fork, for example. And that's why it doesn't make any sense to talk about exec. Exec basically just replaces the virtual mapping, which we don't have. There are other small cavits as well. But they're mostly, I mean, it shouldn't be too hard, but we're not going to run anything. That's not our goal. So the question is whether we have any performance number for the Java workloads. I mean, we have some. We're not necessarily disclosing all of them. You want to comment on that? So we saw some performance gains with Cassandra, which is a particularly important workload because it's also a no SQL database. I don't remember that the numbers are not very high. They're not huge improvements right now. But we've only just started with optimizing the JVM. But they are better than Linux guests already. So we're ready better. But I'm not going to tell you how much better because it's a little bit unscientific so far. Hyper V. Yeah, so you're from Microsoft, right? So the question is whether you have any plans for other hypervisors and we would like to run everywhere. So as far as hypervisors are concerned, we would like to run everywhere. There are not that many hypervisors out there. What is in our plans is we have KVM running very well. We have Zen that the Zen support is still getting better. Yeah, the drivers. It's not just the drivers. It's also the timekeeping. But it's really not very difficult to port OSV to a hypervisor. It's just a few thousand lines for the drivers and just some glue code for detection. It's pretty simple. So if you happen to have a different hypervisor than send patches and we'll be happy to merge them. All right. So that's basically what we have. If you have any more questions now that we're done, I can go around with the microphone. So you don't have to repeat your questions. We'd like to thank you before we make a question. Don't forget to collect your boxer shorts and he has the first question. Hold on. And by the way, we are hiring. So if you like what you see, feel free to drop us a line and we'll be happy. So I know C++ but don't know Java. Am I still able to contribute something valuable to you? So yes, of course, we have places in all positions in the stack. If you know C++, you can work on the kernel code. If you know Java, you can work on the management stack. And if you know all of them, then you can contribute on the changes to the JVM that we're making that will make Java run Java and JVM based languages run faster on OSV. So the application would be a different part. So we have APIs that allow you to upload a JAR file or a war file or an ear file, whatever they're called. And they just get deployed on the instance and then you can start and stop them. It's pretty simple. I think there was a slide with, well, it would be pretty difficult to get to the slide with, we need an operating system just for showing slides. What happens if you have a bug in the JVM or in the application? So the same thing that happens if you have a bug in the application or JVM on a normal operating system, it crashes. Since it's just one application, all of the value is there. You don't lose anything. So with OSV, everything crashes. It just stops. It aborts and asks for help. So if you have a typical, if you have a multi-user system and the entire system crashes, it's horrible because all of your workloads crashed. But the way we see applications deployed today is just one application per server. So if the application crashes, the system that remains has no value. All of the value is in the application. So you basically restart the VM? Yeah. What about long-running processes? What do you do? Do you do live patching and stuff like that? Do you want to do that? No. The way we see it, that's not what people do on Cloud Workload. They run their instance and if they have a new version, then they just throw it on a new VM, points a load balancer to the new instance. If it's good, they kill the old one. If not, they kill the new one and try again. It's a lot more robust than the old methods. Is OSV capable of taking advantage of kernel same-page merging in Linux? So if you have multiple of these applications running on the same box, they're able to share memory resources? So OSV is completely agnostic to that. If you're running KSM on the host and your applications are similar and it will find similar pages, it's not really an OSV question. It's a KSM question and, yes, KSM will happily merge similar pages from different VMs. Okay. And a follow-up kind of unrelated.  Yeah. So I think that OSV is really a great fit for that because it's so simple. It allows the developer to concentrate on the application instead of the packaging. You have just the operating system image and you can push your application on that and then push the whole thing into the cloud. It's really very simple. There are no configuration files that you need to tune on your image. I think it's a great fit for the DevOps. In fact, it was designed. It was inspired by the DevOps workflow. Questions? I'm curious. Why did you use CFS? It appears to me that all this rate code and set pools and all this stuff is also not really necessary in the cloud environment where you usually have enterprise storage backends that can do all the stuff and can do checks something in this. So why the overhead there? Yes. So good question. CFS is indeed a fairly heavyweight file system. And I think it contributes most of the code to our system. But it does have features that are good for the cloud. You don't want to spend too much time running FSCK or things like that. So a system like CFS which has self-healing properties is good for the cloud. You may not always trust the infrastructure that's provided by the cloud. They tend to be commodity machines very cheap and you might not trust it. It's not all of them run enterprise class hardware with extreme error correction and error checking. Some of them are just disk, local disk attached to the server and you just hope it's correct. So we want to minimize the chances of an error creeping in. And also we wanted something that is compatible with our license. So we didn't want to take any GPL file system. And then CFS became the best content. And there is also something to add to that because you said yourself that in the cloud, most of the time you're not getting to the actual local file system. You have some storage in the cloud. So the file system that you have underneath is not terribly important performance wise because most of the critical workloads are not going to reach the file system anyway. So there were no reasons for us to go looking for extremely high performance file system. All the manageability thing is a lot more important than that. So the fact that, as I mentioned, all the checksumming and all the easiness and so on. More questions? So I have two questions. One is, do you support ballooning? So right now we don't have a balloon driver. It should be pretty easy to add since we already support the concept of my moving memory from the application to the system. So we could also give it to the host. It should be very easy to add. I don't think that balloons are heavily used in the cloud, but it should be an easy fit. Okay. And CPUs, you can't do hot plug right now, I assume yet. But would that be something that you'd be adding? So I could add another CPU to my already running application. Yes, so you're correct that we don't support hot plug. The code is written in a way that we can support it in the future. If there is a need, if we see that clouds start to support a hot plug, right now they don't. I don't think even normal enterprise virtualization do hot plug on a common basis. But if you feel there is a need, then a patch would be welcome. It shouldn't be very hard to add. Again, the system is fairly simple. So it's not completely trivial, but it's a lot simpler than the large multi-user systems. And it's fairly easy to add such a feature. All right. I'm actually not sure how much time do you have? How much time do you have? I guess five minutes anymore. See if you have more questions anymore. One over there. What is your concept regarding supporting different hypervisors? Is there a version for this hypervisor, a version for this hypervisor, a version for this hypervisor? Because you just talked about code recognizing. Is it one thing which recognizes on which it's just running or do I compile it for that, for that, for that, for that? It's just a question of how you do it. Yeah. Good question. So we have just one image and that image knows how to talk to Zen and knows how to talk to KVM and will know how to talk to VMware in the future. We don't have, since it's just a small number of hypervisors, it's very easy to build and support for all of them. Okay. Thank you. Part of your image already? Yes. So there are just six drivers. We need the one each for network and block for each of the hypervisors we support, and that might grow both in terms of the number of hypervisors and in terms of the different types of drivers. For example, KVM has both Verdioblock and Verdioscuzzi, and we will want to support both, but still the number of drivers is on the order of a dozen, so it's very easy to support and to compile into a single image. So there's no issue moving from one hypervisor to the other. You just take the same blob and flop it down. Yeah, no issue. It's just a few megabytes of binary. It's a very thin system. All right. Any more questions? Okay. So thank you very much for attending and come here, collect your boxer shorts.