and actually both me and Mike work at Crealytics, which is an online advertisement management, ad management software firm. So we deal with a lot of data and most of this talk is basically coming from our experience. So bear with me if I go over some details or minor details because we are short on time. So we'll try to cover as much as possible, giving you a high level idea of how we move to Apache Mesos. So like most software startups, we started with a static partitioning setup. So we had a monolithic Rails application. We had some machines dedicated as web servers, some machines dedicated as application servers and then some data storage servers, which were like Postgres, Redis. And yeah, this was a setup that I think most of us are familiar with. We also had a similar staging and integration setup where we used to do our testing. This was all good until we had some real workload. And then like most of us, we hit the scaling problem. And not only just to handle the workload, we also were trying to break up our monolithic application into smaller services. So we went to ops and we said, hey, we need new servers or new somewhere to deploy all the three new services that we are building. And although everything was puppetized and pretty optimized, they were like, oh, new services, yeah, it might take some time. And this was something that we did not want because we wanted things to be quickly available. We wanted them to be reliable, like hardware, one machine goes down, we do not want the entire service to go down. So if you combine all these factors, this was quite hard to set up by a one or two person ops team. So we looked at some other options, like platform as a service and infrastructure as a service. We did some evaluation and for our setup, which we had like around 50 odd servers running on like simple internet, like hosting provider. If we migrated to Heroku or some other provider, we found out that it would cost us twice or twice that much and since we are dealing with a lot of analytical data, we had a lot of data on Postgres and that also was quite costly. And we did have sometimes spikes. So we'd like, for example, added more space or displace to our Postgres partition. That sort of fine grained control was not really available anywhere. So we tried to look at other options like OpenStack, CloudStack, and frankly, they were like not easy to learn on our own. So we continued looking. Okay, we thought, okay, we could also have another option. We can buy like some really large machines, like bare metal machines and have a lot of VMs over them. But we found out, yeah, in that way, the hypervisor would take up a lot of resources and we are really not efficiently utilizing the machines that we are using, that we are buying. Which was already a problem, like for example, you have three machines running as web servers, three as your application servers, with Unicorn or Puma and three as the database servers with master follower replication. So for example, the application servers were really used in the middle of the day. There was not much usage during the night, so all that resource was being wasted. The database also had a peak every now and then, but other than that, it was really not used that much. And the Nginx or the web servers were like hardly really taxed. So we had all this resource that was unused, and at the same time, we were paying so much cost for having these servers. So ultimately, we decided to come up with a list of things that we want for our infrastructure. So as developers, we want ease of deployment. I think we all of us agree, we want scalability, fault tolerance, and as ops, we also want to have high utilization. So we were looking for a solution and we then stumbled upon Apache Mesos, which we found was actually quite suitable for our needs. And hence the stock here, we'll give you a quick introduction to that and how it helped us. But the key thing with Apache Mesos, and if you just want to remember one thing from the stock is that it is this, that it treats the entire cluster as a single resource. So we know distributed systems are hard, like trying to maintain tons of servers as an individual components is very difficult. But what if you can treat all those servers or all that distributed system as just one single resource like your laptop, that makes your life really easy. And that's what Apache Mesos does. It takes all your servers and treats them as a single resource. So if you have eight servers with two CPUs and two GB RAM each, it will treat it as a single server with 16 CPUs and 16 GB RAM. And that is like a great abstraction for us to work with because we are good at working at single computers, right? So how does Mesos work at a high level? So Mesos has the concept of leaders and followers. So leader is a server or a bunch of servers that basically handle the requests, the workload, and then it distributes the load to the follower nodes. So it is recommended to at least have three leaders because in case one of the leader goes down, you still have your application available and then anything from five to 5,000 follower nodes. And how the leader works is that it uses Apache Zookeeper, which is again a distributed system platform for maintaining a quorum. But if you don't want to go into that detail, it's fine. You don't have to work with Apache Zookeeper at all, but it is used by Mesos internally to elect the leader. So if you have three servers that are configured as the Mesos leaders, it will use Zookeeper to elect one of them as the actual leader. In case that one server goes down, one of the other two will be elected as the leader. So a ball of text here, don't worry about it. I'll give you a quick intro in simple layman term. How does Mesos work? So Mesos is just the abstraction over your hardware. Like I said, it abstracts you away from the distributed nature of your computing resources and presents you with one single resource. It does not do any work itself. The work itself is done by frameworks that are running on top of Mesos and we look at these frameworks in a demo later on. And basically the leader acts as the traffic director, so to speak, between the work and the work distribution to the follower nodes. So the follower nodes tell the master that, hey, I have these resources available. It publishes it to the frameworks. The frameworks tell the master, hey, I want to do a certain task. And based on that, the master distributes the task to the follower nodes. So basically it's acting as a man in the middle and distributing work while also handling the requests that are coming from outside. In simple terms, yeah. If you look into the details, there is a whole lot of documentation available on the internet. If you really want to go into the gritties and you can create your own frameworks if you follow the API. But again, we'll not go into the details of that. So if we combine this with Docker, then actually we realize this has great potential because we can run our application in a isolated environment inside one of those follower nodes and let the master handle the load balancing or how the work is distributed between the nodes. But before we go into the details of this, I want to talk about Docker a bit because we realize not everyone might be familiar with it. But I guess can we have a show of hands who has worked with Docker really? I think almost all of us. So that's good. Then I don't have to go into the details of Docker, but I'll show you the small app that we have. We all know Docker is great for reliable deployment, isolation. So it uses the containers to really isolate, which is really good for us because we have multiple services running on the same machine and we do not want that the services should know about the existence of the other. So it gives you that isolation. You can build the image on your local development machine and ship it off without having to worry whether it will run on that server or not. So it's more lightweight as compared to the VM. Like I talked about earlier, we looked at VMs, but the hypervisor does take up a lot of resources, which is not the case with Docker. And of course it is open source. It's on GitHub. You can see the code, contribute to it, and it has great traction right now, which means that, yeah, it just keeps on improving with every release. So we have a live demo and I hope it works. So let me just try to switch over. We have a very simple JRuby app here. It's a JRuby Sinatra app. And it basically, this is the Docker file, but what the app does is really simple. It's a hello as a service basically. So you pass a message to the service and it says hello back. And we already have secured 20 million funding for this. So please do not try to copy it. That was a joke. Anyway, so yeah, we have this Docker container defined for this. This is the Docker file. We use JRuby 9K and actually the magic of Docker is that we don't even have to install JRuby 9K or any software on our servers. The Docker container takes care of that and I have a user there, which is basically where we run this app. And basically it runs a Puba server on a certain port and that's how the Docker container is configured. So I can do a Docker build for this and run the container, but since you all are familiar with Docker, I will not do everything. I'll just run the container, expose the port and hopefully this works. Just takes a while. And of course, the live demos are always a problem. Let's see if we can quickly see what's going on or otherwise, okay, seems to be working. Are the boot to Docker IP or something? It's actually back, so we have to use boot to Docker to run the Docker container. But yeah, it was working five minutes ago, honestly. But you get the idea. Basically what I have to do here is I pass the... Where is my mouse? So here is my Sinatra app. It has a route where I say greet and JRuby and it basically says hello JRuby back to be. So it's running as a Docker container without me having to install anything. That was our demo actually, which did not work, but let's just assume it worked. So Docker solves the packaging, distribution and provisioning of our application, but on its own it does not solve scalability. Like what if my application is a real hit and I'm getting 20,000 requests a second, one single container will not be able to handle it. And if I really want to scale up the containers, there is no easy way. Like I don't want to run the Docker run command onto individual servers by logging them or by SSH'ing them into them one by one. So that is what's something that Docker does not address on its own, but we did talk about Messos earlier where we can address the scalability concern. So we can combine Docker with Messos somehow and get this whole thing to work, which brings us to the next part of our presentation. Hello everyone. Let me switch over to a slide that you may have seen before, and this is something that Rocky talked about before. What if we have that Messos thing that I don't know how many of you are actually familiar with and we, a nice show of hands actually, has anyone ever heard of it? Oh, a few hands, cool. Nice. So the idea here is that we have those nodes acting as a single computational device and we run Docker, if it works, on there. So we don't have to have anything installed on there except for a Docker process, and we are good to go. Actually, Rocky covered this pretty well, but let me do a little bit of a refresher on the terminology here because I will be showing you those frameworks, so schedulers and executors, that run on top of Messos, and those can be schedulers, which are, think of them as crown tasks. So you need a nightly cleanup of logs or something like that. You can use a scheduler to schedule those kinds of tasks, and executors are the frameworks that actually run your long running application. We're talking web application servers or background processing, that kind of thing. This is an example of a scheduler. It's called Chronos. It has been developed by Airbnb, and we will actually not be showing this just in the interest of time, but it's fairly simple. It gives you an API that you can post to and basically just run all of your recurring tasks. An interesting tidbit, we actually decided to run all of our migrations as a Chronos task before we actually deployed to our execution layer, which is nice because being compliant with a 12-factor app, I don't know which factor it was, but I think it's a nice way to go. For the Messos execution layer, we have actually decided to use something called Marathon, and Marathon has been developed by a company called Mesosphere. It's getting a little bit, the terminology is getting a little bit out of hand, I think, for everybody here who isn't really familiar. So maybe we want to clear out what is Mesosphere? Mesosphere is a commercial company that has their own commercial product built on top of Mesos. They built something called DCOS, Data Center OS, and they actually sell this as a commercially-backed Mesos implementation. So if you have a large enterprise and you have a couple of thousand, or tens of thousands, I am not really sure, euros per month to give to these guys, you can, so that is one option. But the nice thing about these guys is that they open source everything that they do, including the executable layer, Marathon, which is why we chose to use them. There are other executors, of course, but this one we chose because it felt like it's the most supported and it works the best. So maybe I can show you our little Docker application that wasn't working before. Ah, now it's working. Hello, Jeremy. Okay, so this is our Docker thing running on our machine. But let me show you the Mesos. This is the Mesos dashboard, which actually is set up on my one-ping little machine with four CPUs and yeah, I gave it 2.8 gigabytes of RAM. And it basically says here shows that all four CPUs are available and all of my RAM is available, nothing is being used so we can start deploying to the cluster. We can also, let me show you, here's a list of the registered frameworks that I was talking about earlier. So we have Chronos registered here and we have Marathon registered here. And a list of slave nodes, we will only have one slave node here. Yes, so this is my master that is also running as a slave, basically a very simple setup to show you guys how this can actually work. So this is the underlying Mesos that provides that single computation interface. And here is Marathon, the execution layer. And here you can actually, through this runner, you can actually deploy your applications. If there are simple apps that aren't Docker containers, you can actually use the UI, but we won't really be using that. We will be actually using the API that this thing provides. And I don't know if I should actually, can you guys see it or should I zoom in to the payload? I guess this is better. So that Marathon dashboard that you saw also exposes an API that we can use. We can deploy an application as simply as posting to a well-defined API, where we set and send a body as a JSON payload, where we specify some variables that we want our application to have. For example, we want it to have two CPUs. We want it to have a certain number of RAM. We want to start off with just one instance. And we will tell it to use a Docker container that we have conveniently published on Docker Hub before. So we won't be really doing that now, but you can imagine doing that. Or you can have a private Docker repository. It doesn't really matter. Also interesting is that you can set up Marathon to do a simple health check of your application. So it will actually is capable of doing an HTTP request, which it will then show as green if it gives you a back of 200. So let me zoom out again. And let me send that post request. Postman is not really cooperating on this small resolution. Here's the send button. Let's send this request. And now, if we go back here, now you can see that the application is deploying. And after a while, it should turn green. We can also look at the underlying Mesos execution layer, which should now show that we are using two CPUs and we are using 512 megabytes of RAM. And it should say that the application is running. Yes. So let's go back here. Oh, and it's healthy. So let's maybe have a look at where the application is running. It's running on this IP, on this address. I have it conveniently in a tab here. And it's greeting me back with a very convenient name. Let's maybe show you how to scale this application. Let's say that I'm having a hard time keeping up with my request. I can scale it simply as saying that I want, let's say two instances here, so that I don't go over my limits. And after a few seconds of deploying, I will have two instances, which is very convenient and very hierarchal-like. Okay, so we have two things running. You may be asking yourself, okay, how do I see what the application is doing? How do I get it to logs? There is one solution that you can use, and that is the Mesos dashboard itself. We can actually have a look at the standard out of those containers that we are running. And also the standard error, for some reason, has the HTTP request, but I guess that's not true for you. And yeah, that is a solution. The real solution, let me get back to my slides here. The real solution would be to ship your logs into an elk stack, elk, for those who don't know this elastic search, lock, stash, and cabana. And the important thing to keep in mind here is that your services need to be logging to standard out, so don't log to your lock files or anything like that. Just log to standard out, ship that with a lock spout, and then use elk to be able to look at your logs really easily. So let me recap the benefits here, because we already said most of them, but they are easy scalability. Like you saw, I can just do a few clicks and I can have multiple instances to handle my load. Fault Torrents is a great thing with this kind of setup because you can have your servers going down willy-nilly and the infrastructure will redistribute the workload being done, and you shouldn't really feel that your servers are going down due to a hardware failure, unless it's all of them, but then you have a whole different kind of worms. You can also improve your utilization this way because you don't have statically provisioned boxes. You just have processes running on your infrastructure and you don't necessarily have to care about anything else. Yeah, these last two are very simple, but easy to scale down is also an important aspect because you can just take a node out if you are not using your infrastructure and again, you don't necessarily have to care because it's very simply configured, there's nothing running in there, there's just Docker and that's it. I would say that, well at least I'm always interested in the pitfalls of any kind of technology that people are touting, so let me name a few. The most, the one that I think gonna bite you the most is that the single abstraction paradigm here is not perfect. What I'm talking about is if, for example, let's say you have five slave nodes and you're using almost all of the CPUs on all of the nodes, so you only have let's say one CPU per node left, you will not be able to provision an application which you want to have two CPUs. That is something that it's not really possible to distribute an application that wants two CPUs across two boxes, so like I said, it's not perfect the abstraction, but you should just keep this in mind and not have your boxes completely full. You can have them somewhat 80% full but not completely full, otherwise you're gonna run into issues. You also are on the cutting edge, so you will encounter some bugs, but you are in good company. I will show you a slide a little bit later. You usually have good responses on mailing lists. And also worth pointing out is that the setup is not a simple traditional Rails setup with Puppet Run. It is a little bit more involved and I would recommend having a good operations engineer who helps you with this, but once you're set up, you hardly have to do anything. So you invest once and you reap the benefits for the near-term future. Like I mentioned, you are in good company. This is a list of companies that we know of use this in production. It is, I think those are pretty big deployments. I think especially the Siri one and Twitter one. But yeah, usually you get good responses on mailing lists and a few bugs that we encountered were patched pretty quickly. So I would say it's not really a problem that you are on the bleeding edge. Okay, that about sums it up. Here's our handles if you want to contact us or talk to us later. We want to do a special shout out to one of our operations engineers who pushed us in the company and who introduced us and helped us set this up. So yeah, we want to thank him again. And with that, I don't know if we have time for questions.