Hello everyone, thanks for being here, it's a real pleasure to be here today for the last day of the OpenStack Summit. I know it has been a really tough week and all the parties and everything, so thank you very much for being here. I'm Sebastien and I work as a Cloud Engineer at Innovance. Innovance is a multi-cloud provider, so we basically run design and build cloud platforms. We have several domain of expertise, among others OpenStack and Ceph. My daily job is mainly focused on OpenStack and Ceph and I also rotate between the operational, the development and the pre-sale team. Apart from this, I devote a third part of my time to blogging, so here are the details, my personal blog and company blog, so don't hesitate to have a look at them. During the next 30 minutes, we will be discussing the Ceph integration into OpenStack, so I'm going to briefly introduce Ceph or do some review, I'm not familiar with it. Ceph is a unified distributed storage system that started in 2006 during the HBHD. It's open source under LGPL license, so no vendor lock-in for the open source. It's mainly written in C++ and it's basically building the future of storage on commodity hardware, which is quite good actually because we don't have any restrictions, so we can just choose several really diverse hardware to build your first cluster and it evolves according to your own needs and it's also fairly easy to run a park and do tests, so that's quite good. Ceph has numerous key features such as self-managing and self-healing. The main point is it's a really dynamic cluster, so if something goes wrong, if you lose a node or if you lose a disk, Ceph will just trigger a recovery process because there are tons of health checks between each component, so as soon as the cluster detects that something is wrong, well, it will just self-heal itself. It's self-balancing, which means that as soon as you have a new disk or add a new node, the cluster is dynamically load balanced, so all the data are just moving. It's really penniless for scaling because it's fairly easy to add a new disk and to add just add a new node. Thanks to the tremendous puppet modules, Ceph modules and Ceph deploy, so now it's really easy to deploy Ceph and also to scale Ceph. Ceph is really unique because it has a really cool feature called CRUSH and it's just a data placement. CRUSH stands for control replication under scalable hashing. It's a random placement algorithm, which means that we don't do any, well, it's just fast calculation, so every time we want to store an object into the cluster, we have to compute the location, so we don't store anything into a hash table or something, we just always calculate the location, so it makes it deterministic and this is, well, this is pretty good. It's statistically uniform distribution, well, as mentioned earlier, as soon as you add a new node, the whole cluster gets rebalanced, so that's fairly easy to take advantage of the full hardware and it's a rule-based configuration. The really cool thing with CRUSH is that you can reflect your physical, well, you can logically reflect your physical infrastructure, so basically you have a, well, CRUSH is a map and within the map, you have your auto-atopology and the topology of your physical infrastructure, so you have your nodes, you have your disks, and it's rule-based configuration, well, I just said it's topology-aware, and then you can build rules, and within these rules, you can just specify replication count and things like that, so that's something really unique, and the really cool thing is that you can just specify that you have several hardware, you have diverse hardware, you have SSD-based systems, and you have a SATA disk system, so you can just say, okay, within this pool, store all the objects on SSDs or store all the objects on the SATA, so that's what's useful to have a really good placement algorithm. Just to give you, well, the final big picture of SEF, this is how SEF looks like, so everything is built upon the RADOS object store, so everything is stored as an object, and just on top of this, we build several components, so you have several ways to access your data and to store data, but first you have the lib-RADOS, it's just a library to access the RADOS cluster, so you can basically build your own application, and then from this, you can just write and access all the objects, so it's fairly easy to plug with the lib-RADOS because it has several language bindings, just Python, C++, Ruby, Java, and, well, a lot of languages binding. The first component of SEF is called RADOS Gateway, it's just a RESTful API, just really equivalent as what Amazon F3 does and what OpenStackSwift does, it's just a RESTful API, so you, oh, sorry, it has multi-tenants capabilities and it does code as well, you can do, it supports geo-replication and also disaster recovery features, then the second component is called RBD, it stands for RADOS plug device, and it's divided into two pieces, the first one is a kernel module, part of the kernel, so you can basically create a device and then map it on your machine, so you have a new logical hard drive disk, it's, well, quite useful, it's just the same way as ASCUSY does if you want. And then the second piece is just a QMU KVM driver, so you can create images, they are thin provisioned and they support snapshotting and co-pian-write clones, full or incremental, so that's really useful and it's well integrated into Xen and KVM. Then the last part of CEP is the distributed file system, it's called CEPFS and it's the POSIX compliant file system that supports snapshotting as well. Worth mentioning that all the pieces are really robust, well, except CEPFS, which is, well, not really, how do you say this, it's not production ready, that's, sorry? Almost, I think that's, yeah, it's mentioning always, yeah, almost awesome because everything is really robust and already really good, so that's, we are almost there to have a, to have a fully unified storage. Now it's some of the first consideration while building your first set cluster, it's really performance oriented, but it's just a general, just like a methodology when you want to build your first set cluster. The thing is how to start, first of all, you need a use case and well, when it, within this use case, you have to establish several rules like a, well, ideally you might be able to tell, okay, I'm more doing IOPS or I'm more doing bandwidth or perhaps it's mixed, but this will radically change the way that you build your cluster, if you want more IOs, you won't just pass this with more IOPS and if you want bandwidth, you might want a really large network bandwidth, for example. You might want to also establish sort of a granted IOPS for this, so ideally you will like to be able to tell, okay, I want to deliver this amount of IOPS and this amount of bandwidth for all of my customers individually, obviously it's really difficult to tell us, but well, if you can, let's just do it. You definitely want to know if you use SEF as a standalone solution or if you use SEF combined with the software solution, if you use with OpenStack or other cloud solution, for example, because it's, at some point if you have like performance issues, you must really want to know that this is implemented that way, but first of all, you know how SEF works, basically, and you know how it's implemented with the software solution, so if something goes wrong, you know directly how to look, so that's something that you want to consider as well, then you need to establish, well, what's the amount of data that you want to start with, and use both data and not group, then you need to tell, okay, SEF does the replication, so it just, you can specify a replica account, you know, you just have to decide if you want to start with three, three, four or more. Ideally, you would like to establish also a failure ratio, which means that when you build a cluster, you don't really want to build high density nodes. If you have 100 terabytes, for example, you don't really want to build 3D, well, each node, we have 33 terabytes each, because if you look at the node there, well, you have a lot of data to rebalance, so you need to just establish like a percentage ratio of the data that you're willing to load balance if something goes wrong, according to performance, and because as soon as you, as soon as you recover, well, you have to write a little bit more, and client keeps writing too, so that's kind of something that you definitely want to consider too. Ideally, you would like to have also a data growth planning. If you know that, I don't know, maybe every six months you're getting 10 more or 100 more terabytes, this will definitely change the way that you build your initial cluster, so maybe you will spend a little bit more of money, but every six months that's going to be, well, way easier to scale, and well, obviously, you need a budget, so I won't go through any consideration about the budget, but yeah, you definitely need something that backed all your requirements. Things that you must not do, but I just want to highlight that don't get me wrong, this is really performance-oriented, so obviously everything is doable there, but if you want to avoid like necessarily troubleshooting, you might want to follow these considerations. Usually, you don't want to put the red underneath your OSDs, but the OSD is the object storage daemon, and the general recommendation is just to use one disk for one OSD. Cep already does the replication, so it's quite, well, useless to do more replication with that, so you lose space, and yeah, it's not quite efficient. Just also think that, because you don't necessarily only want to do red one array, you can also do red zero if you really want to burst your performances, but degraded red breaks the performance, and then if you don't have the right tool to monitor everything, you might get into trouble, because usually what we tend to do is that the speed of your cluster is the speed of your, well, the slowest disk within the entire cluster, so if you don't want to drop down all the performances or have like spikes, well, just don't do this. As mentioned earlier, you don't really want to build high density nodes with a tiny cluster, because you might have a lot of things to load balance, and then potentially get a cluster for you if you have too many data to load balance. We could argue on the last one that don't run safe on your hypervisors. As mentioned, this is doable, obviously, but at some point you might think that you could get like way more performance, because if you have your storage layer and your hypervisor layer on the hypervisor, then you can just directly access your cluster, so the first it is just really fast, because you eat locally, and the second one is a little bit less, but well, that's my main concern there is about memory and also about consistency on the platform. Usually storage servers do only storage, and hypervisors they only do memory. Cep needs memory as well, because the more memory you have, the better for system caching you can get, so in this case, both of them require memory, so Cep wants memory, and obviously the hypervisor wants memory, so at the end you just end up with a really bad memory, but it's mainly an assumption. Now let's dive into the state of the integration into Evana. So basically, what Cep is good with OpenStack, because it unifies all the components. Originally it was present in Glance, and then in Cinder, and recently in Evana. So it unifies all the components, so you just have this really single layer of storage, and then all the components are plugged into the storage layer, which is quite good, because you don't need to have diverse storage solutions for one component or another. You just have the same abstraction for storage, and that's quite good. Evana's best addition, well first of all, there were a complete refactor of the Cinder driver, so now it uses the lib rados and the libRBD. This is really cool, because we can get a better error end link for that, but that's something that we had to do. Thanks just for doing that, by the way. We have new features like flattened volumes while creating the snapshots, because what happened on the background is that if Cinder detects that Cep is also the back end storage for Glance, when you create a new volume, then it creates a clone from this. So if you don't really want to have too much dependency on the chaining of the snapshots and the clone, you might want to just flatten the snapshot every time you create the volume. And we have also a new policy about clone depth, so that's also what I just said. As soon as you create more clones, at some point you just want to say stop, and then flatten the original image, and then continue cloning everything. Then Cinder backup was already present in Grizzly, but the only back end was Swift, so now we can do backups from Cep to Cep as well. This is, well, the way we can do it is you can backup within the same pool, which is not recommended, because it's just the same machine, so you don't isolate anything between domain failures. But if you have different pools, this definitely points to different machines, so well, you can isolate that. Well, ideally, you do DR with this feature, so you have one location, and you have another Cep cluster running on another data center. It supports RBD stripes, and the really important thing is that differential, so we already do actually incremental backups when back-upping Cep from Cep with another cluster. I know that yesterday we had a discussion, we've seen the guys for implementing an incremental API for backups, but it's just already there if you use Cep. And one of, well, at least for me, one of the biggest additions for Havana around Cep is the Nova Libre image type. Originally, this flag is set to file, which means that every time you create a new virtual machine, you get a file on the system under World Libre Nova Instances, Instance UID, and this file is just the root disk of your virtual machine. You also have a second implementation with LVM, so you specify your volume group, and then every time you boot a machine, it creates a new LV, and then it attaches to the KVM process. Now what it does is you specify a new pool of Cep, a new Cep pool, and then you create a new RBD image, and it just connects the pointer to the KVM process. So you just directly boot all the VMs within Cep. This is definitely in a further decision, so the client or the user doesn't know anything about this. This was a really huge requirement from the community and from our customers, too, to just can I just boot everything within Cep instead of always doing boot from volumes, for example? So it's kind of hard to automate boot from volumes and to item everything. So now we can just directly boot everything within Cep, which makes operation, like, live migration way easier, yes? It's only for KVM, yeah. The other question was, is it also compatible with Xen? And, yeah, sorry, but OK. It's not a Nova, well, it's part of a Nova and a Cinder addition, too, so now it opens up support QOS, which is quite good, because Cep doesn't do any QOS. At the moment, so every IO request are just restricted from the hypervisor itself, so this is quite useful to allocate a certain amount of IOPS or bandwidth from your hypervisor itself. It's bound with Cinder volume types, so that's good. That's the big picture of today's VANA integration. So what we do is we can just boot a VM, so it goes into Cep, and then we can attach a volume, so it's also calling Cinder for doing that. And we can also do Novaya Evacuate, yeah, that was the point before the question. Live migration is made easier when you have everything into Cep, so because you just have to move the KVM process and then you just reconnect the link on the RBD image, so it's just really, really fast. It's also fairly easy to trigger on Novaya Evacuate. If you lose a compute node, you can do either Novaya Evacuate or Host Evacuate, and it's already on the Cep cluster, but if you have this directly on the hypervisor, it's quite hard to reboot the virtual machine. And it's just like the workflow, as I just explained it earlier, we have the multi-backend capabilities, so as soon as we create a volume, we just do a copy on white clone, and we do just our RBD incremental backups on the second location. But the question is, is Havana the perfect stack? Well, unfortunately, we are almost there, I would say. We are missing some really tiny features. The problem is that we were about to submit a new patch, and then the patch for it was just rejected because we were just after a feature freeze, so now what it does is you just create a new VM, the compute, well, Glens downloads the image and streams the image into the compute node, so you have to download the image on the compute node through Glens, and then you have to import it into Cep, which is quite inefficient. But Josh has already a patch, and it's already on the pipe, so the idea here is just to do the same thing as we do already with Cinder, so when we create a new VM, and if the image is already present in Glens, then we just do a copy on white clone, so it's just really fast to put a new VM. So that will probably be for the bug we release for Havana. One of the things that is not implemented is the Cep snapshotting, so now every time you want to, even if you put a VM into Cep, and if you take the snapshot of the instance, what happens is that, well, this does the really common snapshot with QMU, so you have to download the image, well, just snapshot the instance, and it goes locally on the compute node, and then it goes into Glens, but in the future we could just call a Cep snapshot, so the operation is quite in one. You can do this instantly. If you're in a hurry to go into production and if you really want to patch everything, I think there are just only three bugs, and George already built a new branch for that, so this is really if you really want to fix everything already. A little bit about the road map, Icehouse road map and beyond. This is, well, at least for me, this is really personal once again, but this could be the Cep integration for Icehouse, and maybe for Jane. What's missing is something that you might want to do is to have the ability to store snapshots and images into different pools, because at some point you just want a replica count of two, for example, for the images, but potentially snapshots contain customers' data, so you might want a higher replica level, like three, for example. This is something that we just want to trick. Well, the Qco implementation is already on the pub, so it's not worth mentioning it for the Icehouse road map. Things that we would like to see is the migration support because currently Cep doesn't support the volume migration on the cinder. Basically, the volume migration is when you want to migrate from one backend to another, NFS backend into whatever other backend, but it's not supported when we use Cep. Something that we could easily implement is the Nova bare metal function. The bare metal is basically when you want to boot a new VM. It's actually not really a VM, it's more a compute node, so it's a dedicated type of, well, bare metal machine for your customer. Thanks to the kernel module, we could just easily enable the kernel module and then create a new RBD device and just map it to the physical host, so that could be really easy to do. There is also this LFS implementation going on, which is just a restful, really a, well, agnostic restful API, which can talk with Reddls and with a Swift cluster. So ideally, you will have this LFS API that talks to your Cep cluster, so it's not really a replacement for Swift or things like that, but it's just that basically for the OpenStack, you could also use the object store from Cep from the dashboard or whatever. So you will get a complete unified storage solution because that's gonna be just like everywhere. And well, potentially the Manila support. Manila support is an initiative from NetApp, I guess, and it's the distributed file system as a service solution. So we could also just add a new driver for CepFS and just create a new distributed file system for one of our customers because it's also a really huge requirement for legacy applications to use a distributed file system. This is, well, this is the Heist House roadmap, but it's just basically what I just said earlier with the picture, but it's just for you guys to have this kind of reminder there. What's coming up in Cep for the next release? We don't have that much like really new fancy features for Ampros or we just directly jump for Firefly, which should be landed in February, 2013. We have the cheering functionality, which is the basically have this notion of cold and hard storage, so you can have like a pool that has a bunch of SSDs and then everything goes into this pool and then, well, periodically we just flush everything on the back end with SATA disk when the data is less requested. We have the Erasure coding with it, which is just, well, more or less like a red five on the software defined storage fashion, so you can just like have a really large compression of your data. ZFS support for the fast system of the OSD, it's quite good because we, but it's really, really deep detail for this, so it's just a good thing because we can use red parallel modes with the journal, so we just do the same thing as we are supposed to do with ButterFS, but since it's not production ready, we can't use it, so that's definitely good that we can have the ZFS support, and obviously we will do, well, all the efforts that we can to fully support the OpenStack Icehouse release, it's both Intank and Community Roadmap, so that's it. That's it for me, so I'd like to thank you for your kind attention, and if you have any questions, it's time for questions. Made it pretty fast here, so we have like 15 minutes for questions, so yes, yes. It's more nova, but yeah, yeah, but well actually it doesn't support QCAL too, every time we do this, you must ensure that the image is in a raw format, so if the image, well, the image is already in glance, and it's already a raw format, so yeah, but the question is, yeah, it does QCAL because we do a clone, so it's copy and write cloning, yeah, because what we do is, as soon as we store the image into glance, the image is snapshotted and protected, and then from the specific snapshot, we just run a bunch of clones as soon as we create the new virtual machine, yes, I'm not really familiar with LFS, but as far as I know, and if someone knows more about this, feel free to jump in and explain us what LFS is, but basically it's just a RESTful API, agnostic RESTful API that can talk to several whatever object storage backend, so you just do a request on LFS, and then you can have either a Swift backend object store or a Rattles object store, there's just a way to unify all the APIs and just to have a single abstraction layer and no matter which object storage you have, and if I really have no idea, I don't think so, and I don't really know the status or know the progress of the implementation, so you should ask the GloucesterFS guys because they initiated the initiative, so yeah, yes, yeah, second is you and then yeah, yes? Encryption, I know that there is something, but it's a SAS specific, and there is also something on encryption, but I barely looked at it, so I don't have the answer, sorry, about the encryption. I know that there is a module that might have been needed for a VANA, but I'm not sure if it does anything. Yeah, I just don't know what to do. Yeah, it's always really complex because you don't really know where to store the key and everything. I know there is a project on this, like Barbican or something that does the key management system. Okay, that was you and then, okay. Well, that's a good question. Yeah, I mainly work on Debian-based systems. I don't know about the any kernel recommendation. Maybe you know Sage? Well, I think Sage is definitely the person to ask if you have any more questions. Yes? What is the single point of failure? Four? No, there is no single point of failure because it doesn't work like Swift, for example, where you have like this proxy that requires the object storage. You directly talk to the object servers, so you don't have any single entry points to retrieve your data. You just have to talk with the object servers. Okay, the question was if there were any spoke on SAP? Yeah. Largest installation? Largest installation? Well, as far as I know, it's five petabytes, the most or more or less, yeah, five petabytes. Any more questions? Yeah, sorry. What is more commonly used? Yes? What do people mostly use that for? Is there one use case that is a predominant use case? No, no, it's really flexible, so no, there is no specific use cases for that. Yeah, so what's the amount of, how many disks should I put into a single machine? Well, you have to take into consideration many things. What's your network bandwidth? What do you want to achieve also? If you're like really high ops, well, from a high ops perspective, you can pack a lot of high ops on a gigabit link so you can do like half an SSD and then your gigabit network bandwidth is full. If you have 10 gig, obviously you are like limited almost, but in terms of bandwidth, if you want to achieve performance in terms of bandwidth, just keep in mind that a single SATA enterprise hard drive disk can just fulfill your world gigabit bandwidth. Then if you go with a 10 gigabit network, you have to think about, so well, you can mostly deliver, well, one dot two gig. And SEF has a really specific design, which means that first you hit the journal and then you flush the data. So basically this more or less split into two, the IOs and then the bandwidth. So general recommendation is 12 disks per machine, but you can go to 24 if, well, that's only the theory on the bandwidth.  We'll just make sure that the red controller supports it. But yeah, that's, if you want to use an SSD, for example, just to burst the performance and don't have any impact because as soon as you hit the disk, you would eat at 50 megabytes, for example, and then you flush at 50. But if you use an SSD, the SSD just absorbs everything. But one more time, if you don't put too many OSDs on a single SSD, so based on work calculation, you have like an enterprise SSD can deliver 500 meg sequential writes per second because the journal is only sequential.  So you need to establish your own ratio for that, but traditionally I recommend like six OSDs for a single journal, which leads to, well, 12 disks in the end.  I don't know that much about Infiniband, but I've been doing that. So this will change everything if you go to Infiniband, but with gigabit link, yeah, that's the answer. That's gonna be personal on the internet, yeah. I'm gonna post it on Slideshare and that's gonna be tweeted by Inovance, definitely. So you will find it anywhere. I'll grab your email if you want. Okay, that's, yeah, gonna be soon on the internet. Yes, the region support? Ratio, yeah, okay. Oh, the interracial coding, yeah, and that's what you're all asking about. Okay, and I'm sorry, what's the question? Are we guys doing? I'm not doing anything for it. I'm not the main developer of the interracial coding and I'm not sure it's like there, I don't know, but I mean, yeah, once again, Sage knows about it. We have two more minutes. If you have any more questions. Okay, thank you very much, everyone.


Felly, mae'n yw'r cyfarfod a'r ardyfwyr i'r arddangos ar y cyrraedd cerddau yw'r cyfrifwyr wedi'i gweithio i ysgolion gweithio'r cyfrifwyr. Felly mae'r cyfrifwyr wedi'i gweithio'r cyfrifwyr. Felly, y fwrdd yw'r rhai cerddau bryd yn ddweud o'r cyfrifwyr ar y cyfrifwyr. Rwy'n ei gweithio'r cyfrifwyr, rwy'n eitio'r cyfrifwyr. .. implicat is the only way you'll be able to submit a new article for publishing on Wikipedia's live space, for the first four days of your new account. What it does, it will submit your article for review by another experienced Wikipedia editor and they'll decide if it needs extra work or if it goes into the live space. That only happens for the first four days After the four days, you can publish direct to Wikipedia's live ruthless yourself as long as you've made ten edits to Wikipedia, first of all and if you've drafted your article in your sandbox it's likely that you will have already accrue those ten edits anyway so... You can submit your draft review that's an option Mae'r efferddiol wedi byw yn ei weld yn ddechrau'n golygu, oedd yr aelodau aelodau, mae'n gweithio'n gweithio'n gweithio fel y dyfodos mewn ei wych. Yn dechrau'n golygon, mae efallai wedi oedd ychydig eich несu newydd. Felly, mae'n gweithio, i hyn i ethio y pwzeuant y cw süffyr, fydd o'n allan i'ch gweithio'n gweithio'n gweithio eich sydd yn y pwzeiwn i ymwg i gwychitellu'n gweithio. I'm going to change that to article space and change this to the article I'm writing about. So you want the title of the page to be exactly as you want it to appear. So if it's a person's name, you must have a capital letter. And you must also check that no similar page exists, first of all, with the same name. But if it does, you should disambiguate it with a little extra information that will make it more unique. And then a reason why you're moving the page. Well, created new article would be that. And you can tick this box to say that you want to be notified of any changes and watch any changes to the source page, your original page and the target page where you're moving your page into the live Wikipedia. And then just by clicking that blue button, that's all it does. It will move that article into Wikipedia's main space. And here's one I made earlier. There it is in Wikipedia's live space. So if I click that now, I would create another page with a slightly different title. But that's what it would look like. At that point, we would add categories at the bottom and hot cat can help with that. And we can also create a talk page for the page. And we can include templates as to which Wiki projects should be notified that there is a new article on their field of interest. And if you're not sure what templates to use, have a look at another article on Wikipedia. Because all it is is just a little bit of text with two squiggly brackets either side. But you can check other articles to see which templates they use to see which ones will be appropriate for your article. The one template that you should put on the new articles template is that this is a new user article.  They should offer help and guidance if need be. So there we go, a little smiley face. Now that's probably a good way to end. Cheers.


Paul Dawson, I work for a consultancy called CitrusBite. We build stuff with Elixir and Ruby. Somehow I got pulled up on stage to do live coding demos. I blame this guy. And we should also blame him if anything goes wrong. I'm not the one doing the live coding. I am James Gray. I work for No Red Ink and we use Elixir in production, which is kind of awesome. And I think the most exciting part of this talk will be driving down that ramp when it's over. All right, so sort of as we mentioned before, Hinabi is a collaborative game wherein you're not supposed to be able to see your hand at all. You're sort of giving each other clues. And you can see on the screen, my hand right here, this is sort of like a buildup of the clues that I've had thus far from James in this game. This is what I know about the tiles. But there's something that I think could make this a little easier for us, is if we also had a little bit of additional information about things that maybe we don't know. What do you think? You're saying cheat to win? I didn't go that far. Yeah. Okay, so let's see if we can make this happen. This, by the way, the game is just running on a digital ocean box. If we look here, I have a couple of scripts, no magic involved. Specifically, this upgrade app script is the one I'm going to use. We can look at it real quick. It's not fancy. It basically just builds a directory for release. It copies this file into the release directory. And then it does this fancy little upgrade command right here to upgrade to the next version. So if we run this, we get made release permanent 2.0, which is pretty cool. If we go back to our game, continue playing. I'm going to discard this five because I like to live dangerously. Please don't do that in a real game. And there's only one five in the game, so that was a terrible move on my part. But you can see here, we have a little bit of extra information down at the bottom that's been built up from the clues that we've gotten throughout the game. For instance, I know that tile in the middle is not white. I didn't know that before. That information is not apparent to me from the thing, which is nice. So we live deployed, and I didn't break it. That's always a plus. I saw a juggler one time, and he showed this trick. And it was really fast, and he said it took him a year to learn and get good at. And he did it, and everybody in the audience is just like, yeah, I guess. And he's all, I know it does nothing for anybody.  And then you could see what was going on, and it was super amazing. So my hope is that what you just saw is like, yeah, I guess. But maybe by the end of this talk, you think it's cooler than you do now. We'll see, we'll see. Are you saying this has to be super amazing? Well, it's after lunch, it better be. All right, so yeah, so a good place to start in our exploration of what just happened would maybe be at the beginning. Whenever we set out to build a game, specifically Hanabi, we were thinking a lot about what is Hanabi, right? Like what sorts of moves can you make? How would you represent Hanabi in a computer? We were thinking about our desired outcome that we want to play this game. We were not thinking so much about how we would put it in a database or anything like that. And so often whenever we're building applications as software engineers, we find ourselves in the situation where we're trying to put together something. We're trying to put together the pieces. We have a user story. We have a project manager who's saying, hey, let's build this thing. And we have put ourselves, we've grown up in these sorts of conventions like REST where we build these fancy routes that are loosely based on some database tables and some relations that we think should exist to try and make our lives a little easier. But over time, the intended experience that we're giving gets sort of precluded by the amount of plumbing that we have to build up in order to build that application, right? We end up in a case where we're doing 30 or 40 database queries just to put us back in a particular state. Just to put us back at a particular step in our application. In spite of all of our best efforts and conventions, this is sort of like the ball of mud you hear described sometimes whenever as applications grow, they get more and more complex and the plumbing gets kind of all intertwined and kind of leaky. Hey Paul, I don't want to interrupt your rant here and I definitely don't want to take away from this amazing picture. But can I tell you about another way to build applications? Does it involve Docker? No Docker required. Okay, let's do it. Okay, so I'm going to try this kind of radical idea, I guess. But instead of doing all these queries and stuff to re-establish the state of the world, another thing we could do is just keep the state of the world loaded. We could just have it in memory. And this allows us to avoid all of that redoing queries to remember where we were. And this also means that whatever this thing is we have in memory, it's probably just some kind of boring data structure that we can make simple changes to. And if we can remember what happens on both sides of the connection, then the messages that we have to send back and forth are really small and trivial, like move this thing over here or update this, right? And this may sound kind of radical for web development, but it would be totally normal for all other kinds of development, right? Like in a NERVs project, you would have your state in memory, for example. So in a traditional web app, if you wanted to store things in memory, we do this a lot, right? We use things like Redis or memcached to push something outside of the stack, right, so that we can remember it and keep track of it without having to worry about this request response cycle where we kind of have amnesia every time a request comes in. But we're on the beam. We have the beam options, so we can use processes and have them loop over memory. Or we can have ETS tables where values are stored outside of processes, right? So I would like to tell you that the memory state of this application is super fancy, but it's actually super boring. It's just a normal data structure that has values that you would need to represent a game of Hanabi.  And this is just a trivial data structure. Then we have to change state. We just have these pure functions that take in one set of state, fiddle a few bits in the state, and spit out the new set of state, right? It's just a simple change in state. Our messages, these are messages like that come from the front end to the back end, and remember I said those are simple. This is actually the largest message that the front end we were just playing with sends to the back end. And you can see that it says a player wants to give a hint to someone. And that's the full message. So if you're gonna keep this in memory, then you have to manage memory. But that's not too difficult to do. If you wanna make a new game, you can spin up a new process. If a game is done and you wanna reclaim that memory, then the process could exit, right, and that takes the memory back. You can even have processes check on themselves periodically and exit if there's been a period of inactivity. Say if two people are playing a game via email or something. And so they only move every so often. Okay, so that's about processes. Another tool we can use is ETS. This is a key value store that's available in Erlang, and of course, Elixir Bikes Engine. And it does simple insert, lookup operations, and also supports some not very complex queries. So if you want to do querying. The syntax can be slightly strange because it uses that concept called match specs, which is a little weird in Erlang. And then if you're getting at it from Elixir, that's like two layers of weird. So it gets a little weird. But for simple stuff, it's simple. This is useful for if you need to share state between processes, so outside of processes. It's also useful for just temporarily keeping track of something that you may not need to keep around long term. This is what an ETS setup looks like, or a very simple one. You can name a table, set some parameters for that table, what kind it is, and things like that. This is what an insert could look like. There's different ways to insert, but this is one. In this case, we're inserting a registered user into an ETS table. It just lets us keep track of their PID. And the value of the stored is just a term. It stands for Erlang Term Storage, I think. So it's just an Erlang term. In this case, a tuple, right? And here's a lookup, which is the opposite operation. From a different part of the code, the games table. This lookup is seeing if there are any games already prepared to start with a given number of players. So when you use this kind of strategy, you end up with this scenario where your state is living in these kind of ephemeral places, right? It's surviving a memory. These game manager processes, they're looping over game data structures. And those game data structures are the memory. And then the ETS table lives outside of the processes and different processes that are communicating with it. And this is where the heart of your application lies. So we have this neat abstract concept at this point. But outside of my test suite, there's no way really to interact with it, right? So what we want is a way to move tiles around to do stuff, to interact with it. And that means we need an interface. Wait, are you saying, so we have this cool process in the background and we want some way to talk to it from the outside world, that's what we're looking for? That's what we need. I think I have an answer. I read about, I think Rails is doing this thing with web sockets now. Let's do that. No, not really. Yeah, so obviously we should use web sockets. We're not going to use Rails. But yeah, so having a stateful connection to an outside user via an interface is what we're looking for. And web sockets are super cool. How many people have played with web sockets? They're great. Yes, see lots of hands because they're awesome. I find it very close how we tend to think about message passing within Elixir, like it's very event driven. You have subscriptions, you're paying attention to what's coming down the line and using them and Phoenix happens to have a very great abstraction for using web sockets. So I notice a lot of us are familiar with them, hopefully using Phoenix. The channel abstraction is pretty awesome. It kind of is reminiscent of how you would expect to use a gen server in Elixir. In our case, we have the game engine on the back and then we're using an umbrella application. So the Phoenix application within that game is actually completely isolated from the game engine and it's only reason for existence is to provide an interface for web sockets for the front end. There's no Ecto in the Phoenix app. There's not anything except for basically just the channel abstraction. And you can push messages to it. You can handle messages and this is a way that we, this is how most of our messages look in the application. If a play message comes down the line with a specific payload, then we handle it. We basically just hand it off to the game manager, which is just another process in the sky and does the right thing. Yeah, which is pretty great. On the actual front end, we chose to use Elm, which is a fantastic pairing to the two web sockets and two Phoenix and web channels specifically, I believe, due to some side effects of the Elm architecture. And without getting too deep into it, this little bit within the dotted lines is what is commonly referred to as the Elm architecture, whereas the model up at the top is a representation of a sort of user state. And the view is a means for representing that state. And then the update function is a method to update the model. And it is very specific and explicit in the ways that you are able to do that via certain actions. And so you can trigger actions from within the view if a user clicks on a specific button or in our case you can tie certain events that may come down a web socket through a channel to specific actions that get triggered in the update function and then cause the modeled update and then eventually the view, which is pretty cool. There is actually a Phoenix socket library. This is our example that we use for assigning the game and subscribe to the game and the game player message. Another side effect of the way that we chose to build this application is on the front end there's not really much logic that has to do with the actual rules of Hanabi.  And so really the only message we're having to listen to on the front end is I got a new game, what am I supposed to do with it? And that's what this message does. It basically subscribes to that particular channel and then uses a JSON decoder in order to decode what the game is and then redraw it. And similarly, publishing messages back to the back end is pretty easy. Here you see, I mean, we're just, we're publishing a discard message back to the back end that a user has made and it's the same thing, includes a JSON payload and pretty simple, it's pretty neat. So in doing this we've sort of created another problem, right? We have this very stateful connection between the user and we have this very stateful process up in the sky that's holding all of the state of our game. And this is kind of problematic because we live in the real world. And there's things like spotty Wi-Fi and computers crash. And computers are just computers and do things that we don't really understand sometimes. And that makes having all of this in-memory state and very, very stateful specific connection kind of problematic.  I have been trying to break that habit. So it's obvious that we need some manner of long-term storage to protect us against this problem. And we have to decide how we would go about that. And I think my instincts tell me that I should rebuild my logic, my game logic surrounded by these models, this database abstraction. But actually I'm gonna try to argue that that's not what we should do. Let's see how that goes. It's clear we need long-term storage to protect for the things that Paul mentioned are a problem, failures, things like that. He did forget to mention we needed to keep track of how many games I've won, which I feel is pretty important, but I'm gonna toss that out there. So in our case, we were using PubSub to communicate with the front end. And the reason to do that is though only one user makes a move, right? I wanna give a hint to somebody, they make a move. But actually the game updates for all the users, okay, maybe not in the case of a hint, but if you discard a title, for example, the game updates for all the users. So we send the moves up normally, but we respond, the game responds by publishing an update on a channel based on that game ID. And then all the players are subscribed to that channel or topic, I guess, sorry, to use Phoenix Terminology and get that update. Well, that means that storage can be just another subscriber to the PubSub system, right? So storage hooks into the PubSub system, it sees the events as they come out and it records them in a database. They reused append-only storage for this. We're gonna see the events as they come in, so it makes sense just to take the move and shove it in a database, right? And not try to keep updating and keep this complex game state in sync with whatever's going on on the other side, better to just record them. If you someday need to load a game, then you pull those events out of the database and you replay them on your data structure, right? And the advantage of doing something like that is that in the case of what we did on stage where we changed how the game works, when we replayed those events to refresh that game in progress, then it caught up with the state of where it was, but now it had that new information involved, right? Like you can change your mind about something. If you're calculating a score for something, you can change the algorithm, replay the events, and get back to your new score calculation using your new rules. This can require roll-ups if your data gets huge. Like if you're just continually adding to one table forever, it can become prohibitive to go back to the dawn of time and replay all those events. You can usually fix that problem by like snapshotting the data periods of time. And then grabbing the latest snapshot and then playing only the events forward from that usually solves that kind of problem. I'll also admit that it's not ideal for all tables. It works really well in the moves that we're doing here in the game. I also used it in the game table and I kind of regret it there. It would have just been better to have a normal record that I set, finished at flag on when it was done. So it's not great for everything, but it really suits the move use case. So loading games, we did have to make a tiny change to the game manager. And that's just to have it take a call back that it can use anytime it needs to load a game. So that might happen because of, in our case, hot code reloading. But it could happen because a game was shut down for an activity and then later brought back up or something like that. And so this is the only change we made to the game manager. It takes this call back. And internally, the call back just makes an empty new data structure. So our game engine still has zero knowledge of the Phoenix component and also the storage component. It knows nothing about them. This is, I'm not gonna show you the full code to load a game because a lot of it's just database concerns. But this is the beating heart of it right here. You take the saved moves and you just apply them one by one to your game data structure until it's back to where it was. So there are lots of pros and cons to the system. I mentioned that the game engine doesn't have any knowledge of storage. Storage is literally an add run. It's the last thing we built and we were playing with it before it was built. So it works fine without it, save that you can't resume games, of course. Also, there's an interesting kind of independence here. If your database gets a little behind, that's fine. And events can queue up in your storage systems mailbox. And you'll get to them when you get to them, right? Get clear of that. But that won't slow down gameplay necessarily, right? The gameplay is independent and players can continue interacting with each other. Now the con to that is you probably need some form of back pressure, which I haven't put in yet, or we haven't put in. The back pressure is needed because if the games continually swamped the storage system, then eventually you can get some kind of memory error or something like that. So you need to be able to push back and slow down if we're too far behind. Also, with this scenario, it's possible for some catastrophic failure to leave the two systems disagreeing on what the current state of the world is. That is a solvable problem. If you have to have that kind of assurance, you could use a strategy more like this, where the user system sends the event to storage, storage persists it, and then storage forwards the event onto the game engine, and then the game engine could respond to players with the update. That way you wouldn't process a move until it was persisted, so you would know it was safe. So this has different trade-offs. Again, you can still have the independence and the add on nature of it. That's fine. And we can guarantee that the two systems stay in sync, which is good. We give up that easy independence and processing speed. You can still do it, but you'll need to do something like a worker pool at the storage layer or something like that to get your kind of independent processing. Also, we have kind of a new trade-off in that you have to be okay with storing illegal moves, or in having the game engine ignore those when they come back to it, or validating in the storage layer before you save, that the move is valid. Our system doesn't have this problem for two reasons. One, the validation is in a separate module from the actual engine itself, so it would be no problem to use it at the storage layer. But two, the Elm front end is architected in such a way that it never sends an illegal move. So there would be no illegal moves to worry about, but you do have to think about these things. Finally, I want to mention one other potential strategy. You've probably heard the advantage of lazy computing, that if you make something lazy, then maybe an advantage is you never get around to invoking that code, it means you don't have to do it, right? If you don't design your system around models and you add it on as a planned feature, then maybe you don't have to. So like in the case of our ETS tables, they store games that are being put together in progress but aren't yet in progress. So one player joins and another player joins. They're waiting on the third player. If we had some kind of outage or something and that went down, players would have to reconnect to the games that they were in. But that's fine, we've lost their connections anyway. So having that information long term persisted has no value to us. And we just don't have to do it, right? Okay, this is all a great, we have a full system now. We can play games, we can restore things. We can keep it going until the time comes and it always does when you have to deploy, right? And then this is kind of bad, because now we've built up this whole system. Everything's in memory and we're playing everything from there. And basically at that point, we flush it all, right? So we lose everything. Cool, this is my best Morpheus impression. What if I told you that we never had to reboot the application? I would take the blue pill. Perfect, yeah, so we are working in Elixir, right?  The tool that we chose to use was Distillery. We've heard it mentioned a few times today. It is an awesome tool, it is super easy to use. This is an example of building a Distillery release. The one that we actually used today was built in this way. One note, even if you're using umbrella applications, specifically with Phoenix, you will have to compile your assets whenever you build the release. And then, yeah, you build a release and you can just run it, it's pretty nice. One other thing that I neglected to show during the live demo is that we bundled all of the early runtime system and everything together in the release. So that server on DigitalOcean that's running the game right now actually doesn't have Elixir installed, isn't running the OTP, isn't doing anything. It's just running from that one release that we pushed out. Similarly, to upgrade a release, you do the same thing. You tell which version that you're upgrading from and you build your release. However, I did notice that you do have to have the version that you are upgrading from on the same machine that you're building the release that you're upgrading to, if that makes any sense. So on the code side of things though, a few things had to change in order to facilitate doing a hot code swap. One of those is the versioning of the data. So we have this data struct in memory that represents our game and the current play state that it's in, right? Whenever you run a hot code swap, there's this code change callback that gets called and that manages the migration of your in-memory state to whatever the new state of the world is in your running application. And you use this VSN attribute up here to show that we are moving from this type of data structure that doesn't have the insights feature that we pushed today to one that does. And this is how you would go back and build that feature into the data struct. So in this case, we just had to call this replay game function that went back, read all of the game events that had happened and rebuilt the game state in memory, taking the new rules into effect that also included that extra clue information. Some of the advantages of doing this are that, I mean, we have years, we have decades of OTP release history and like how that is all handled as far as hot code swapping. It was actually really easy. I was kind of nervous when we were preparing and building this application and putting this talk together about being able to do a hot code swap like on stage while running Linux and giving it, I don't know, it seemed like a terrible idea at the time, but it actually worked out pretty well. Like distillery is a super neat tool. One thing though, this is sort of, I have it both on the pros and the cons list about you do have to be very specific about your application architecture, right? You have to be very concerned about where the state in your application is and how you are going to move it to a new version. So in our case, most of the game state was in a game module and it was very easy. We had to change a few things in order to be able to handle that code callback and to be able to replay everything back to where we needed it to be, but in a large existing application, it can be very complex if your state is spread across many modules and there can be some co-dependence and so being able to do a hot code reload like that does require some specific architectural decisions. Some of the downsides of approaching deployment in this way is you do have to build your releases on the exact same architecture that you're planning to deploy to. As I mentioned before, I run on Debian Linux full-time on my laptop. I was unable to build a release on my laptop that would then ship to an Ubuntu machine on digital ocean. And through some troubleshooting and open SSL conflicts and whatever, you can maybe make it work, but just generally you'll need a build server to build for the same architecture that you're deploying to. Another thing, there's this big section in the distillery documentation about how you should probably not do live code reloading in production. And I'm not sure, so I mean, I don't know. The way I kind of look at it is there are definitely cases, like this is a very simple example that we use for this talk, right? There are definitely cases wherein you have to deal with something like a database migration or you have multiple modules that are dependent on different versions of your data that you're trying to migrate during this hot code reload or there's some external dependency that you need to manage. And in all of those cases, you need to have an understanding of what the app file does, which distillery generally generates for you. You need to have some knowledge of the internals of how an OTP release works and how to build that out. And so that's another thing you kind of, maybe it's just easier to do a rolling release than it is to go back and really think about those types of things. And then like I said before, it would be difficult, I think. I use Elixir at work in production and it would be difficult for us to do a hot code swap on our application just because of some of the architectural decisions that were made in building that. So we built this application for the purpose of this talk and to show off hot code reloading and how it works, but we really learned a lot in the process of doing this and building it and making it work. This is some of our key takeaways for what they're worth. First of all, as Paul just said, you have to stay aware of where your state lives. If you spread little bits of state all over the system and then do something like a hot code reload where you gotta change bunches of those bits of state, that's gonna be harder, right? Significantly harder. So you do have to stay aware of that. This next one was a big aha moment for me, but if you can have this continuous conversation with both sides, where both sides are remembering all of the state, then it just gets drastically easier. You're not having to ship back and forth all the time, all these breadcrumbs so that you can reestablish the state of the world on the other side of the net, right?  So if you imagine a complex front end where you make some Hanabi move and it updates its internal representation of the game and then sends messages to the back end, trying to keep the back end in sync with what's going on, it gets so complicated, so fast. But in our case, because the front end is just allowing a player to click buttons that makes sense and the back end is doing all the game logic, the conversation is like player A would like to give player B a hint about it as fours and then the game engine responds with okay, then this changes this and you change those things and both sides are just very simple. I think we should strive to make storage just another feature in our application, right? That we shouldn't make it the center of the world that we base all things off of, right? If you have it as another feature, you can plan how and where it gets used and how that affects your application. You can choose not to use it at times and it doesn't have to get all tangled up with what you're doing about, in this case, running a game or in another case, surveying surveys or whatever it is, your app does. And then finally, this kind of strategy of app building does require a plan.  You can't just kind of look your way into this, I don't think. You have to think about it a little. Great. So that's what we have. All of this code that we use today is available in open source on that repo that we've been working on for the last few months. James wrote an awesome read me for that so hopefully you can poke your way around. Feel free to reach out to us for questions afterwards. Yeah, thanks for coming.