around it and implement the same functionality in user space that was a crazy idea but now after four years we can definitely say that it works for us and it works for everyone else so the main idea here is to use slash proc file system to gather some information about the processes running there's a lot of info in there about the process so we can checkpoint them by using that information the other thing is we can use some debug facilities we also use some other mechanisms that are about to tell you that worked but in some cases it requires a few kernel patches so far we have about 200 kernel patches that are merged in the scope of the CRIU project in the Linux kernel and starting from I don't know one kernel CRIU works on that kernel so in some cases there is no info in slash proc we have to patch the kernel add some mechanisms and so on and so forth well I should have clicked the space so one another thing is currently CRIU is like a low level tool it lets you checkpoint and restore and it's it's no big use by itself unless you're doing some advanced thing it's better to be integrated in some other things so far CRIU is integrated in virtual and open it is now used for live migration and checkpoint restore instead of that old in kernel implementation that we had it's also integrated in Docker I don't remember which version though there is now say docker checkpoint and docker restore and it used CRIU to checkpoint and restored a docker container also integrated with Alex C and Alex D same thing integrated in RunC which is a low level of docker and so on there are some more projects so we are very well integrated as for CRIU itself we are currently at version 2.11 and we are doing predictable monthly releases so it's mostly it's to make the whole process predictable and make the district vendors more happy so we know the date of the next release and so on so let's take a look at what checkpoint is now the objective of checkpoint is to collect the complete state of a set of running processes everything that the processes are in order to be able to recreate that state later in a different place maybe it all starts with freezing a process tree so when the state is fixed and it's no longer changing so first thing we do we freeze the process for a process tree or a container next state is we carefully collect and dump the complete state of this app of a container that includes lots and lots of things opened files sockets, pipes all the established network connections all the signals, all the memory mappings, CPU register, credentials UIDs, GADs all the troops and the chairs and all these timers there's a lot of things to collect and we collect it all and we save it to image files, just some files on disk and we use three mechanisms as I mentioned earlier there's a lot of info to be a lot of info available from slash proc we use ptrace debug mechanisms for some stuff finally we have very interesting thing that we call parasite code injection there is certain information about a user space process that can't be obtained by anyone else except this process itself that includes the process memory that includes some other things too so at some stage of dumping what we do is we have our own very small code that we call parasite we inject that code into process and we let it run on behalf of that process and it collects some things for us the author of the original idea about the parasite code injection is Taehyun Heo who if I'm not mistaken delivered the presentation this morning right here in this room so kudos to Taehyun finally we collected everything and we can now kill this process set or we let it run it depends on what we want to do so this is a very simplified view of what checkpoint is and the details are available at our website criowork.com restore restore is the same thing in the opposite direction we have those image files with complete state of the processes and we recreate those processes we buy more things itself into this set of processes so criow runs reads all the image files figures out which resources are shared like there can be an open file descriptor which is shared among a few pids we cannot like do open three times we need to do open just once and then you know inherit a file descriptor through inheritance so we need to at this stage we need to carefully figure out which resources are shared and what is the proper order of creating those resources because of proper inheritance next stage is we actually go ahead and fork the process tree all the process that needed to be there to just do the forks at this stage actually we had to recreate the exact process IDs because pids are not you know you cannot change pids so we have to recreate exact process IDs and so on and when the restore some basic resources like those open files we jump into the namespaces if there are any we recreate private virtual memory areas, sockets although it deers in its truth finally there is that little thing that is called restore blob that's another small piece of code not unlike parasite code that we have for check pointing but it exists here because currently what we have that process tree we just created it's crew and we don't want any traces traces of crew in there so we need to unmap all the crew stuff and map all the stuff that has become the application so we need some intermediate intermediary code to unmap the crew you know unmap itself and then it becomes what we restore into and that restore blob also restores some particular resources like timers for example so they won't fire prematurely it restores the exact mappings and it restores credentials because for some things that we've done before we needed root now we don't need it so we do and it also restores the threads again it's a very simplified explanation of what restore is doing and the details are on that URL now the default application for checkpoint and restore is live migration and live migration works on top of checkpoint restore and works like that we freeze the processes or container let's say container and we checkpoint it then we copy everything to the destination server from server A to server B we copy the container file system, container files we copy those crew images we just created we copy some metadata I don't know docker config of a container and on the destination system we run the restore and we unfreeze this and it keeps running so here comes the live migration you can move a database server you can move pretty much any container that is not directly tied to some particular hardware and containers are usually not tied you can live migrate any set of processes that you wish to if it's dumpable and restoreable there is something wrong with this live migration any guesses maybe ok I'll give you a hint it's not really live this copy stage it takes a lot and during the copy the container is frozen it doesn't appear to be live it's not dead not exactly dead it's still there but it's frozen so we have some tricks up a sleeve to make live migration much more live and one trick is to use iterative memory migration that is a lot of that data that we have to migrate is just application memory you can run a huge database server occupying a terabyte of RAM and we need to copy that terabyte over to different server and as this terabyte of memory is not really changing every second what we can do is we can dump the memory of running process or running set of processes and then we copy that memory over and our server is still running fine and then we do the dump again this time we dump only pages that were changed so this is one part we had to modify the kernel to support so called soft dirty bit dirty memory tracking so we can ask the kernel please keep a list of the pages that are modified and then we dump everything and then we ask for that list and then dump it again and we do so iteratively hoping the set would shrink and it usually is shrinking and finally at some stage we decide to freeze it and do one last dump which would be way less than one terabyte we do one last dump and then we copy and restore as usual that way we can save a lot and you can save the frozen time from minutes to maybe seconds there's yet another approach to it it's called lazy migration so you either use iterative migration or the lazy migration so iterative migration is then we pre copy the memory lazy migration is then we post copy a memory instead of copying everything we just copy the freeze and copy the bare minimum and then we unfreeze the container but it doesn't have a lot of memory itself before unfreezing we had a sort of a network swap device with swap being the source server RAM and it tries to access those pages it got page faulted and those page fault handlers they go to the source and fetch those pages and in parallel with that we proactively migrate those pages and the app is still already running there there's another way to reduce the frozen time it's still a work in progress with OCRU there are people working on it as we speak and it's implemented through the user foldfd feature of the Linux kernel now yet another trick to make live migration more live I'm not sure if this is very visible but this is what we do in order to copy the data then we do first of all we do create dump and we write files to the disk then we do like scp or rsync or whatever to copy the data over to the destination meaning we read everything from disk and we write everything on the disk on the destination finally we do create restore that reads the data from the destination disk as you can see the problem here is we do twice amount of reads and twice amount of writes and it's slowing things down especially taking into account that disk is not blazingly fast so we can use couple of tricks to make it better and it's called disk class migration in our lingo before doing migration we either use existing or mount our own tmpfs instance which is basically a file system in rem so we don't have to do any disk IO and we run a very simple thing that is called create page server that just receives the pages and writes them to that tmpfs and on the source we say create dump or create pre-dump hydrate of migration and they say don't dump it to the disk dump it to the page server here's the ip, here's the port number and it dumps it right there to the destination and it goes to rem and then create restore just reads it from tmpfs which is a memory to memory copy so no disk IO and no double reads no double writes I could probably talk about live migration for hours but there is just one final thing I want to tell create is not a live migration create is checkpoint and restore it can be useful live migration right? and the full live migration is done by another tool that we have and it's called p-hole process hauler there's a little humpback course you know carries your process around it's create.org slash p-hole and that concludes my presentation just kidding beyond the live migration if anyone read the description of my talk you know some of the scenarios that create can be used for one such scenario is a safe button for a game that lacks one so before going around the dark corner you just do create dump and then you can return to this place and you can obviously do so with any other app when if you run I don't know some kernel compilation and forgot to move it to screen you can checkpoint it and run the screen and restore it inside the screen or t-max you can save a state of the loan job not for same thing but it's not for the game consider you are bioinformatics or that sort of thing and you run in some huge data number number crunching like I don't know protein folding and there's an application that does lots of computation and it takes weeks to you know finish the task it goes down and you lose weeks of work so you can do periodical checkpoints like every half an hour worst case scenario you lose half an hour of work in case of some disaster so this save the state is for HPC stuff mostly and then you can always do load balancing within the cluster move your containers around the cluster in order to balance the load better on your system and of course it's migrated together with the network connections so your users won't notice it this is live migration application of course not specific real migration and then there's that thing I don't have a good name for so I went for open heart surgery idea here is to do something with images in between checkpoint and restore so you checkpoint your process tree you checkpoint your container then you go to those image files and you modify something you modify a file name that is opened or you modify some ID inside there so we have a tool for that it's called crit and it lets you basically convert those binary images into say JSON and then you can modify the JSON and convert it back to binary and then you restore not exactly what you have checkpointed sometimes sometimes you have to there are some funny cases then you want to do something like that and I'm calling it open heart surgery because actually this is like a live patch and you checkpoint change something and restore and it keeps running but it's a little bit changed now anyone from the audience have any other scenarios in mind any cases yes please well Docker image format is just a file system what we mean here we don't save a file system create doesn't deal with the file system we save the state of the running process and to do live migration you need to copy those files over and in case in order to do a successful restore you have to have the exact state of the file system like if some file is changed or lost you might not be able to do the restore fortunately we have like file system snapshot and mechanisms for that and this is like out of scope for create so you can use like dn snapshot for that for example there's a feature of the Linux kernel you can use zfs snapshot for that yep yep yep that basically that means that you can like pose and unpause everything or yeah that's right yes please yeah of course yeah you can have some like states that you want to start from and you start right there from the state instead of recreating you restore from it right right I'm actually going to talk about this case yeah I have a whole plot about this case that's great yes please so you mean instead of starting from scratch every time you can start from some intermediate state that that applies to simulation or some networks that also can also apply to some qa work like there is that specific state you want to have the application to be in in order to be tested and it takes a lot of resources to go to go into that state you need a lot of you know steps and dances around for the app to be in that state and the next time you want to test it you just test from the checkpoint and it's the same yeah good so I have a few more cases but a little bit more details the first case is really very unusual we use crew to test the Linux kernel the thing is crew is pretty advanced software it does a lot of things with the kernel that no one else does and the other good thing is we have pretty extensive test suite that comes with it's like I don't know a few hundred test cases that around in different scenarios and of course we use this test suite to test crew to make sure we don't you know break everything and we test it on this table kernel to check if we if crew is still working for us but we also use the Linux next from Git that is to become the next Linux release and we run the same test on table crew this time to find potential bugs in the kernel potential regressions so far we have found more than 10 kernel bugs this way just by running some Jenkins jobs with Linux next Git report there's that URL crew or Linux next at least some of those and we decided that we need to document those cases next thing let's call a time travel or if you want to be less fancy that is reverse debugging or rewind same thing basically you do periodical checkpoints and in between you do some work and then you can restore rewind time travel to any old state a good example is that runkit.com service which I believe one of crew developers work for runkit what they do is some web environment for javascript developers and they do have that big rewind button so is a quote from the website next time traveling debugging using the revolutionary new technology called crew which allows notebooks to snapshot the entire environment that means you can rewind some processes the file system really just about anything in your session of course as I said before crew doesn't care about the file system for that I believe they use DN snapshot or something like that speeding up applications instead of starting the application it's time for scratch you just start it once when it's fully initialized you checkpoint it and next time you use restore instead of start well that's I believe the same thing as just mentioned about doing demos we did a quick test with Eclipse IDE which is famous for starting very slow takes about a minute and a half to start and it takes about 15 seconds to restore and in both cases we had the very same state of Eclipse alright you want to restore the SQL server in like a hot state and it caches already filled in I see yeah that's right and I know someone who's using this mechanism for a patch and PHP probably because there are a lot of modules in PHP and it you know does lots of reads and then places these modules in memory do some linking interesting thing is it's faster to just restore it and one last scenario is would be the booting of the mobile phone like you can boot it once and the check pointed at you're in the last restart you instead of cold start you can use hot restore I'm not sure but I think either Samsung or Huawei was working on something like that using Clio although I don't know any details this scenario was also mentioned this is the deferred or remote debug we actually talked to Jynx guys and they said they have that big problem like I don't know they have some big bank as a client and a pretty expensive 24-7 support contract and then the in Jynx server needs behave they want to debug it and the client of course want to restart it and then there is a dilemma if you restart it the bug is not fixed it will resurface and it will bug you if you debug it that means downtime well you can alleviate it of course but sometimes not if it's like a front end or something the only choice is to restart and then yeah you can check point it and then you can have both restart and debug using those saved image files same thing as just mentioned as forensics one another pretty weird scenario we use Travis CI extensibly to test crew anyone in the audience using Travis you guys so Travis is also my love it but one problem is they only provide that specific some that old version of Ubuntu 16.04 and what we want to test some of the latest kernel stuff that we added recently we want Ubuntu 16.04 and we want like a latest kernel and you can have neither you cannot have your own kernel you cannot have any other distro with Travis you're locked you can have it in container but you can't run your own kernel so the initial idea was pretty simple just upgrade that Ubuntu in place you just change trust it to the Nile in sources let's do get upgrade, get this upgrade and then you need to reboot into the new kernel the problem with this approach there is that specific control process that Travis run to keep control of the VM and once we reboot the process is gone and Travis loses this VM and it thinks it's not existent we cannot do anything it becomes unusable after reboot and I guess some garbage collector code just to kill that VM some time later because it's lost the idea is to make that control app survive a reboot it has the network connection to Travis it's like an SSH session and then they run something under that SSH session so we can do all the same thing but before rebooting we check point the Travis control app it's not an app it's like a again it's a small tree of processes and then we restore it it ended up not as easy as I'm telling you there are some interesting well work around that we have to implement because with a massive upgrade from 16.04 to from 14.04 to 16.04 pretty much every file on the system is replaced by something else then there are a few things but this is the article that describes all the magic tricks that we had to implement and it took about a day to make it work but it's working so in a sense we hacked Travis VM and we can now enjoy running latest and greatest Ubuntu plus our own compiled kernel in the very same environment and use this fantastic Travis server and it's working and the good thing is we already found one particular kernel regression using this testing using this testing in Travis so there was a fix that came in two patches and the subsystem container applied the first patch and didn't apply the second one so we got broken and we found this exactly using this scenario alright that concludes the part about different use cases and now I'm going to tell about a couple of things that we are working on lately leap soccer which is socket checkpoint and restore is library that I'd like to tag as LEGO for TCP sockets what it does is checkpoint and restoring of TCP connection of the established TCP connection TCP is pretty complicated and it takes a lot of effort to be able to checkpoint and restore TCP sockets so we had to this is yet another case that we had to patch the kernel for it to work we introduced so called TCP repair socket mode that lets you basically lets you recreate a socket without any network activity it's like when you that you can do sand and it's not sand and it just fills in the buffer you can set various options but that doesn't lead to any network activity so CPU uses this TCP repair feature to restore network network connections to restore TCP sockets and then we figured out that CRIU might not be the only user of the feature we took the code from CRIU and ported it out to a separate nice library which does this thing for you it's basically a library for the kernel functionality that we have it's part of the CRIU now but it's a separate library so if you're only interested in moving network connections between machines or checkpointing and restoring network connections or doing some tricks with network connections the open TCP sockets you can use this lip-socker thing one scenario is you can have a front server accept the network connection and then it moves it to a different machine and then someone else works with it and you don't have to use the full CRIU load for that then there is another thing that we ported out to a separate library and it's called compel that's that parasite code injection thing I mentioned earlier that's the parasite code injection for the masses so anyone can do things like that with compel you can do two things you can perform system calls on behalf of someone else or you can run your own code inside someone else's process and this is what we use during checkpoint stage this thing is currently in development and it's only available from the CRIU dev branch but we are going to finally release a development preview of the feature this month in CRIU 2.12 or you can just check out the CRIU dev branch and CD compel and see what's going on about it again this is still work in progress we can change the API so please don't rely on it being stable it's 0.1 0.1 at the moment and there is much more info about it at CRIU work.com compel I'm not sure if we have much time left if we do I can show you a simple case of using compel so you include compel compel h and then you do this you call compel stop task to freeze the process identified by PAD then you call some magic called compel prepare that prepares the process to be victimized by our parasite code then we can immediately execute some system calls inside this foreign process in this very example we execute close this call so we close any file descriptor inside any process finally we run compel cure to cure the victim and then we resume it once it's resumed that file descriptor that was opened is closed now I'm not sure maybe it will crash maybe it will figure something out but this is a good way to for example do some fault injection actually nowadays S trace can do fault injection as well but this is another way of doing it so that's as simple as that and new stuff here is preparation and you know roll back executing your own code inside a foreign process is a little bit more complicated I mean we tried very hard to make it as simple as possible oh one final thing you write this proggy and you compile it with some specific includes and then you link it with some specific libraries this is how you do it if you're doing it from the command line well makefile example is similar execute in your old code that we call parasite code first of all of course you need to write it you do some proper includes and then you have to have this function which is like a main for a parasite called parasite demon command and it has two arguments which is like a command and then some pointer to some data and you can do it like this if command is closed then you return this clause and the pointer is the pointer to the integer which is the file description and so on so forth so this is the very short example of the parasite code we're going to inject once written we need to prepare it first of all we compile it with some specific C-flex because there are some things that GCC wants to do for us that we don't really want in the parasite code so we have to disable something, enable something that's all hidden under compile C-flex then you link it with some not C-flex but LD-flex of course then we link it together with some plugins that I'm going to tell about in a minute and finally as a result we have some binary blob this is our code compiled and ready to be injected finally currently it works this way you generate a C-heater that contains this blob as well as some auxiliary functions for this thing to work and we end up with parasite.h which contains that binary blob inside it now we are ready to write the injection code the code that is making this parasite code work so of course we include that heater that we just generated and then we do the same thing as with CIS calls like with top task we prepare the compile they prepare the victim then we do some other things like parasite setup the heater is doing some magic to link the parasite then we run compile in fact that actually infects the victim with this parasite code and then we prepare the arguments supplying the type of the arguments it could be like int the type here could be int and then we initialize that in this scenario we are going to close file descriptor number 42 and then we do compile rpc call sync we supply the command and we and that's it the argument is already initialized the argument passion is through the memory you cannot pass it any other way once you're happy with executing those calls you can do as many as you like you can do it for threads and I'm not going to cover that and you can do it in a sync or a sync manner and I'm not going to cover that either once you're happy with your parasite work you cure the victim and you let it continue and this is what we do during the checkpoint stage can I have the next slide please open office some constraints about the parasite called there is no leapsie in there leapsie is in that most obviously most probably it's linked to that binary but we have our own code and it's not linked to leapsie so there is no string functions there's no memory functions there's no printf you cannot do anything from there even call this is called so the solution is we have some very small libraries that we call plugins that lets the parasite do some actual real work and we have the standard plugins so far which is linked by default and with that you can do system calls and with that you can do sterlan or memcopy and it also has some probably ugly and basic printf and vprintf and so on implementation so it's not usual v then you write parasite code but we try to make it as good as possible the other plugin which is still work in progress it lets you to set up a shared memory region between the parasite and the master process and it's still not ready for print time anyway there was compel compel you can use it for some things and I'll just give couple of examples here you can do log rotate without application knowing about it so you can just close file descriptor copy the file over open it and then keep the application running and your log is rotated even the application is not supporting it you can monitor process from inside for that the parasite needs to clone itself so it will keep running and then it will have some pipe and report some vital stats about the application being that application being a threat of that application you can update an application just like a runtime patching of the application you can actually be used to patch a security hole without restarting everything and I know that big vendors like Amazon they have problems with that they cannot restart everything immediately and once some next dirty cow will surface everyone is having a big problem so we had a guy who was trying to work on that and that means you can change some code running app or you can re-link it to a new version of libraries once these libraries are available and then finally you can probably have some application that works badly and has memory leaks and a little threat of yours can do freeze and leak cure those memory leaks so you can keep this cripple thing running forever there's much more again crew.org slash compelling usage scenarios I hope you have some questions is that normal questions yes we work with TTYS so we work with terminal about X application I think we work with X application as long as it's not X server itself if it's just some X application it uses some standard mechanism so we can checkpoint it either well of course we cannot checkpoint and restore everything that's sort of like it can be a problem so one thing you can do is to give it a try and see if it works for you we support more and more cases with every new release like for example we were not able to checkpoint and restore non-established TCP connection that is like in the final state or it's like in the pre-accept state and we still have to have a couple of very patches to do that we cannot checkpoint and restore everything that deals with some real hardware because part of the state is in the hardware and we cannot checkpoint and restore it but it works for lots of things I mean worst case scenario of course is you checkpoint something and then you try to restore it and it fails and if you haven't if you killed it that means you lost it so the way to fix it is to keep it in a frozen state unless you make sure you have restored it it works. So this is how live migration works it keeps the source in the frozen state unless this thing is fully restored once it's restored on the destination it gets killed on the source and if something is not working we still have the source frozen which we can unfrozen say well it didn't work but nothing is broken any other questions yes I believe it's like from 3.10 that is pretty old kernel by standard yeah you can do you can use Krill with that and then we can have Krill check command that tells which features are supported which features are not supported it checks it properly kernel and we can have Krill.org upstream kernel commit page which lists each and every patch that we merge into the kernel and what it does I said it's about I think it's 201 patches so far so but yeah again the best way is just to give it a try and see if it works for you and if not we are here to help I'm here to help use my contacts here it's either college kernel Twitter or Krill at openvz.org and Krill also has its own Twitter and we have pretty good community that is to help each other any other questions yes well I doubt that you mean gdb as new debugger no you cannot run your own code inside a foreign process context you can just take a peek at what's there you cannot just close the file descriptor you can just see what's there you can stop it you can analyze it you can see the variables but that's that's about it you had a question please well Docker is a good example you do docker checkpoints and docker restore and you know inside it's all Krill I can't think of any at the moment alright if there are no more questions thank you everyone for coming and I hope you enjoyed it yo turn it down you hear that buzz that's lovely