Okay. So, my name is Alan Jude, and I'm going to speak about Geleboot, which is booting from a fully encrypted disk on FreeBSD. So, I've been a FreeBSD server admin for about 13 years, and over the last couple of years, I've become a FreeBSD doc committer, and then a source committer, and then a member of the FreeBSD core team. I've also co-authored two books, FreeBSD Mastery ZFS, and FreeBSD Mastery Advanced ZFS with Michael W. Lucas. At my day job, I run a video streaming company called ScaleEngine, and I host the BSD Now podcast, and until recently hosted the TextNet podcast. So, I do a lot of work with ZFS, in particular, helped build the ZFS bits in the FreeBSD installer, so that you can more easily create a ZFS-based system, and then I did work to integrate boot environments, the Solaris concept of having basically a clone of your root file system, before you make an upgrade or a change, so that if it doesn't work, you can boot back to your base system how it was before, but your other file systems like your home directory, and bar log, etc., are not disturbed.  So, once we had that, I integrated into the boot loader, so that at boot time, you could select this alternate boot environment, because the most likely case when you would want to use a different boot environment, is when the current one doesn't work. So, that worked. The problem was if you used FreeBSD's disk encryption system called Geli, which is a block-level disk encryption, which you take your raw disk, you put Geli on top of it, and anything you write to it that gets encrypted, and then you can put a ZFS pool on top of that. The problem with that is that ZFS boot environments depend on the kernel and the root file system being the same file system.  That way, the boot loader could load the kernel, load the module that provides the disk encryption, and then mount the root file system. But having that two separate file system layout, it makes boot environments not work because you need to have your kernel be in sync with your root file system. So, I devised what's called Geli boot, which is a system where the lower-level bootstrap that loads before the boot loader could actually understand the disk encryption to be able to load the boot loader from the encrypted disk. So, especially when I started this, I was very novice C programmer. I had written one other tool before and that was, I like to use Shellscript and I wanted to use this new library previously had called libucl, which is a config file parser that also supports JSON. I wanted to use it for my Shellscript, so I wrote a little C program that would allow me to use the functions of this library from my Shellscript. That was all the experience that I really had. So, I attempted to build this, thinking it can't be that hard, right? It's only the boot loader. So, I implemented a very minimalistic version of Geli, which is the previous disk encryption subsystem, for the two different boot codes that are. So, there's GPT boot, which allows you to boot UFS and GPT-ZFS boot, which can boot ZFS. So, I spent a lot of time just understanding what was already happening in the existing boot code before I tried to modify it, and working with that, and I had to learn a lot about C because I really didn't know what I was doing, and it didn't help that all the existing boot code is terrible. It's all clutch that's built up over the years, and as soon as it starts working, everybody stops wanting to play with it. It's like, it works now, nobody's touch it. Then, support for GPT, the more advanced partitioning layout was kind of grafted on, and then it's like, nope, don't touch it. So, I had to navigate through quite a few obstacles, and just to figure out what was what, because there are four different sets of boot code, and most of the code is copy-pasted, but there are subtle differences. There are enough differences that you can't just diff the files to see what's happening, but not so few differences that you can't, a lot of the code looks the same. So, to give a little background, before we dig into what actually I wrote the code for, I thought we'd cover a little bit on how a computer actually boots. This is being like an i386 x86 type computer. So, when you power up your computer and after the BIOS does its post and props all the drives and that type of thing, it reads, the BIOS reads the first 512 bytes of your hard drive, which is your master boot record, and that consists of a 446 byte bootstrap program written in assembly, and then the remaining 64 bytes or so there give you a partition table where you can have up to four partitions. That's MBR. So, that bootstrap is then executed, and that bootstrap in the source tree, you'll find it as boot0.s, so it's called boot0. What that does is it examines that partition table, and in the case of MBR, finds which of those four partitions has the active flag set, and then inside that partition, you will find the volume boot record or VBR. So, it reads the first 512 bytes of whichever partition is selected as active and loads that into memory and executes that program. That one is called boot1 in the source code. That 512 byte of assembly program then reads boot2, which in the case of UFS is the first 15 sectors of the UFS partition, which the file system just purposely doesn't ever use so that you can write the boot code in there. What the boot2 contains is just a minimal enough bootstrap of a read-write version of UFS. So, all the code you need to be able to read and write files from UFS fits in 15 512 byte sectors. Now, that's not the whole file system, it doesn't have all the features, you can't do snapshots or anything like that, but it's enough that you can say, hey, in slash boot, give me the iNode number for kernel and then let me read the blocks from that iNode. So, once you've loaded boot2 and now have the ability to actually read a UFS file system, it loads the file slash boot slash loader, which brings up the beastie menu, which then after you wait for the timeout or choose an option, then the bootloader actually reads the kernel using its UFS driver which is more advanced and then the system boots. So, if you want to do that with ZFS, it actually turns out to be a bit evil. So, to boot ZFS from an NBR form at a disk, the problem is that the first 15 sectors of a ZFS partition are used by the ZFS label, so you can't just stick the boot code there, and the boot code is much bigger because ZFS is kind of more complicated than UFS. So, again, when you're booting off NBR, you have that boot zero, which is the first 446 bytes of the drive, which reads boot one, but boot one is different in ZFS. What it does is seeks into an offset after the ZFS labels in the ZFS on-disk format. They purposely left this region of three and a half megabytes of unused space to hold things like boot loaders. So, in there, you have this code that can read ZFS, and it's originally was 64 kilobytes. It's bigger than that now. But since it's 3.5 megabytes of space, it's fine. But in the code, it's actually padded out to a specific size so that the boot one assembly code can just say, copy this 64 or 128 kilobytes of data off the disk, put it in memory, and run it. So, that's the ZFS boot, which is basically a boot two. That one has enough understanding of ZFS to read only, so it can import the pool, understand read Z in mirrors and compression, and all that stuff to be able to let you boot ZFS. Then what it does is it actually reads slash boot slash loader and calls it with an argument of the globally unique ID for that Z pool. So, when the loader starts up with its more advanced understanding of ZFS, it knows which pool you actually want to boot off of. Because when it runs, it's going to find all the ZFS pools.  Once it reads the loader from ZFS, the loader presents the menu and then can load the kernel and you can continue. But most of us have switched over to GPT or grid partition tables. Biggest advantage is that you can have 128 or more partitions instead of four, which is handy. More importantly, you can have disks that are larger than two terabytes, which is pretty run-of-the-mill nowadays. But in order to make sure that Windows 98 and Windows XP don't offer to reformat the drive when they don't see a master boot record on the beginning of the drive, a GPT actually has as the first sector a protective MBR. So, it's a fake MBR that looks like it covers the whole disk, and it just stops legacy operating systems from deciding that that disk is empty and I can just reformat it. But again, the first 446 bytes of that are a little assembly program that boots the operating system. So, in the case of FreeBSD, that PMBR instead of looking for the active partition and seeking to a certain location, instead what it does is it actually understands GPT, looks to the partition table, and finds the first partition that has the type of FreeBSD-boot, which is basically just one specific grid that was decided to go years ago. Then it loads of all that into memory, either the full size of the partition or 545 kilobytes whichever is less. The main reason for that limitation is that, in this 16-bit real mode, you can only use the first 640 kilobytes of memory. So, we don't want to load more than that or we'll run out of memory. So, on a UFS-based system, this will be the GPT-boot code or in a ZFS case, that previous reboot partition will contain GPT-ZFS boot. That then contains what's called GPT-loader, which is actually a boot one. Again, that's a little 512-byte assembly program that just loads the next part. Then the boot two is GPT-boot or GPT-ZFS boot, which is again, a little program that understands UFS or ZFS and enough to actually read the loader or the kernel off that file system and start the operating system. What's interesting here is the GPT-loader has to relocate itself in memory to the location in memory where the BIOS is going to execute the operating system. So, to do this, it copies itself in memory, but to make sure that it doesn't overwrite itself, it copies it backwards, which becomes important later on. So, I now understood how the computer started up, and where all the different pieces were, and I wanted to now be able to boot from a file system that was completely encrypted, and so normally, I couldn't lead the loader file at this point. So, the first thing I did was find GPT-ZFS boot, and just copy that whole directory of source code to GPT-Gellyboot. My idea was instead of having GPT-boot for UFS and GPT-ZFS boot for ZFS and then making encrypted version of each of those, I'd make one big bootloader that could boot anything. That's still a good idea, but that's not what I end up doing. The first question was, if a system has both a UFS and a ZFS partition, and your bootloader supports both, which one should it boot from? There wasn't really an easy way to answer that question, so I decided that's for somebody else to figure it out. So, instead, I started by implementing Gelly in both of them separately, but doing it in line rather than having doubling the number of bootloaders that were available because it was already too many. Yeah. So, the ZFS boot is MBR only, and it's a fixed size, so I decided not to touch that because, I wouldn't be able to fit all the code in that fixed size and I didn't know what it would take to expand it at that time. The first thing I learned is that working with the boot code is difficult because there are no debugging facilities. If something goes wrong, most likely the machine will just hang with no output. You can't really attach GDP to it or something. Your only option is lots of printfs and hoping you can read text that's going by really, really fast. So, in order to actually get somewhere, the first thing I had to do was how to tell if this partition is encrypted or if it's not. So, the way free BSD's Geom classes work is the very last sector of a partition will contain some metadata that says what kind of class it is. So, in this case, the first couple of bytes of the last sector will say Geom, Eli, and then that means I know that it's a deli encrypted partition. If it doesn't have that, then it's not and I should treat it differently. What was interesting was trying to actually find where a partition starts and stops. So, in the boot code, there's some code that understands GPT and NBR, and it populates this struct disk. But depending how it started, it may or may not have a start offset set for where the partition starts. Then, so you have the partition table which has a start and a length for each partition, but then the disk objects you're working with may or may not have these start offset set. So, you have to just make it relative to what it says, but it's not always what you think it is. It turns out, depending which boot code you're working with, the numbers will be different and that makes everything more complicated. So, it turns out that the ZFS boot code actually made my job easier. So, what the ZFS boot code does is, instead of reading the disk directly, it actually uses a callback. So, it calls another function to do the reading. So, there's a function in ZFS and it takes as a parameter the function it should call to read from the disk. So, what I was able to do was just replace the regular read function with my encrypted read function that would then call the regular read, do the decryption, and then return the result to ZFS. So, that's what I did to move that and then I actually decided that I like that approach a lot. So, I changed the UFS code to basically do the same thing, so that I could implement it the same in both UFS and ZFS and try to make it easier to understand. So, after I figured out if the partition is encrypted or not, the next thing I need to do is actually decrypt it so I can start reading from it. So, by looking at the Gehli metadata, which is in the very last sector of the partition, I get the basic information I need, like what algorithm it's encrypted with, how big of a key it uses, and an encrypted copy of the master key. Once I have that, I can then try to decrypt the master key by asking the user for the passphrase. If the user provides the correct passphrase, then an HMAC will match the signature in the Gehli metadata, and I know that they entered the right password. If not, I can prompt them again in event and then eventually just give up on them. So, at this point, the problem was I needed to have some crypto. I need to be able to decrypt that master key, and the bootloader doesn't have any crypto by default. For UFS, the bootloader was 14 kilobytes. There wasn't any extra scope in it. When I started, the ZFS one was 47 kilobytes, and most of that was the checksum algorithms and the compression algorithms that ZFS needed to be able to read the disk. So, I looked at how Gehli does it, but it uses the kernel's crypto API to support offloading to crypto acceleration cards and stuff like that. That was way too big and complicated to try to put in the bootloader. So, I used Google and I just looked for like tiny basic C AES implementation, and I found this thing called tiny AESC on GitHub, and it was BSD licensed and it allowed me to get started. But as I started working with it, I realized it only does AES CBC 128, not 256, and it doesn't do AES XTS. So, while it would get me started, it wouldn't actually solve the problem. So, I purposely created some encrypted disk using the one algorithm I had support for at this point, and fiddle with it enough until I got it to actually work. So, by stealing some functions from Gehli that read the metadata and do the decryption, and just replacing the calls into the kernel crypto system with the basic C implementation I had here, then I could decrypt and validate the master key, and then using that, I could calculate the HMAC and make sure that the master key was correct, and then I could use HMAC that Gehli uses to calculate the sector key and the initialization factor for the actual encryption, so I can decrypt random sectors. Then, I looked at what other bits of code I was going to need, and it turns out that Gehli uses MD5 for the signature on its metadata. So, this is something across all the GM stuff is that the metadata that's written in that last 512 bytes on the disk has an MD5 hash just as a sanity check. It's not really meant to be cryptographic or anything. Inside Gehli itself, it uses SHA-256 for all generating all the unpredictable IVs for the disk encryption, and then it uses SHA-512 for all the HMACs for verifying that the master key is right, and if you actually have the data authentications or not. So, now I needed all three of those hashing algorithms to fit in the boot loader. So, I first tried to just include them, like you're supposed to be able to do, is just grab the headers for each of them and go, but it turns out doesn't work like that. So, elsewhere in the boot code, when they need some code, they actually used include on the .c files, just bringing that stuff in line and try to make a small boot loader. But the problem is that there was a bunch of conflicting defines in AS-256 and AS-512s, where defining the same function name, but they actually do different things or have different size values. So, then the approach I took was, there's this LibStand 32, which is a 32-bit version of LibStand that the boot code uses. So, I just hacked up the make file, so it would pull in the existing chunks of code into this library and expose all the symbols, and then I could use them from the boot loader. Eventually, I replaced this by creating my own library, LibGallyboot that brought in all the dependencies I needed for that, instead of loading LibStand, which is also used by other things, and was causing all the binaries to grow instead of only mine. So, then I had to prompt the user for the password to be able to decrypt the disk. When I was first writing this, I just hard-coded test pass as the password in the boot code and created encrypted disk that were with that password and eventually got it to work. But then I needed to support actual passphrases. So, then I was like, it shouldn't be that hard to ask for a password, right? So, I went into the common console code and got the getster function, which allows you to get text from the console. I changed it, so instead of echoing back the characters you type, it would echo back a star and it should be just all well and good. But it turns out the loader reuses some of that code, but it actually uses completely different functions to read from the console because it has to support serial. So, I had to write a completely different implementation of the same thing. But then during code review later on, we found that actually both of those functions, the one in the console and the one in the bootloader contain the same bug that was found in NetBSD like eight years ago and fixed. One of the break or something, they didn't use curly braces on one of the if statements and they had two things in the if and it looked like it was working because of the indentation but it wasn't. So, this would cause it to not stop when you reach the maximum number of characters and you just keep overwriting memory and doing crazy things, which is not what you want to do, especially with a password where you've assumed it's only going to be this long and you're going to overwrite that memory with zeros later and then not overwrite anything in excess. So, instead I actually found the version from NetBSD and get s that was actually correct from our LibStand and wrote my own pwget s and put that in my little library. Future work for some days go back and fix all the other ones. Actually, here's the code that's wrong. So, if you haven't overflowed the buffer, then this should be inside this if instead of not. So, if you haven't overflowed the buffer, keep counting but otherwise put the character there anyway. So, it doesn't do what it's supposed to do. So, at this point I thought I had enough working so I could actually take it for a test drive. So, my expectation was that Boutu would start, taste the partition, determine that it was encrypted, then read the master key, which is encrypted in that last sector of the drive, decrypt it with the passphrase I just provided, and then stand ready to determine the sector key and decrypt individual sectors of the drive as I tried to read them. So, I booted it up in VirtualBox and VirtualBox got some kind of triple fault and it just crashed. We're like, what's a triple fault? So, I wrote it out to a USB stick and tried it on my laptop and it just rebooted in a loop. I'm like, this is not going to work and I have no idea why. So, it turns out by adding all this, I'd actually cause the boot code to go beyond 64 kilobytes, which at the time I didn't know was a magic number. So, now the boot code was like 70 something kilobytes. It turns out the GPT loader, that little 512 byte assembly program that grabs GPT ZFS boot copies it to the right place of memory and lets the BIOS execute it, only copies the first 64 kilobytes, because that'll be enough for anyone, right? Because when I started the project, the UFS one was less than 16 kilobytes and even the ZFS one was 42 kilobytes. So, 64 was actually probably enough. But once you started adding the encryption and hashing algorithms, all of a sudden that was not going to be enough. But because this is 16-bit real mode in the OS, you can't actually copy more than 64k at once. So, it wasn't just a matter of changing the assembly code to copy more data. I tried that, it didn't work. I tried that and the assembler actually laughed at me, and then put it back to 64 kilobytes. Yeah. So, I was over 90 kilobytes and this wasn't going to work. So, I was like, okay, put that aside, let's try UFS. It's small enough, I think I can fit it all. So, I stuck with just UFS, even though this whole project was only originally for ZFS, and I didn't think I cared about UFS, but I was like, UFS can't be that hard. So, I decided to use UFS and then basically did all the same, reused all the same code to decrypt the master key and everything. Luckily, when I compiled it, it was under 64k. So, I put it out and booted it up and the decryption actually worked. I'm kind of glossing over like a month of it not actually decrypting correctly and printing out Hex stumps to the screen and trying to figure out what's not working. But anyway, so now GPT boot decrypts the file system and it was actually able to read slash boot slash loader and run it. So, I had actually decrypted something and run it. I was very happy. So, the loader starts and immediately failed because it tried to read the disk expecting UFS and got gibberish because it was encrypted and so, it was like, I can't do anything with that. So, it had all worked except it didn't actually go anywhere now. So, I could load the loader but then the loader would just laugh at me. And of course, the boot loader isn't much better, although slightly better than the boot code, which is the bootstrap that loads the boot loader. So, there's no kernel, there's no libc, there's no malloc implementation, there's no panic, there's nothing you can do. There's even complications with that. When you're trying to use code that's meant for user land, it wants to include string.h. But when you're building the boot loader, you have libstand which has its own string implementation and you can't have both at the same time. So, that complicated things. So, in the boot code, the actual like UFS and ZFS implementations, there is a malloc function, it just doesn't do what you're used to it doing. So, in the case of the ZFS one, there's a three megabyte stack variable set aside. And when you call malloc, it just increases the counter to what offset in that variable you are, giving you back a pointer to some memory you can use. But there's no free, right? So, you have three megabytes of memory and you can allocate, but if you free, you don't get anything back. And so, you can only have ever used three megabytes, not you can only use three megabytes at once.  So, the next step was to teach the loader how to actually speak Gehly so that it could load the kernel so that the operating system could start. So, I had to figure out where in the loader it actually reads from the disk so I could do this same hack of, oh, I see you read some data from the disk, let me decrypt that for you before you try to use it. So again, I had to do, taste the disk and determine if it's Gehly, read the master key, prompt for the password, do all that stuff. So, in the loader, it has this array called file systems and it has all these different file systems defined, UFS, ZFS, NFS, et cetera. So, my first thought was, oh, I could just add, you know, Gehly underscore UFS is a new file system and wrap all those functions. But when I looked a little bit further, it turned out all the different file systems that I actually expected to read from the disk used libI386 to actually do the BIOS calls that read from the disk. So, I intercepted the data there instead because it was less work. Probably not the best place to actually put it, but it was less work. So, ideally we'd implement it as part of probably the read cache that Thomas Soome did for Alumos and then port it back to FreeBSD, but that's future work. So, after I basically redid all the work I just did in the loader instead of the bootstrap, which is relatively different code, I was stuck here. So, I had successfully actually booted a Gehly encrypted disk with a UFS file system, but I only had support for AESCBC128, which nobody wants to use. There was no support for ZFS. I had this 64 kilobyte binary size limit, which meant no ZFS and that if I added much more complexity to it, I was going to have no UFS either. So, while I had done a lot of work and actually accomplished all these things, I was actually nowhere. So, in order to go any further, I needed to get rid of this 64 kilobyte limit. So, I tried some naive things. I tried to compile with optimized for size, but that didn't really help. I tried optimizing it more or switching to O2, but that just made it bigger. I tried increasing the number of blocks. So, in the GPT loader assembly, it's like copy this many 512 byte blocks. It's like 80 because that's five hundred or 64 kilobytes. I tried increasing it, but the assembler is like that number is too big. Sorry, I'm putting it back to the maximum. The fact that it still compiled and put the number back to a small number instead of just failing kind of amused me. So, then I reached out to various other people at the FreeBSD project to see if I could get their help. The first person I asked was my documentation mentor, the person who was mentoring me for my documentation commit bit. He looked at it and thought, well, this shouldn't be too hard to convert to 64 bit, but it turns out that doesn't actually work like that, or to make it copy two blocks of 64k instead. But he didn't really have time to write the assembly for me, so I tried someone else. John Mark Gurney seemed receptive to the idea at first, but once he understood that it was the boot code and that it was complicated and 16 bit real mode, he quickly suggested I ask someone else. So, then I asked the original author of GPT-LDR, John Baldwin, and he suggested that you shouldn't make the boot code bigger, you should solve this some other way. Like having a partition that had only the loader on it rather than the entire slash boot and hacking around it that way. Well, that's not too terrible, it's just a lot messier than the way I was hoping to do it. Then Peter Greenhan is like, oh yeah, sure I can help you with that. Then he's like, actually here, let me teach you how to use QMU and GDV to debug the assembly instructions as they go by. I'm like, if I knew how to do assembly, I wouldn't have asked you. Then at VBSDCon, the BearSign conference would have been 2015, my friend Dylan was there so I asked him and he started trying to write new assembly and sent me a couple of bits of code but they didn't work, and he spent the rest of the conference with a pad of paper drawing the memory stack, trying to figure out how to do it. But after the conference was over, he had to go back to his day job and didn't have time to help me with it. Then at UroBSDCon 2015 in Stockholm, Sweden we were having the Dev Summit and a one-point call in personal approach means I heard you're having trouble with some 16-bit assembly. I'm like, yes. He's like, ah, that's 16-bit assembly. I know. It turns out that was the only assembly he had ever done, but what he had done when he was in school and still remember how to do it. So that night he wrote up a draft patch for the assembly code and sent it to me, but it just crashed the BTX client, which is one of the little pieces of the bootloader. So his first patch didn't work, and then he sent another one and that still didn't work. Basically, he sent me a new patch each night of the conference until it was over, but none of them worked. So we all went home and I kind of figured I was out of luck still. But then a couple of days later on IRC, Colin poked me and sent me a new patch. This one still only copies 64 kilobytes, but it does it by copying two 32 kilobyte chunks. So try this on a regular size bootloader that we know would work and see if it works. So we tried that and it did work. So we were moving somewhere. So then a couple of days later, he sent me a later draft that copies four 32 kilobyte chunks and it actually worked. Now, if you ever need more, you can just increase this number. So it's future-proof. So now that I had solved that and whacked all that together, I could now boot ZFS from AES-128 encrypted volume. But that's not the disk encryption people want to use. You want AES-XDS because it's faster and designed for disk encryption or at least AES-CBC-256. So now that I didn't have the space constraint anymore, I could ditch the tiny AES implementation and get a real one. So I actually stole the canonical one from the kernel and started just by including the C-files into the GPT-ZFS-boot-C. Play with it some more and then I needed an AES-XDS implementation, which uses this code but has all the functions that do the tweaking for the initialization factor. So I stole that from OpenCrypto, which is a framework originally created on OpenBSD, I think, that provides all the different crypto functions and offload versions of them for crypto cards and so on. The problem with that was all that code, it was in one giant file, xform.c has all the crypto transforms. So it's got Cast, Blowfish, DES, AES, AES-XDS, AES-CBC, AES-GCF, all the different ones. That was a lot of code that I didn't need. The other problem was it used kernel malloc implementation, which on FreeBSD has extra fields because you define what type of memory it is you're allocating and which bucket you're allocating it from. So I made some changes to the implementation, which are basically I copy and pasted just AES-XDS from OpenCrypto into a new file and use that and modify it so it wouldn't actually malloc any memory, it would use stack variables. So once I had that, I had a working UFS and ZFS that actually did the disk encryption people might want to use. But it worked now, but it wasn't something anybody was going to let me commit to FreeBSD because it was a mess, a very ugly mess. So at this point, the code was all ugly, it's full of debug printfs that are printing out bits of pieces and things so you can tell. I'm just pounding including a bunch of C files out of the kernel, it worked but it was ugly and I couldn't just include some of the stuff from OpenCrypto because the file was too big and it had, so X4.c had in addition to every crypto algorithm had every hashing algorithm including all the Mac variants and the deflate algorithm. It's a really big file. So I asked the FreeBSD security officer, what I should do about this and they said, it probably makes sense to actually break up OpenCrypto into the individual components. I'm like, they'll let me do that. So they did, so I actually used SVN copy to copy X4.c to X4 underscore each of those different algorithms and then deleted the XS code. I'm not sure if that was the best way to do it, but in the reviews, in the diff I created out of it, it was clear that I didn't add any new code. I just copy and pasted that same file and deleted everything except for the bit I wanted for each file. So while it made a bigger diff to review, it showed that I didn't actually change even accidentally any of the implementations. So I'm not sure if it was the best way to do it for the code review, but it was for me, the easiest way to show that I didn't accidentally break any of the crypto. Because that was what I was most worried about. So at this point, most of the GeliCode had survived not being that modified. I just copy and pasted it. I hadn't had to go through and remove Malix or anything. So I looked at just being able to use the code from the kernel by linking it into the lib GeliBoot instead of copying and pasting it. There were a couple of places, the struct GeliSoft-C, I had switched to the simpler metadata struct. So I had to undo that. So the soft-C has a whole page of variables in the struct. It's everything you need for running the disk encryption in the live kernel. Whereas the metadata is just what data is actually stored on the disk. In my implementation, I only needed a subset of fields but I had to switch back to the other one in order to use the unmodified code. The other thing was the crypto and the HMAC were all in one file. So I split those into two separate files so that I could include only the bits that I needed. In particular, I didn't want the crypto bits from the kernel version of Geli because it used the kernels crypto framework. I wanted to use just the straight stuff I had, which doesn't support acceleration or multiple threads. But in the case of the bootloader, that's all I wanted to have. Then the other problem was that a couple of places, there was kernel-only stuff, so I had to add some if-def kernels to Geli so that the code would compile in userland. So I had to move a couple of things around, but it was only minor stuff. So once all that was working, I tried booting off a ZFS mirror, which resulted in me having to type in the password six times. Once for each disk in the bootstrap, and then once for each disk in the bootloader, and then once for each disk when the kernel actually starts it before mount root. I was like, that's a lot. It's like, what if I have a RAID Z of six disks? That's going to take all day. So previously, Colin Percival had done some work with Devanteschi and Chris Moore to create a system for this. So Colin originally created a password caching system in Geli so that when you decrypt the first disk, it caches the password until it's done mounting all disks, and it attempts that password on the next disk. If it's wrong, it doesn't count against your three attempts and it reprompts you. But if it's the right passphrase, you only have to type it in once for all your disks. So I was like, I will shamelessly steal that concept and applied it to the bootstrap and the bootloader. So now, when you boot with two disks or three disks, you only have to type the password in three times. But Colin had also worked out a way to pass the passphrase from the bootloader to the kernel. Because Chris Moore wanted to replace grub in PCBSD, and Devanteschi had wanted to move the prompt for your password into the bootloader because when it's done by the kernel late-arriving USB devices could overwrite the prompt with de-message stuff, and then you don't realize that the reason your machine is just sitting there and not booting is because it's waiting for you to type in the password. So I re-used that mechanism to pass the password from the loader to the kernel. So now you're down to two copies of the password being entered to boot. So then I actually looked at how ZFS has the answer for this. How do I pass data from the bootstrap to the bootloader? So ZFS have been doing this by passing the Z-pool ID number so that it knows which Z-pool you're trying to boot from. The way it does this because it has to pass a couple of bits of data is it passes a struct, and the very first member of that struct is a size, which is the size of that struct. This way, if the code is out of sync and you have say a newer bootcode but an older loader or the other way around, the loader and every time before it accesses any member of the struct, it checks if the offset of that member of the struct is greater than the size. If it is, then it knows that its definition of the struct is older, and so that anything that's going to be off the end, it won't try to read. So this way, we have forwards and backwards compatibility. If I add a new field to the struct, the loader is going to make sure that its definition of the struct is new enough, or the struct being passed by the bootstrap is big enough. So if the bootstrap is older and passes a struct that doesn't have say our new password member, then the offset of that member is going to be smaller than the size that the bootloader passed, and it won't go read uninitialized memory, thinking it's passphrase, it'll know that bootstrap is older and just didn't give us a password. So I hacked onto that and basically added a galley password field to the end of the struct that gets passed and then gets zeroed out. So I implemented the exact same thing for UFS. So now we can pass parameters and we can always grow the list of parameters and it's backwards and forwards compatible. So now you only have to type in your password in the bootstrap and it passes it to the loader and then the loader passes it to the kernel and the system boots. So that's how it works. So the things that aren't finished are that currently it only supports passwords, where you type in a password. Galley has always supported keys where you can have like a USB key or like a four kilobyte file of randomness that you can throw into the mix for your encryption. The bootloader currently doesn't support that at all. I'd like to add that but it's going to be a little complicated. Because where do you store those keys? Based on a bunch of conversations I had, I think the easiest way is actually to create a new partition type like freebsd-galley key and we can just say have a USB stick that'll have this specially defined partition. It'll be whatever size you want and it'll just contain raw key data rather than a file system or something. I don't know how well USB keys show up in the bootloader in this case and in particular what happens if you try to attach one too late. If you have to reboot to attach your key every time you want to decrypt your drive, it'll be kind of annoying. Galley itself supports blowfish and camellia ciphers for disencryption as well. But I don't think there's that much interest in them because with AES and I, using AES XTS is the only way to get no performance degradation from using encryption. So I think it's fine to not bother implementing those algorithms. Galley also supports sector authentication where it actually has an HMAC of each sector. So your sectors have a little bit of extra data. Turns out that's not used very much and especially in our case if you're using ZFS, you already have a checksum so don't bother with it. Then the big limitation currently is it doesn't support the EUFI booting. Although Eric McCorkle has a review out for a version that implements it there completely differently, but hopefully that will go through and then we can have it for everything. Then once that's done, can we actually improve my code to make it closer to his code? Then there's a couple of other things I'd like to do some day. In Galley, I'd like to consider replacing all the SHA-256 bits with SHA-512 truncated to 256 bits, because on 64-bit AMD64 processors anyway, a SHA-512 is about 50 percent faster. Now I can't make the hash longer because all the metadata only has room for 256 bits of data, but to steal a trick from ZFS, I could use a SHA-512 and just truncate it. Then there's various other cleanups that I'd like to go fix all those get-ster bugs and so on and get back to creating just one unified bootloader that can do UFS and ZFS instead of having to have two separate ones for that. Also, after playing with Galley so much, I realized how nice and elegant it is, and it might be cool to create a fuse or other implementation for it so that you could use Galley and other operating systems. Especially, half the point of ZFS is that it's a file system that's portable and you can take it to another computer and just plug it in and it works. If you can have a Galley encrypted USB stick with ZFS on it, it'd be great to be able to mount that under Linux or Lumos or something. I'd love to hear your ideas of what else to do. Also, spend the time resuming the entire scheme? I talked a little bit about that yesterday at the previous DDEV summit with Baptiste. It's like, suspend resume hasn't been great on previous DDEV. It works on my laptop but not newer ones, but suspend to disk would be great, and it can't be that hard, right? Just force a crash dump and then have the loader just load that back into memory and turn on again, right? It'll be easy. So we have all that working very nicely, take our code. Okay. I'll have to look at that. We solved all that. Does that be nice to have? Yes. Here's the advanced ZFS book that Michael and I wrote. You can get that at ZFSbook.com. I do a podcast, BSDNOW.TV every Wednesday. We interview developers about things and talk about what's changed in free BSD and open BSD and so on. That's good. That's it.


Selamat pagi. Selamat pagi. Selamat pagi. Sekarang ini akan dipakar dalam perjalanan micro-services oleh Pn. Hari Pn. Pn. Hari Pn. Pn. Hari Pn. adalah konsultan software dengan lebih daripada 10 tahun pengalaman dalam domen seperti Banking, Network Security, Telecommunication dan Retail. Dia telah bekerja di beberapa perjalanan konteks. Dia terlalu terbangsa di mana dia adalah teknologi director. Dia berminat dalam perjalanan micro-services dan perjalanan micro-services dan perjalanan Pn. Terima kasih. Terima kasih. Terima kasih kerana menerima. Terima kasih kerana berjaya. Mereka telah lakukan kerja untuk saya. Jadi saya tidak ada banyak kerja untuk membantu. Ya, ini adalah perintah saya. Selain daripada bahagian ini, bahagian yang saya cuba bekerja pada hari ini adalah Closure dan Golang. Itu sangat tentang saya. Jadi mari kita mulakan. Sebelum kita mulakan, bagaimana banyak kita di sini bekerja dalam perjalanan micro-services yang berdasarkan perjalanan micro-services bergerak ke dalam perjalanan micro-services. Itu sangat menarik. Bagus. Bagaimana banyak anda tidak dengar tentang micro-services? Baiklah. Kenapa perjalanan ini? Sebelum tahun lalu, saya telah bekerja pada beberapa perjalanan yang bergerak dari sistem monologi ke perjalanan micro-services atau membangun perjalanan micro-services. Dan salah satu perjalanan ini telah berhasil pada Python. Dan itu telah berlaku dalam perjalanan yang baik dan baik. Dan perjalanan ini adalah perjalanan yang menarik dari apa-apa perjalanan saya. Dan saya ingin berkongsi bahawa orang-orang tidak perlu melalui semua perjalanan yang kami lalui. Jadi itu tentang perjalanan. Dan... Ya. Hanya untuk mendapatkan perjalanan micro-services dalam 2-3 menit. Dengan perjalanan micro-services, tidak banyak. Ia hanya sebuah perjalanan yang lebih di-interpretasi dari perjalanan micro-services. Saya yakin banyak kami telah melakukannya. Sebenarnya, anda hanya mengambil sebuah monolith dan tanpa mempunyai segala-galanya di satu tempat, anda mengambil segala-galanya ke kecil dan itu yang berlaku. Mereka mengambil sebuah api, tanpa mempunyai sebuah monolith. Sekarang, kenapa ini terlalu spesial? Ini bukan sesuatu yang baru. Dan apabila anda mengatakan micro-services, semua orang mengatakan bahawa perjalanan micro-services seperti anda mengatakan bagaimanapun micro-services dan bagaimanapun micro-services atau... Itu adalah pilihan yang besar. Tapi kemudian, sebelum kita melakukannya, kenapa semua perjalanan micro-services berlaku? Saya rasa setiap conferensi perjalanan sehingga kemudian anda akan melihat satu-satunya perjalanan micro-services. Dan itu akan menjadi sebuah pilihan yang benar-benar untuk menggantikan sesuatu yang sudah ada di sana. Anda dengar semua ini? Semuanya kecil, mendapatkan, memperkenalkan, siapa tidak menang? Dan... ini bukan baru. Kita sudah melihat semua ini sebelum ini. Semua pilihan lain telah mempunyai sebuah cat jadi saya perlu meletakkan di sini. Jadi, sesiapa yang berlaku dengan Java J2w, architektur tak conversation seperti braid ke-message EJV Saya melihatkan masalahnya. EJV telah berkenal dengan apa yang yang kita inginkan untuk menghasilkan modularitas, terutama mdbs, braid ke-message. Anda dapat menggemaskan peta-peta di jika mereka akan digudi ke pementsyaan menggunakan weblogik atau mayan-paman app, dan mereka akan terbentuk Mereka mempunyai autoskale dan mempunyai jemes yang betul-betul membuat kerja yang baik Dan mereka mempunyai sistem reaktif yang kita dapat membangun Ia sangat bagus dan saya harus kata anda dapat mempercaya mereka Mereka sangat mengalami masalah Apa masalahnya?   Sudah tentu, kami mendengar banyak bahagian data Dan kami mendengar banyak penghantaran komputasi Dan sebagainya Python adalah sebagainya bahagian yang besar Dan juga, sebagainya bahagian web aplikasi Saya tidak ada pelajaran data atau sebagainya Saya hanya sebuah pembentangan pembentangan Saya mendengar aplikasi Saya ada beberapa pembentangan web atau sebagainya Tiada apa-apa Bagaimana pembentangan ini? Semua anda yakin dengan pembentangan global Ia tidak mempunyai anda untuk melakukannya Anda mempunyai pembentangan pembentangan Python Tapi anda tidak mempunyai kemungkinan untuk melakukannya Bagaimana anda mempunyai kemungkinan pembentangan atau sebagainya Sudah tentu anda boleh melakukannya Tapi terima kasih Bagaimana anda mempunyai kemungkinan pembentangan? Bagus Jika anda tidak mempunyai kemungkinan Ia adalah sebuah program paradigme Di mana anda melakukannya Banyak anda datang dari community Node.js Ia adalah program paradigme yang sangat bagus Bagaimana anda melakukannya? Ia adalah tempat yang sangat bagus untuk memulai API Microservices adalah semua tentang servis pembentangan Dan API Jika anda mempunyai sesuatu yang menyebabkan Bagaimana anda melakukannya? Tapi Bagaimana anda mempunyai pembentangan Python Microservices app? Saya tidak mempunyai Bagaimana anda melihat pembentangan pembentangan? Bagaimana anda mempunyai pembentangan?  Kita mempunyai pembentangan Tapi kita mencuba mempunyai pembentangan Dan pembentangan Ada sesiapa yang mempunyai pembentangan?  Jadi jika kita mempunyai semua filter ini, kita harus menghentikan data, mendapatkan semula dan kembali lagi dan menghentikan kepada keadaan keadaan keadaan keadaan keadaan keadaan keadaan keadaan keadaan. Untuk ini, Lambda itu adalah cara yang sangat baik. Dalam perlukan yang mencukupi kita pergunaan sebenarnya dan melakukan semua perangkatan untuk semua perangkatan, tapi seluruh itu, semua perangkatan yang kita masih ada di python dan Flask bukan benar-benar tempat yang baik untuk kita. Sebelum kita mulakan, saya rasa banyak kita memiliki aplikasi monologi, berapa banyak kita memiliki aplikasi web di python? Flask, jango, okey, bagus. Jadi jika kita belum berjumpa lagi, bagaimana kita melakukan perangkatan pertama kita? Ini adalah cara yang sangat murah untuk melakukan. Saya hanya memilih perangkatan paling basic kerana ini adalah apa yang saya gunakan untuk menunjukkan orang lain dan team saya bagaimana kita mulakan dan apa yang kita cuba lakukan. Kita memiliki perangkatan yang mudah dalam perangkatan. Sebenarnya, kita perlu berkata negara dan kemudian ia tidak terlalu dekat. Ia hanya metadata. Ia boleh menjadi apa-apa lagi. Ia boleh menjadi model perangkatan, mobil perangkatan dan apa-apa saja. Sebenarnya, mencukup informasi. Ini adalah aplikasi monologi. Apa yang kita lakukan? Kita hanya mengambil perangkatan di dalam perangkatan. Kita mengambilnya secara separat. Perangkatan tetap di dalam aplikasi monologi. Data masih ada dalam database yang penting. Kita tidak benar-benar membuat apa-apa tentangnya. Dengar sini, kita bekerja dan melakukannya. Kita melakukannya sekejap sekejap. Jadi, ia sangat panjang. Iterasi. Kita tidak mempunyai masa untuk melakukannya dan semuanya dan kemudian melakukannya lagi. Dan kerana kita di dalam perangkatan terus, kita perlu menunjukkan setiap hari atau setiap minggu. Jadi, ini adalah semua yang kita boleh lakukan dalam satu setiap hari. Kita hanya mengambil perangkatan separat dan melakukannya di perangkatan yang berbeza dan menggunakan perangkatan yang sama. Jadi, ini adalah satu perangkatan. Pertama perangkatan kedua, kita bergerak ke Redis. Kita bergerak ke metadata ke perangkatan separat. Jadi sekarang, mulai mempunyai perangkatan dengan diri sendiri. Ia boleh dikatakan dengan kurang-kurangnya. Dan ia boleh mempunyai perangkatan sendiri. Well, setelah itu, perangkatan itu tiba-tiba mulai berkata, okey, itu adalah perangkatan yang sangat muda. Jadi, anda boleh melakukannya dengan perangkatan. Sekarang, kita ada sesuatu yang sangat kompleksan, seperti kita perlu membangunkan sistem perangkatan. Dan ini sangat susah kerana ia mengejar banyak perangkatan dalam perangkatan. Dan kita tidak mampu membangunkan perangkatan, kan? Sekarang, orang mulai mendapatkan sedikit perangkatan yang mudah dan mereka berkata, okey, mari kita balik ke perangkatan perangkatan. Ia sangat susah untuk kita menjelaskan perangkatan dan membangunkan itu sebagai perangkatan baru. Jadi lagi, apa yang kita lanjutkan adalah, instead of building it inside the monolithic application, we started building the new features and services separately and now the view layer started separating out slowly. So essentially, this is a strategy called strangling. If anyone has read this old book called Enterprise Application Patterns, there is a pattern called strangling and what you do is instead of going and rewriting an entire application, you keep the existing beast alive but you start taking nibbles out of it. You start building stuff around it and then the center core becomes a useless dummy piece of code and then you can remove it. So this is the strategy we started following and then we started pulling out services more and more into from the monolithic application and as we went on, we had different storages. Most of the cases we had the master DB as the integration point because we could not afford a message bus at this point. I'll get to it but right now the integration point is still the database. Not a great idea but a good place to start. It's okay. We've all been there. We can integrate through database. When you talk about a typical microservice, even that simple lookup table, it's just got redis and it's got a service in front of it. All it does is get request. You just ask for it and it just gives you data. Even that simple service requires like it's got its API, it's got its logic, data, then you have dependencies. Like for example, even though it's just serving countries and states, sometimes it needs like certain information from other systems which need to come in. So that dependent services are always problem and then there is automation around it.  Whatever it is, the automation itself is not easy. And then there is admin. When you have a monolithic application, you have the database. Some of us are okay with just going into the database and making modifications but in case of enterprise and especially audited enterprise applications, you have to have admin and the admin has to like you know, the admin interface should be audited enough that you know who did what. Now, when you have separate services, what happens to the admin interface which was on the monolithic application? We started building individual admins for individual services. Now this was becoming a pain and then there's monitoring. Sure, you can monitor, you have like data dog and you have like all that cool developed of these days. You could just you know, put Neuralink on all the boxes and then have one single dashboard. Sure, you'll have like a NASA like sure, you have a huge control center from where you can look at the entire architecture of your application. It sounds easy but in reality when you have like so many different pieces running amok and you have like admin and monitoring setup, it's easy to you know fail. For example, the service went on for two hours and we did not even know that was happening. The main application was still running but unless until the future, I mean, user started reporting issues we do not know. So that's quite a lot and it actually takes a lot of discipline to get there. So there is if you read up a little bit about microservices, it's called a post agile movement. Can you say post agile? Okay, getting to agile itself is subjectively hard thing. For example, would you consider yourself an agile software development company? That question itself cannot be answered very easily. Like what is agile that goes into a big discussion? First of all, you got to have iterative development. You got to have sprints. Then you have continuous integration and delivery. If you have one monolith and that goes through a CI system, goes through staging and goes to production. That is a good setup. Now that is an absolute necessity when you're going into microservices because now you need pipelining. You need multiple of these applications getting built, integrated and then deployed. It's not just code getting integrated. You need services which are getting integrated and then deployed. Then you have something called release chain. Release chain is basically if you have service A version 1, service B version 2 which are dependent. Then when you draw release chain, essentially it should when you pull that chain, you have to get the entire release out which has all the dependencies. That's hard and by the time you get to that stage it's going to be quite daunting. Now, why am I saying all this? I just built a really small microservice for a lookup table and now I'm just presenting all this information to you which makes it sound like we should not be doing microservices. Well, the reality is you might never get to an ideal state. This is the case. Probably a Netflix doing it so long as it has been doing. It has matured and whatever it has built, the tools which it has built is very much suited to Netflix itself. And when I read up Netflix literature to understand how I can apply some of the concepts into our applications, some of them don't really fit and microservices very, very specific to how you're running an organization. It's very much possible that you might have one monolithic piece and then several microservices around it and it's perfectly okay to be there. But still for me the motivation to slice an application into services into smaller pieces should come from like something concrete. For example, technical. So in our case, this lookup table was taking a lot of the database queries. Like for example, every single time the user goes up and fires up the home screen, the first thing they do is do lookup. And when they lookup, it's all cached, sure. But then every time there's caching validation, it hits the database. And every time we have to modify the database, we have to invalidate all the caches.   So we moved that aside. That also ensured that we need not have to buy additional infrastructure because the main application could sufficiently run on whatever you know, instances it had on our AWS. So that is one. Increasing resilience as I said is pretty much this, right? If I have to just change, okay, new state was added in India and I had to go update it, I don't have to go modify the original application. That itself gives me resilience that as long as I don't touch it, it's probably going to stay stable. So it gives me a lot more resilience to push it out. So the motivation here and my justification to my boss in terms of like, you know, okay, I have to build this thing. I need a week's time. So how do I buy that? Obviously, I can only the only thing I can justify is, okay, here's your JIRA tickets. Here are all the bugs that came up because of this piece being inside and this is the time I spent on it. So here's the money that you could save by pushing this out. Essentially putting a cost benefit aspect to it. Even though it's a purely technical topic, the reason I'm bringing this in is because you have to have some sort of a motivation in order to do microservices. Doing microservices just for the sake of it. Probably doesn't make sense. The last bit is external versus internal facing APIs. This we did mostly from the angle of security. Some of the APIs that we were using internally also happened to be useful to, you know, be to be clients of ours. So we had to expose them.  But then do you want to have public IP addresses for most of your services? And when you expose one service to the external world, do you think the internal services should also communicate over that same IP? Probably not. What you want to be doing is you put a layer in front of it and create another service which is purely exposed for the external world. And you keep the internal world secure inside. Am I making sense? So this is from a technical angle. These are few things which we considered when we sliced our application of. From the business angle, when we started slicing this, business started coming up and telling to us, okay, now that we are aware that you can split this big application into smaller pieces, can we carve out products by themselves? For example, we had a region search, search nearby and very specific to the business functionality which I cannot reveal. Can we carve this out as an application and put a metered API on top of it and generate revenue from this side project? Sure, pull that out, keep it aside, which means now I have a service which I can internally also consume as a client and I can also make money out of it. And there are pieces which are like the non-performing assets in your application. You never know the kind of money you're sinking into it. For example, there was an RDS, a huge RDS instance which we had on AWS for this particular feature and that was sinking a lot of money in. Now our business wanted to understand why are we spending so much money on this arcane feature. So we had to split it out and now we really started controlling and metering the API and figuring out how much money are we putting in and how much money are we getting out. So this is again a very useful way of doing this. I mean in the sense that again a motivation to split your monolith into a separate application. Right? The last two bits are very, very simple, AB testing. So anyone heard of blue green deployments here? Okay. And canary deployments of course. So when you have blue green and canary, okay, just to back up and for the rest of us, blue green deployment is when you have an existing feature or an existing application and you have full feature parity, you deploy the new application and cut over in one go. You pretty much shift all your users to the new system and essentially that's one go. Right? And then you have canary deployment where you slowly move people from the older deployment to the newer deployment. Right?  It's a little safer because you don't transfer the entire load at one go and you just over a period of time test the resilience of the system as well and then you transfer over. When you have one monolith, you have only a choice. You either do blue green or you either do canary deployments. Right? Now with microservices we could choose like you know for certain services we could do blue green. For certain services we could do canary. And it was a choice we could leave it up to individual teams. Instead of it always going up to like CTO or director of engineering level decision. Individual team leads could make that decision. Build versus buy. So for example one of the services that we were using is the Google Maps API and we were pretty much taking a lot of money sinking a lot of money into it because we were doing this routing algorithms like we would find distances and we had put an obstruction API on top of it. So we would get you know information from Google and then send it over to the rest of the application. We wanted to cache some of this and we were okay with inaccuracy and serve that information. Now when you are building your internal routing algorithms or you are big enough that now the Google APIs are expensive and more expensive than you can handle and you want to build that routing algorithm yourself you might want to build that as a service put a load balancer in front of it and start figuring out which is working better. Then you can do again investment return on investment calculations and then switch over. The last one is Organizational. Now this is probably the easiest. I think there was a team which was sitting in US there was a team which was sitting in UK and we were sitting in India and we had to write code what naturally happens is each team picked up an individual piece and started writing code and then we would integrate at the end of the day or at the end of the week. This is a fancy way of putting it as converse law. So to read the law it's very simple Organizational which design systems are constrained to produce designs which are copies of the communication structures of these organizations. An example I gave is geography but then even if you have more than seven people or eight people it's going to be hard to communicate among yourself and automatically there'll be silos which are formed. Anyone who's not read The Scrum Book I definitely recommend reading this book called Scrum Getting Twice the Work Done in Half the Time. This is an excellent book and it talks about team communication issues. Again, why am I talking about all this in microservices? It is also a team dynamics aspect and it is extremely strongly dictated by how your team is structured. We'll talk more about this but in our case I think what we figured out is each team having up to four to five people and then splitting up into separate teams is better than having large teams. Initially we started out being like eight or nine people all working on all of the services but then this is not scalable. I'll get to the problems in a bit. So microservices architecture in short is nothing but distributed systems, right? Anything that you find in distributed systems as a problem you're going to find it here. Split brain problems or two-phase comments. Anything in your computer science classes which you have heard about for distributed architectures. It's very much possible in microservices. First one, identity. Sure, we've built all the services now. But when you're communicating between the services what do you do? Not all actions are directly going to be hit to the service from the user. For example, as a user I'll be hitting the view. I'm saying I need to book a ticket to a movie and that goes to a booking service. Now as a result there are other services which are hit. I have just taken a sample which is payment and notification. I am redirected to the payment gateway and something gets cut from my wallet and it comes back to the main system. And then there's notification service which has to send me the SMS with the ticket information. Now all of these are independent systems and think about all the details that you have to send across the systems. When I send the book ticket information I'm sending my user details to the booking service. The booking service is aware of it. But does payment and notification know who are the people who are actually making this changes? And when you go audit the notification service and you figure out that an SMS went out to some number that has prohibited then do you know that this action happened because of someone booking a ticket or someone going and hacking up the API and sending an SMS out from the system? This could very well happen, right? If you have internal teams which are extremely huge and you cannot prevent frauds this is very much possible. Someone could just hit the API get some token back and get the money in their account or hit an API to send bulk SMSes outside. So how do you control that? You have to have identity information propagated across services. If you have a monolith you probably are following an OAuth or something like that, right? You have a resource owner and you have a token service and you just do your authentication and authorization and you're done pretty much. But when you have such a large system how do you pass this token around? How do you pass your OAuth token around to other people and each service now has to keep hitting the resource owner and asking permission can I go ahead? Can I go ahead? And this is actually a on the one hand it seems secure. Okay let's we started out with this we actually started out with okay let's do OAuth and each service has to independently ask authentication permissions and then validate the token and then progress. But it's not a very scalable idea if you have one action which hits more than 25, 30 services it's not a very good thing to do. So ideally what you want to do is have something called an open ID connect mechanism. This is what we're trying right now essentially it's a delegation mechanism where you do the ticket and you get it verified in one place pass the information around and then record it. So when you go individually to each service and audit the service each service will tell you why this action happened otherwise you wouldn't know who did it. Am I making sense here? Any questions so far? There's more to it I think I can if at all we have more time after this I'll get back to it otherwise we can catch up after the talk. There's something called service traceability. It's an extension to the identity problem that I'm speaking about but what is service traceability? Every action that happens in the system should be associated with the primary use case. For example in this case I'm booking a ticket for that ticket I have a booking reference number and every action in the system should be happening because of that. Now if that information is not propagated and if I simply call APIs then you don't know how to trace the calls. Why is this important? When you look at this it's very clear it's an orchestration flow you have several systems doing different parts of it that is booking this fulfillment then there's payment and then there's notifications. Now if you have orchestrations like these if you were to build a dashboard if you're admin someone calls up I have booked my ticket but I have not got my SMS so how do I go to the theater and show the box office that I have booked my ticket indeed? What does your call center person do? They need to have a dashboard level in a picture of this entire thing they should be able to say okay you booked it but I'm so sorry that your phone number was incorrect so which is why the notification field I'm going to send it to your email for that purpose you need to have service traceability you should be able to say with service where what happened so that you can dashboard on top of it Reconciliation So when you have distributor transactions and especially when you have microservices you don't want to disappoint your users for simple things for example I'm booking the ticket the notification fails it does not mean that my ticket flow should stop I would be disappointed if you have like 30 services on one of the service fails for which you would rollback my entire process of booking the ticket that would not be a good experience and even if you wanted to do it it's not easy because if you started committing stuff onto each system remember now we are on a microservices architecture so even your data is federated your data split across the system so each system would have committed its own changes now if you go ahead and say okay the last part failed rollback everything else that's not easy so usually we have to follow an eventually consistent model in the sense that we agree that things can go wrong and then let it go wrong end of the day go reconcile or end of the hour go reconcile what's the easiest way to reconcile cues every single message that goes from you know service to service we usually push it through cues as much as possible anything that is possible to make asynchronous make it asynchronous and operate on callback mechanism only request response cycles are usually view based we don't do request response cycles anywhere else everything else is cues why is this good because cues automatically have a dead letter infrastructure if something fails it's gonna say for example we started using piris and later sidekick on the rails part of the application it would have a dashboard which says what all failed we just go retry right and then later we moved to Kafka so when we moved to Kafka we had an entire messaging bus across the system this is going back to the old diagram that I showed in the beginning of the presentation right you have all the services they all publish and subscribe messages from the central messaging bus and the central messaging bus has a dashboard on top of it so the end of the day when you look at it it's very simple to say what failed and what needs to be retry if something fails again you know which service is failing you can go fix it independently sounds simple but when you get to it and when you have to set up the messaging bus infrastructure and the dashboard on top of it it's quite some work but you can get there but you can start with having cues independently between services okay take a side how easy or hard is it to manage a project which has microservices architecture it's actually very weird my product manager went crazy after we started splitting this up okay so how many of you follow sprints here like sprint cycles or scrum cycles good so usually if you have monolithic application it's all good you have a single sprint board you put your stuff there and then cards move along nicely and then you're done and then you're deployed what do we do here for a simple feature to book ticket i have cards across all three teams payment service booking service and notification service and if you notice the card for feature one is pretty much in different statuses in each of them how do i even account for this in a sprint usually when we account for sprints internally we account for it after the business value is delivered even demo on staging is not considered done we actually push it out to production not only if it's in production we call it done so when we have such kind of discipline if someone cannot say i just push payment service out so the business value is delivered what was delivered nothing you just have an ability to pay pay money but who's going to pay the money there is no booking there is no notification so how do you pull this change across so the way we are solving it right now i don't know if this is the perfect way to do it maybe if you have suggestions you can let me know we have masterboards we're using jira right now so we have masterboards where we have epics which define the feature and then we have individual projects where stories point to those epics and then we are tracing this it's not easy it's been little difficult but why am i bringing this up once you have microservices architecture it's not easy to plan features as you were probably doing it when you had monoliths because every time you do this you have different teams you have to figure out you have to figure out the contracts between each team and then deploy right so service i mean release cycles are going to be a little longer to begin with and later when you get into the rhythm of it you will figure out the best approach for your company because i can't give us a little bit talking about api because the communication between the teams is all about api so we had you know anybody who's looking at that kind of a sprint board is going to say why are all the stories not in progress at the same time everyone wanted these stories to be in progress right so the first thing which came to mind is okay if you guys agreed on the api to begin with and then if you started work you all could finish in parallel and then we'd integrate at the end isn't this going back to the old days when you don't have continuous integration this is again the same problem reinvented right so we anyway tried we did not want to say no straight away so we tried but then what did not work is when you decide on the api very much up front you miss out details and when you miss out details there is rework and that makes it even more longer so when you had to usually solve a crime they say follow the money right so when I when I when you design microservices architecture I would say follow the data interface to implementation when I say interface it could be UI it could be api it could be any end point from which an operation is triggered if it's going across multiple services follow the data figure out the data structures that have to be passed on in between the systems figure out all the non-functional aspects what are the SLA's of return times what is can it be a a callback mechanism can it be you know a request response cycle figure everything out and only then go into setting the api is on stone this may mean that you may have to develop some parts of the application before you can develop the others it cannot be parallelized that's the reality and every time you write a new api call usually the normal tendency we started getting into is can we can we push this api into this service does it belong in booking service does it belong in you know payment service it's it's a very easy mindset right when you have categories when you have a new functionality just feel like suffering it into something that exists but it's it's really really important at that point every new functionality that comes in is it it does it belong in a different service or should we write a new one altogether right it's hard but I think we're getting there slowly apis we already discussed there are different types and we really have to be very conscious about what we're trying to achieve do we want synchronous call backs as much as possible avoid synchronous synchronous have problems if you can do it asynchronously better okay last but I think we are running short of time in terms of code when you had a monolith okay before I go into this again I strongly recommend anyone who's interested in microservices to read this book by Sam Newman about microservices excellent book it actually gave a lot of guidance for us when we worked on it and the other book about microservices is from Jonas Bono that's about reactive microservices again an excellent architecture book and some of these things which I'm talking about were inspired by that but I'm just making points which did not work for us from there so one of the points which comes from that book is about how do you maintain your repository when you have a monolith you have a single repository you check into it your continuous integration system fix it up it goes to staging it goes to production simple now when you have microservices to begin with what we did is we had all the services within the same repository it would still build all over every single service you check in the entire repository builds and goes to production but that's okay that's a starting point but eventually you will have individual repositories for each service that's where you really want to be because that makes sense you have to have a repository per service why that makes sense is because of the next the way teams can be organized for code is usually around open source i think the best idea for a service is for each service to have a service owner when i say service owner if i am leading a service technically i should pretty much be so protective about it that any code comes in i should pretty much reject the pr the pull request why because i would try to keep the service as small as possible that's my objective what is a feature owner feature owner doesn't care about services he doesn't think services he thinks feature this is what i need to get out to the you know audience so that's what i'm doing and then next comes the business owner right the business owner is more important because here comes the money like why are you even doing this and how how do you justify when you have these three roles acting on top of a service you usually have that good pull and push and which generates dialogue around every action that you're doing so i strongly recommend eternal open source because you could then submit a pull request and then if the team agrees great but otherwise sure i'm running short of time here so i'll just stop but this i think we can talk more later so that's thanks a lot and yeah i hope it was okay so yeah questions i think i stopped i did not talk about a lot of topics especially around the protocols for api we started using jrpc for some time then we used trust based protocols then even plain old tcp we came down to that level because it makes sense in some cases but there are more topics so any questions how do you insoconsistency between services between services yeah uh but how is anything confirm to what is sure things like that how do you ensure that so uh we pretty much came to the conclusion that it's not going to be consistent we followed an eventually consistent model so in some cases data will not be propagated we designed a system for failure in the sense that we assume that certain services will not be up at certain points in time we also had legacy services which were buggy so which would always written 500 for no reason right we had to retry all of those the agile book is scrum the art of doing the microservices is microservices by Sam Newman the other one is by Jonas Boner i can share that with you later anything else the easiest answer to that question is it depends in my opinion i would strongly resist the urge to go towards a microservices approach i think monoliths are okay after having built one or two microservices applications i personally feel unless there is a strong business traction for your application that you actually need to scale and you know that your business is going to survive the next six months because working with startups the problem i always have is i don't even know if the business is going to exist in the next month so if i invest a lot of effort into doing a microservices architecture it might well be that the next month the business folds doesn't make sense usually i resist the urge to move towards unless i have like a clear road map for the next six months in terms of traction and like i usually document non-functional requirements in terms of scaling then work it backwards like how much cost does it take to run a monolith versus what cost savings do i have to split stuff so any other questions i'm going to be around so catch me if you have any other questions also i'd i'd love to hear your feedback or you know any other suggestions you might have for how to build microservices so thanks again thanks a lot


Today, I feel like it's a nice calm, relaxing day and I feel French Quarter is definitely the most quietest Disney World resort. So I kind of planned out this thing where I wanted to come, hang out, sit in the massage chairs, get some jambalaya, some gumbo, maybe some beignets and just walk along. Where are you going? Yeah, I'm pretty hungry. So let's go do this. And first thing on the agenda, my favorite ride at Disney World, the massage chairs. I just love them. I love them. I love them. I love them. Well, now that that massage business is taken care of, let's take a look around at the resort a little bit, eh? So peaceful, so quiet here, literally. You can hear crickets, but there's no crickets out right now, but I'm sure you can hear crickets when they do come out. I don't hear crickets. No, it's a... They don't purr? Crickets? Crickets don't purr? It's just so peaceful, the fact that you can just stand here and look at the fountain and actually hear the water and the drains gurgling. Is this your story? Yeah, it's amazing. Relaxing I say. And look at the way I'm talking. I'm talking in a relaxing, calming voice. Making my way down Jazz Boulevard. I guess that didn't go well. But isn't it beautiful? Look at the facade of the rooms. Is that the proper word to use? Facade.  The face? I don't know. Actually what I call it. I don't usually say it. Well, we're going to head down by the water because it's actually pretty down there. Then we're going to make a roundabout back up to the eatery area because I got to get some gumbo. Gumbo. Yeah. Or some gumbo. Gumbo. Gumbo. Also the horse buggy straps right here because you know the carriage rides go right through here. I knew I hear crickets. Oh, okay. I hear them. That might be cicadas though. Oh, they might be cicadas. This is carriage path which actually is named after the path of carriages. Okay. Yes. Down by the bay here. Where the watermelons grow. You can always see the boat that takes you from Port Orleans to French Quarter to Disney Springs. Or if I do. Your mother will say. Have you ever seen a cat wearing horse mats? Horse mats. Yeah. Down by the bay. One of the cool things I want to point out is the Benye Dash Fun Run. Every Sunday morning at 8 a.m. they do a race around the resort and at the end of it, you know, all that hard work, you get to reward yourself with a tasty Benye. Oh darn it. It's on Sundays. I can't do it. Yep. Can't do it. It's not on our schedule. $15 plus tax. But that's pretty good.  I'm really looking forward to that. Oh, maybe next time. Okay. We were going to take you in and show you a little bit of the pool area because it's really nice. It's really nice and themed, but Veronica will tell you why we can't do that. And I mean, I'll tell you it's closed, but Veronica wants to explain why it's closed. It was a dark and stormy night. It was a long time ago, longer now than it seems in a place perhaps you've seen in your dreams. So the story that you are about to be told took place in the holiday world of all. The pool is closed because of the storm coming. Yep. They do have an outdoor bar area, the Mardi Gros, which is open even though the pool is closed. And they got their, you know, cocktails and beers. I'm going to see if they have a menu we can take a peek at, eh? Look at that. This is like one of the most well-themed, like, lounge menus I've ever seen. Like, mostly it's like the same ones all over the place, like the little binder, but I feel like they have different drinks here. Huh. Here we are right outside. Where are we? Gator Alley. Gator Alley. This is as soon as you leave that lounge, and that lounge seems like a real cool chill place to actually hang out.  Look at them. They look thrilled. Excuse me, Mr. Gator. Do you know what horse mats are? Still trying to figure that one out. He knows. He knows. Oh, I bet you he knows. Look at that. Mm-hmm. He won't tell you though. Mm-hmm. Well, I'm getting a little hungry. So, we're going to go in the sasagula. See what nice New Orleans style food they have. I love jambalaya, gumbo. I like spicy food, but not too spicy. I like it mild spicy. Does that make sense? No. Huh. Man, this place is empty. Well, that's because it's like 2 o'clock. I think everyone's at the parks right now.  And then a cheese steak po-boy. And then over here is the Benye shop. On this side, pizza and pasta. Shrimping grits as well. But where I'm going, that's where the jambalaya will be. Ooh, jambalaya and gumbo. I'm going to try them both. I'm going both. I'm going both. I'm going brave. Jambalaya and a couple gumbo. I can tell you what. I didn't even take a bite of this yet. But I already know this is going to be the best Disney meal I've ever purchased for $14. You can't go wrong. All together. Huge bowl of gumbo with rice. Jambalaya. Look at the cornbread. $14 for all of this. That's an amazing deal. I'm so excited. Yip is not a big fan of jambalaya or gumbo. I am. I wish I was. In fact, jambalaya is up there in like my top three foods. Meatball's in there too. I'm a meatball for three years. I like macaroni and cheese, grilled cheese. I like cheese and nuggets. Yeah, your basic kids meal menu items. Yeah. So I'm going to dive in. And of course if we're eating, you know what that means. I'm going gumbo first. No, that's not what they know. Oh. I love it. The spice index on the gumbo is a little higher than the jambalaya. I was playing Tiana. Oh, is this Tiana's gumbo? Yeah. So the gumbo I believe was $4. Yeah, $4.39. And I gotta tell you, this is filling. Is it? Are you filling up? Well, I'm like, this is a big helping for $4.39. Oh. Like really is. I mean, I'm definitely going to get the jambalaya, but I don't know if I'm going to finish it. I'm sure I'm going to force myself to finish it. Oh, took a nibble of your cornbread. Hey, your cornbread sneaker you? So I decided I want to put my cornbread in the gumbo and see what it tastes like. I'll put it right there.  Very good idea. Thank you. We're on to the second dish. Well, he took the rest of the gumbo and put it on the gumbo-laya. Because the rice was a little dry. Oh wait, so now is it really gumbo-laya? Like I call it? Mind blown. It's gumbo-laya, gumbo-laya. Oh. Okay, anyway, Nate's going to take a bite right now. No? Oh, yes. So good. It was $9.99. It was served with that gigantic cornbread slice. And it was too big. I couldn't eat it all. All right, so I did eat it all. But it was delicious. And I enjoyed it a lot. It's probably one of my favorite meals, actually. And I think because now I am literally too full because I ate the gumbo and the gumbo-laya, there's no way I'm getting Benye's. I couldn't do it. I couldn't fit those down. Nate's back. What's in the bag? What did you just say on film? I wasn't going to do it. Again. He has lied to you guys about his capacity that he has. Powdered sugar mess? No, don't eat it like that. Pick it up and bite the ear. Bite the ear? Yeah. So you actually get three Benye's for $4.39 or six for $6.39. Oh my goodness. I was definitely told not to eat like this. You're such a mess. No, she just threw that powder sugar on me. No, I did not. No, I did not. Yes, you did. Look at me now. It's on your arm. It's on your, it's on your lips.  He literally said I don't want anymore. I'll take this one home. Stop calling me out. How to know if your husband snuck away and ate Benye's or a funnel cake. It kind of remind me of the same thing. In the bag. Bounce it around. So I get more sugar on it. Just so that you could get more sugar on it. Well, now that lunch or dinner is over or whatever you want to call it. I want to show you some of the cool artwork before we head out inside of the cafeteria area over here. I like how they have the Mardi Gras beads as lights up top and these random floating umbrellas. I don't know what the significance of that but it's pretty cool. So I don't know what the meaning of these pictures are. I'm sure it has something to do with New Orleans art style. It has some significant meaning but I like them a lot. Like this old man Gator right here. Like they're kind of cool. Or they have these ones here. Look at this goat man. He's got a pint. And this one's my favorite. It's like a bee old man Grim Reaper. Oh Veronica probably likes that cat baby. Well that's it for here at the French Quarter. Now that I'm covered in powdered sugar and I got a full stomach. We got to take a nap before I go to work. And Veronica's going to show off some of the mail that we got recently. Oh yeah, I got a lot of mail. Yeah, so let's head home. Okay, we are back home now. We're going to open up some mail. This first one was a card. It says ouch. And then it says thanks for taking one for the team Nate. Never drop the camera and never miss the monorail shot. Hope you're back on your feet soon. Happy birthday yep, it's 5.54 p.m. Nate, you guys are awesome. Hugs, Darlene and Henry Lutz from PA. And then they sent us this because they're from Hanover, Pennsylvania. We are missing the fall season by the way. We are very much. So thank you guys so much. You guys have been watching for a long time. We remember. Next I had a package and it was from Des. It came with this magnet bottle holder with Gracie on it. And I love my donation. This awesome car coaster which is great because these work really well. And I used to have one in my other car but the car got crashed so I didn't take it with me. So I'm happy to have that one and it's perfect. Then it came with a Dalmatian pen. You can focus. Maybe not. Look at how cute. And then it came with it. It was an elephant cookie for Gracie. Not for you. I know. You're jealous. I know. Oh great. I have a surprise. I think. I think. I think we like it. Do we like it? Not sure. She says thank you anyway. She really. Oh wait. She does like it. Okay good. Thank you. Thank you. Thank you. Thank you Des. And then I got another postcard. It's got all the characters on it. And it says we would love to buy you a drink if we see you when we're back home in August. But in case we don't see you here are these look at these. These are so funny and cool. And yeah, they light up. What? These are so awesome. And they're Bud Light themed. Like that's so awesome and perfect. Read them. Read them wants to wear them. She's being oh that's cute. These are from Julie. She actually has an Etsy shop. It's Ears by Julie. I will put that in the description. These are awesome. Thank you so much Julie, Steven and Maria. Thank you very, very much. These are awesome. And then from Amanda and cousin Fred. Live the life you love. This is perfect because I think my theme I go for down here in the living room. Down here in the living room. What am I talking about? Is like a gold. Like I have a gold coral gold clock. Gold dreams. Gold, gold. I love gold. The next thing that I got are these 101 Dalmatian salt and pepper shakers. Which I have never seen these before anywhere in my life. These are amazing. It came with a postcard. With the name of who got them for us. But somebody ate the postcard. Did you eat the postcard? Did you eat the postcard? Oh, you did eat the postcard. It wasn't your sister.  So honestly, I don't know who sent these and it's very sad to me. I think I have the half of the postcard sitting here. But these are so cute and I really wish I know who sent them. If you did send those salt and pepper shakers, please let me know. So I can thank you the right way. Because my little stinker here ate. I think there was two postcards in that package. She ate them both. Both of them. She numbed those right up. Next, this is what I got last. This is from Nancy and Eric. This is amazing. I have to move these things out of the way. Just to show you guys this. This is the blueprint of the outside of the haunted mansion. Framed. Like, if you didn't know, I love the haunted mansion more than anything in the world. So I've been thinking about when this is over, this vlog, where can I put it? Like, would it look good here? I love it so much. I don't even know where to put it. It has to be displayed proudly. Maybe I was thinking also taking this thing down and just putting, like, everything you guys send us on this wall here. I think that's a pretty cool idea. Also in that package, there's a hat box ghost bottle opener. And it's a magnet. And then there was a gift card to enjoy a magical star cocktail and some Bud Lights. And, yeah, that's gone. This is... We enjoyed it, though, so thank you very much. This is Ice Cube Maker, an Ice Cube Maker, which is really cool because this is the bat that I have up here. So it matches my bat. I thought that was really neat. Another thing that came in the mail for me is my Negro Orlando shirt. Your favorite vloggers, favorite vlogger. I thought that was pretty cool. I still want to know who done it. Who ate the postcard? Was it you? That was a great trip to the post office. You guys go above and beyond. It's like, thank you so much. I appreciate everything. Sorry, I put on my slippers. I shouldn't put on my slippers while I'm... This is why I don't do the vlogging that often because I'm just all over the place. I appreciate every single thing. I even appreciate the letters, the postcards, everything. You guys, I cannot thank you enough. I'm really sorry about the donation. I don't even know how she got hold of that postcard, but she did. And I don't know what to say. She's really a stinker. I also pardon that cat butt. I also want to say a huge happy birthday to Amelia. She is turning four years old. I also want to say a huge happy birthday to Holly and Millie. Their birthdays were this month too. So happy birthday, you guys. Okay, I'm going to end the vlog here. So again, happy birthday to you guys. And thank you to everyone who sent us anything. Nate is sleeping. She's sleeping up here on the counter. She comes and Gracie wants me to play with her toys with her. And I just want to say love the life you live and live the life you love. And then I would say that if it's your first time watching Subscribe to our channel thing, I guess I'll say it. If it's your first time watching, subscribe to our channel. You might like it. So toot-a-loo. Live the life you live. Live the life you love. Subscribe to our channel. Are you sleepy? We just got back from the dog park and all of Gracie's friends were there. We would go for a walk and we had all these friends. Oh, here comes Rita. Can't take away from her any attention from Rita.