Fyrtu y blaen, ond yna'r teimlo i risio eu cyfnodr, yn gallun i'w cwestiwn. Mae bydd sicrhau y gallwn anodd ar exertio i'r hyn. Mae fawr yn oni'r teimlo i'w clwlethau. Fe oeddwn i'w cwestiynau bod yn gallu hanesol gyrfaen na gennym. Mae wedi'i cyffredinol i'r teimlo i amdannu cyflwydd. a a i'n hyffordd ar ôl i'r ymu hefyd yn oedd y dyma yn y perthyn. I thathaeth angen. Yna yna hefyd yn gallu ein bod ni'n bisbl yma o'r cyffredinol i'w têl hwnnw. Roedd yw ni'n gweithio i'r perthyn, yw yma wedi bod yn yn fawr yn gweithio i'ch neud. Roedd eば, yn gallu yma o bwyl hwnnw bwyl bwyl Aberaidd, Cucumbo, gan y rhaod? Yn ysbyt. Pa, rwy'n gweith. Rhaeddai yna wedi'i ddud! The extension of TDD, TDD lead-those up nicely onto TDD test driven development. Some people don't have a full understanding of what that means. I suspect most of the people in the room do but specifically here we're talking about not just covering your code with tests, but actually writing the tests before you write the code. a'r prysgau yn ymhelydd yng ngyraed. Ymweld yn ymdweithio'r test, yn ddychud ymddangod, sy'n ddigon o'r cymdeit o'r cod, a'r prysgau yn ymdweithio'r cyfweld. Dydyn ni'n cael ei ddechrau'r ddweud sy'n gweithio ymdweithio'r hyd yn ymddangos. A ddweud o'r llambol. Ymdweithio'n ddwy'r ddwy'r testau sy'n ddwy'r cyfweld ymdweithio. Oni ddweud o'i ddweithio'r cyfweld, mae'n ddweud o sylwg o'r perthynatau ar gyfer cyhoeddiadau yn ddeithas, ond yw a'r cyhoeddiadau yn cael ei wneud am y prosiectau yn cymryd i'r ysgawr, a'r prosiectau o'r prosiectau yn gallu prysail. A'r prosiectau yn cael ei bod yn ddylch yn cael eu llif, a'r prosiectau yn eu ddweud. Dwi'n ddim yn cael eu prosiectau, a'r prosiectau yn cael eu ddweud, Sometimes, very occasionally, the project runs over budget and what's the first thing to get pushed off at the end. It's the testing and it's the documentation. So quite often what happens is if you don't write the tests first, they don't get written at all, particularly with time pressures and things like that around a project. So it pays to write those tests first and that becomes a discipline. Once you've actually started doing that, it can be, it certainly was for me, a real kind of life-changing experience with your work. I've got much more confidence in the code that I write now, particularly changing major components, major areas of the code. There's a certain safety net there. Why isn't the TDD enough on its own? Why do we have this extra step? There's one, there were two things in that final section of project development, the testing and the documentation. Particularly if you're taking an agile approach to your development, by that I mean you're not doing a detailed upfront spec. You're discussing the spec in detail for each feature as it comes into the work log. Documentation can often get missed out of that process. Particularly documentation from the customer standpoint. Even if it does happen, if you're writing code and it's changing a lot, there's nothing stopping that documentation drifting from what the actual code's doing. So quite often the docs end up not really being an accurate representation of what you've built. So, I'm sorry that's drifting off the edge. We do have a certain level when you do TDD. There is some documentation there in your test code. I can look at this and I can understand and perhaps even people who aren't Ruby and Rails developers could probably in a couple of minutes understand what's happening there in that test. How many people think that all of their customers would understand that? Nobody that I can see. So, I've got a couple of customers who could, but most of them you're not going to be able to show them this and say, well, this is what we discussed and, you know, reams and reams pages of this kind of code. You can't interact with the customer who's the expert in their field of the product that you're trying to build. That's when BDD comes in. It's applying the same kind of principles, but instead of writing a testing code, you're writing the expected behaviour in your natural native language. That's something that you and your customer probably have in common and something that you can actually discuss. You write that behaviour, it fails because you haven't implemented that behaviour yet and then you write enough code to make it pass and then you repeat that process the same as you do with TDD. Now, here's an example of a part of a BDD feature. Probably most of the customers that you deal with are going to be able to understand this. I'll just leave it up there for a minute for you to look at, you know, if the customer can't understand that, you might need to walk away from the job at that point. It's kind of a bridging point between you and your customer into their domain, your domain into their domain and a common point of understanding for you. The actual implementation of this, generally each feature which might be represented by say a story card or something that's really got a single line description of that particular feature. So in this case I want to manage articles on a website. So that's all you've written down when you've actually been doing the overall plan, manage articles and you're on to the next thing. You implement a set of scenarios in one file per feature, generally. Usually those files have this front matter. This is not executed or evaluated in any way. It's a kind of formal standard structure to lay out. It's useful as an argument point if the feature that you're about to build for the customer has no value because what you're essentially doing is asking your customer, well, what's the, you know, in order to do what are we about to implement this feature? And if the customer can't answer that question, you know, why are we about to build this? Then potentially, you know, you can say to them, well, you might need to think about whether you really do need that or maybe that's something you think you need if you can't describe a business value for it. So after that front matter usually there's a set of scenarios. Now these are actually the executed code, essentially, of these tests. They usually follow this format given something, some precondition. When something I do something or some interaction happens, then we get an expected result. So the outside-in part of the title of this talk is really, I'm going to show now how you'd implement a very small part of a Rails application. Just starting with this, an empty Rails app, which is what I've built to do this presentation, and this one feature file. And I have this one scenario here where this is what we've agreed with the customer is going to happen. Obviously they'll always be more complicated than this, but you should get the basic concepts here. So I write this feature file in plain text, and I use the rake tool, which anybody who's worked with Rails is going to be familiar with, to run this, how's that read? That's good, that's visible, to run this. And immediately I've got a failure. This is using Cucumber. There are other tools available to do BDD. I think certainly in the Ruby world, Cucumber is the most popular. And you'll see other talks doing all kinds of crazy stuff with Cucumber here at the conference. So, helpfully here as well, Cucumber at the bottom there, is actually telling me the code that I need to write. There's a lot of built-in behaviour in Cucumber that comes from the core and from Rails itself and various plugins to Rails. But you will have to write certain bits of code to tell it when you have this natural language sentence. How do I map that to a test? So here, it doesn't know how to map this to a test. So this is the first line one we're failing on. The first piece of code I have to write is to explain to it how to make that precondition. I want two articles to exist with these titles. So I've just used a regex here, split up the titles on comments, and I'm going to create two instances of an article from those. So I run again. Instead of getting the undefined step definition there, I get a failure. So it knows what I'm talking about now, but uninitialised constant article. I don't know what an article is. It's a completely empty app at the moment. There's no model. So, okay, well, I need an article model then. So simply generate an article. It's got a title as a stream and body text, and I'll bring my database up so that it's got those database fields. Okay, awesome. We've got one green dot now. We've passed that first line. I'm on to the next one. So when I go to the list of articles, the web-runner component here knows when I say, when I go to, it understands what I mean. It knows I want to go to a URL, but it doesn't know how to map the list of articles to a path. So the first thing I'll do is, in one of the support files for Cucumber, I'll just create, and this usually has a big long switch statement with all of your different mappings. You can use regex in there, and the home page one comes by default. So most applications, that's going to be slash. And I'll just map the list of articles to slash articles. Go on. Still failing on step two. There's no route in my application to match slash articles. Okay. So I need to put in the main routing config to create all of the restful actions, to map, get and put and post to slash articles for that particular model. And I'll just create a stub controller, because bear in mind I'm just trying to make this pass at this point. I'm not getting ahead of myself here. I'm just going to try and make it pass as quickly as possible. So I create that action. We're still failing then when I run again. Missing a template file. Okay. So I'll go and create that template file. I don't care about the content at the moment, so I'll just stick that in the default layout. Okay. Two green dots and another failed step. So we're getting pretty close now. The I should see is another part of the aspect of matches that you get by default, which is just going to look at the actual HTML content and try and find those strings. So I haven't seen pizza within the content of my article, which I was expecting to see. Okay. So I now need to just iterate over all of the articles in my template and create a list item with the article title in. And then another failure. Okay. This time that variable at articles is undefined. So okay. Next step, define that. I'm just going to load all of the articles in the database into that variable. Everything's green. So at that point, the features complete. And essentially the good thing here is that Cucumber has done a lot of guiding for us. Some cases actually told me what to write where. It's quite useful, particularly those matching those step definitions. Problems with this approach. Cucumber above and beyond the general slowness of the Rails testing environment itself is even slower. There's a few things you can do to combat that. You can attach tags to the tests. One of the built-in tags is WIP work in progress. So if you have that tag on things, you can just run the tests for the stuff that you're working on now. And then when you've completed that set, you run the whole suite again to make sure you haven't broken something elsewhere and then you're on to the next thing. It's best not to try and map all of your possible error conditions and edge cases using Cucumber. Most people recommend just the happy path behaviour, what you expect to happen when everybody fills out every field correctly on the web forms, and the world's great. JavaScript needs a different approach. I'm actually not sure of the best approach at the moment. There's a few different techniques to test JavaScript, but the way the actual web fetching works within Cucumber, you can't do the JavaScript using default. So if you've got a very heavy JavaScript app, that's going to be an extra hurdle. The last thing is that unlike in the normal Rails tests, you get some access to the internals of the controller layer, so you can do things in your test code like log a user in, which is really, really useful, because generally you'll have one test that tests a user being logged out and a whole stack of tests for an admin user logged in and a regular user logged in. In Cucumber, without doing a lot of hacking, you've got to log the user in manually for every single test, so that obviously adds to all of that overhead. A summary for me, the value of these tools is that you've got documentation in a human readable format that's executable and verifiable that your app says or does what your documentation says it does. I think that's probably a concept that's going to spread beyond Rails into a lot of development frameworks. I think people in the Python community are starting to work with this stuff a lot, and I think it's a concept that most people can kind of see some value in. The Cucumber is a replacement for the Rails integration tests, but not your functional unit tests, as in testing your controllers and your actual model layer. We'd always go above and beyond what I just showed there. At certain points, we'd actually test the controller with error conditions and things, and we'd test the model with unit tests. Really, this is just that happy path testing that you want to do. You still want to test against all your edge cases. Lastly, as I said earlier, it's a common ground and a meeting point for you to discuss and document the project with your customer. Thanks for listening, everyone. Questions now. Steve, there's a question behind you just up there. Just the worst case scenario. Can you make a mistake in your regular expressions and it picks up the wrong thing in your implementings? A wonky regular expression? I think probably the behaviour would be undefined, usually. Is it easy to pick that up? I guess there could be a lot of varying different results from that. Hopefully nothing made you would happen, but we'll have an experiment later and see what actually comes out. Seeing as that's driving your whole development instead of testing what you thought was true, it's telling you what's true. One of the nice things is that, generally, you'll be writing the strings and things that the regular expression is receiving. Hopefully you won't be putting weird characters and stuff in there to mess things up. There's definitely that possibility for error there. Any other questions? The elephant in the room? Why aren't you using BDD now? That's a really, really good question. The two projects that I was going to be working on haven't actually come up yet, and the other stuff I'm working on is probably got enough and complex enough tests for me not to start down that road at this point, but I certainly will be in the next couple of months, I think. Every faux natural language system I've ever used has sucked terribly, because they're invariably not natural language. Do you find yourself having to keep in mind that this is just some strings that are getting pattern-ratched, or do you find yourself trying to write sentences that invariably fail? You're going to get the biggest gains if you actually go with what you know is implemented already, so obviously you are already constraining your language to a point there where I know that I want to write I should see here, and I should see this, and I should see that. The value is not necessarily, I think, just the fact that it's natural language. To me, I can see the biggest value in the fact that your customer can understand it, even if it's a subset of their native language. They can understand what you're writing down. There might be codified ways that you could make it readable as well, but generally the codified testing gets too complicated for most people to understand, I think. Anyone else? If there are no more questions, everybody, please thank Malcolm.


Okay guys, here's what we're doing today.   A lot of people have been sending clips, things that we're gonna laugh at. Ginny also has a punishment wheel. If one of us laughs, we have to spin the punishment wheel. Is that right Ginny? Yes, that is right. Okay, very good.   We're not gonna laugh the whole time. That's the promise, okay? Okay, okay. So are you ready? Yeah, I'm ready. Do you wanna watch my stream right now? What are you doing?  Mm-hmm. Mm-hmm. Oh, I'm sorry. I think I was on very stream for a second. I think. What? Oh, no, no. Sorry, I thought I was on the wrong channel. My bad. Are you watching my channel right now? Yeah. Really? Yeah, it looks good. Oh, it's so hard to. What do you think about, what do you think about my new clothes? I got this clothes recently. Oh, I think it looks great. I think it's... Do you see any difference? No. Do you wanna, do you wanna see something? Sure. Do you still don't see any difference? Looks the same. There's a better one. Mm-hmm. Uh-huh, uh-huh, uh-huh. What about this one? Which, which one do you prefer? One or two? They're kind of off-sided. Well, it's a side effect of boob job.   Is that how it works?  Yeah, but you know what? Because I got new boobs, there is a one thing that I would, I could do before, but I cannot do it right now. Yeah. Breast food. This. What, is it again? You're cutting out. This, I cannot do this now. Oh, I can't do it either actually. Look.  Let me see. Dude, Chad, the first time, the first time me and Ginny met, we went grocery shopping and I did this and she was fricking grabbing me dude, fricking. I wasn't grabbing, I, I, I poked it, I poked it. She did this and then she like squeezed. What do you mean squeezed, you wish dude. She didn't squeeze. Okay, when I play, you can, you can see it too, right? Yeah, it's good. Oh yeah, yeah, yeah.  So why people happy? So what is Cat doing? Let's see if there's a delay. There's no delay, right? So you have to share this screen, the watch together screen on your chat too. Yeah, I am. I love cats. I'm gonna be crazy cat lady in the future and 10 cats and one husband. So this is a test video. Okay, this is the test video. This is test video. Tell me what, if when you're ready. I'm good. I'm working. Okay, good. I'm ready. Here we go. Skip this. This, this is lame. Very much this. Liking. Any skippers? Okay, so first one skipped. Very good. We're off to a good start chat. Second clip. Second video. No laughing, no laughing. No laughing. Okay, yeah, yeah. Okay, no laughing. If we make each, hold on, time out. If we make each other laugh, does that count? Yeah. That counts. Oh, frick. Okay. Espen, look, look. Nope. Point the video. It is my birthday and I was wondering if I could get a feels birth daemon from chat. Really appreciate the streams and content you provide to us especially in times where it is hard to find happiness.  Anyway, look at the size of it. Oh, this is the loudest. No, come back.  Can I hear it? Oh. I know you will keep going like you always do. Damn, man. Oh, oh, oh.  Oh, my God. Next clip. Next clip. She does have three fingers. She does have three fingers. Wait, what? She does stream? That wasn't me. That's the video. You 100% laughed. I did not. That was the video. That was not me. You should you should you green. I smiled. I'm a happy person. I'm a happy person. No, you lost. I can smile. Am I not allowed to? Is it a crime? Is it a crime to smile? Why do you?  A lot because it feels good. Do you do you like doing that? Frank, that doesn't count. OK. That counts. OK, fine. I'll spin the wheel. OK, how do I spin? One gift sub. I have to get one sub. OK, so is it one gift sub to you? OK. OK, I just have to address this because like 10 people have asked, no chat. Ginny did not get a boob job. She just did a stream with Barry. So she's put balloons under her shirt to compensate. No, no, no.  No. What do you mean no? Can you pee without boobing? Of course. But can you poop without pee? Yeah. Prove it.   I knew it because I've seen this video before. Why did I laugh? OK, OK, spin the wheel. The moan made me laugh. I knew the video was coming. I've seen it before. But the moan still got to me, dude. I lost the blow. Wait, they said you laugh, too. OK, guys. Wait, so I got five gift sub. OK. I draw on face. Wait a second. Hold on, Ginny, can you send me the link to your spin the wheel in Discord? Oh, yeah, of course. So that'll show up on my stream. OK, what should I draw? I'll try an Aspen. What about this? Try a teaspoon of Tony's. Yeah! And raw Tony's? How am I going to get raw Tony's? I don't.  Like, come on. Who do you think I am? Oh, nice. Cheers. Yeah, cheers. So one teaspoon. Is this a teaspoon? Yeah, a teaspoon. It's a little big for a teaspoon, but I think that's good. Oh, you love Tony's. Well, I do. I love my Tony's. I don't know if a teaspoon of it is going to be good. Wait, hold on. I should go on my green screen. Don't try too much, though. Big. And then put it in a spin. All right. One. Oh my god, that's. One teaspoon of Tony's. OK, that's not a teaspoon, by the way. But OK, play laugh. Go, Espen. Go, Espen, SMU. Ready, three. This is a tablespoon. Yeah, that's tablespoon. How much smaller is a teaspoon? I don't have a teaspoon. Just get half of that. That's OK. It's whatever. Ready, three, two, one. We've stalled enough, Espen. This, if you're OK. And it's fine, Jed. It's fine. I lost. I laughed.  I'm not supposed to laugh at watching. OK, I have to spin one more time. I got it. Oh, saved. You got it, too? My mouth is on fire. It's actually good. You should not do what I just did. I have to do it. That's a lot. This is a teaspoon. This is teaspoon. This teaspoon. OK, I'm Korean. I'm good at eating spicy food at all. So I think this is going to be easy, though. It's just so mean.  No, just kidding, OK. You laughed. No, I was on a lap. That was a puke. OK. Wait, a serving size is 1 fourth of a teaspoon. So. So that's 12 servings. 12. What is 12 times 15? 12 times 15? It's Fantinio. No, I was joking. I'm not from Brazil. Play the next video. Yeah, we shouldn't. Actually, we should not do that one again. That one's that one's bad. I just had 12 servings of Tony's. Two shuffles, is that OK? What's up? And my friend was two shuffles. How'd you know me? My friends literally messaged me here. Messaged you? Yeah, literally, like, I'm in town with friends who are like, hi. It's me. You can also say, can you say hi, Callum? Hi, Callum. How old are you? I'm 15. Frank. Yeah, I laughed. Noel asked me. I laughed. Frank, I'm sorry. OK. WeirdSelfie on Twitter. What? Yes! On Twitter. WeirdSelfie on Twitter? Do what? Do a duck lip. Do this face. Do I do duck lips? Wait, does she have to choose what I do, chat? Yeah, of course. Does she have to choose what I do? Yeah, so when I get it, you can choose as well. Wait, how do you do this? Wait, why is it taking 10 of them? Like this? Is this good? It's good. Am I allowed to say anything? Or no, I'm not allowed to say anything. Well, you're not allowed to say anything. You just have to be completely, OK, well. Oh, why don't you say, ah, yeah, yeah, together? No, I was already tweeted. You're late. OK, I'm going to check if you did. No, you don't need to check. Oh, I have to check. You did. Oh, you did. Next video. I don't know how to retweet. I'll do. I'll learn. OK, go go. Next video. We saw people in the streets ready to put 2018 in the rear view mirror and say hello to 2019. We also saw this dog with a banana. What the laugh? I'm sorry.  What's wrong with me? What did you laugh? I don't know. Watch out for question mark, question mark, question mark. You're so bad at this game. It's been it. Oh, are you serious? Why'd you put this on here? OK, this is hard. This is hard for me as well, dude. OK. This is this is just not holding it in and then not laughing at your ego is so hard. OK. OK, so what is it? Is it in Korean refers to a cute display of affection often expressed, including, but I'm going to do the cute baby voice facial expressions and gestures. Yeah, cute baby voice. We think OK. Why don't you type in ego in YouTube and then you can copy it after that. You can copy after that. Yeah. Aguio is a way of acting cute that is popular in Korean culture, especially among K-pop idols. OK, so here's how we're going to do this. Oh, yeah. Do that one. That one is easy with this.  I'm Samira put some shoulders into it. I was having kind of I was kind of before. What? Anyway, next video, MF7 recast. I actually saw this on Ginny's stream. And this is actually incredibly impressive. I have to admit, this is incredibly, incredibly impressive. That is really good. The only way that it could be better is if we just kind of like this is this is really, really good. I don't know who drew this. But it is it is pretty much perfect. MF7 LLY fan. I was not a jelly fan. Chad, was I being jelly? Jelly fan. My chat is spamming my. My chat is like spamming jelly fan, jelly fan, jelly fan, people clap. As I realized, you're streaming. Maybe you've been serious. Dude, why are you following us? I like all your streams. I feel strong. You follow us on TwitchCon. All your streams, please. I watch all your streams. OK, let's prove it. What's his name? Tyler's brother.  Why are you laughing? He's Tyler's brother. He's Tyler's brother. Well, hello, Jesse enemy. I'm Dr. Slam the fart. And unfortunately, I just got back the results from your epic gamer tests.  Now, this is a very serious condition that has a high fatality rate. And there is only one conclusive test. If that's the case, the most effective treatment we can try is me banging your mom. Got him a little bit. Anyways, go eff yourself, Baldi. OK, spin the wheel. Spin the wheel. Spin the wheel. Frick, weird Twitter message. Oh, my gosh. Oh, my God, weird Twitter message. And weird Twitter message. What should we say?  I love anime. Kanye, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, Let's just do I love anime, ay-a-ya. Sexy dances, sexy dances, add it to the will. I did. Stop, guys. Stop liking this. Like, people are going to see and they're going to think it's real. They keep liking my tweet. Stop. You just posted this, and then you uploaded I love anime? What? No, my Twitter looks terrible. This is awful. I have this picture, and then I have... What? So that was a tough loss. How do you feel after that?  So that was a tough loss. How do you feel after that? I don't know how to f**k. Okay. Do the sexy dance, do the sexy dance. No, I have to spin the wheel. Why do you want me to do sexy dance so much? No, okay.  Sexy dance! There's no way. How did that actually happen? Because I'm a witch. I made it happen. Did you rig it? Time to... No, not time to coo- Not coo- You're serious. I don't know, good. Yes, that was good. Did that do it for you? Did that do that?  What do you think? What do you think about this kind of... When's it my turn? That's a joke. That's funny. Shameless. It was a joke. Shameless, fans. Stop laughing, fans, please. I'm trying not to. Weird Twitter message. Say I'm horny. What? I'm not tweeting, I'm horny. You laugh too much. I'm not tweeting, I'm horny. I can't believe I just agreed to this.  You were laughing at everything. Why? Are there 200 likes on this in 12 seconds? Why? What are you guys doing? I'm horny. I love anime Ayaya. This and then I am horny. Holy moly. Retweet, retweet. What should I... I'm gonna reply. Comments are so funny. Look, I'm never... I'm never playing this game again.  There's two minutes, 1.3K likes. Oh my... And do not laugh. Okay? Gosh, but it has more likes than the last one. It's fast. Marginal success called meeting tonight. No, that tweet after the horny...  Exactly.


If you're a system admin or have experience in uploading hypervisors, you must be grateful of the long maintenance windows and the VM down times. And even more scared is to face a failed VMM upgrade after having spent a long time in the maintenance window. Hi, I'm Soham and my colleague Prasadthose from Nutanix. Together, we are going to talk about these issues and our solutions for such problems. As part of today's talk, we are going to cover the following in this. Firstly, we'll cover what are the problems with VMM updates. First, we are going to discuss the design implementation of our solution. And finally, we are going to share our results and the future work. So what are the problems with VMM updates? Firstly, the VMs need to be migrated outside of the host, and then only we can update the hypervisor stack on the host. And finally, the VMs need to be migrated back onto the host. Now this process needs to be repeated for each and every host of the cluster. As a result, we end up having a long maintenance window. First, I will highlight few issues that we have often observed at Nutanix while performing VMM updates. The first issue is non-determinism. What we mean is that the VM migrations can fail after an unbounded amount of time. Next is the guest impact. During VM migration, due to, for convergence, we need to throttle the VCQs of the VM. Now this has an adverse effect on the guest performance. And finally, it's resource contention. This tech network, as an example over here, during migrations, a large number of pages need to be transferred over the network, which results in an increased traffic of the network. Now this same network might be used by the storage service as well for IO operations. And as a result, we might end up having a network congestion. Because of reasons like this, we have often seen that the customers tend to defer the VMM upgrades. So how do we solve this? To solve such issues, we propose a novel approach of upgrading VMMs using local live migration in Livevert. With this approach, we can migrate the VMs locally within the host using Livevert's existing migration workflow. There is no memory copy required within the VMs. And as a result, there is no dirty logging and throttling of VCQs required. With this approach, we can migrate the VMs at a very fast speed and with near zero downtime. Since there is no requirement of transferring the VMs outside of the host and then bringing them back, so the upgrade workflow time is greatly reduced. And hence the maintenance window is also minimal. We hope that with future work, we can further reduce the maintenance window to the point that it can be gotten rid of entirely. Next I will go over the high level overview of the local migration workflow. So here we have a typical setup of a host where we have a QMU binary installed and here we have QMU 6.1 installed. And on top of this, we have a VM running, which is being managed by the Livevert service. Now let's say we want to upgrade the QMU binary here. So we issue our EM update command. This updates the QMU binary to the latest version of QMU 6.1.1. Now from Livevert, we issue a local migration command. What it does, it spawns up a new QMU process on the newly upgraded QMU binary of 6.1.1. And after this, it transfers the VM state from the old QMU process to the new QMU process. Once this is completed, we can then finally destroy the old QMU process. This completes the workflow of the local migration. Next processors will be going over the design and implementation part and the remainder sections of the presentation. Hi, I'm Prachatosh and I'll discuss about the design and implementation as well as the results we worked in for our work on the VM in Live updates using local migration. Before diving into the details of the implementation, we'll discuss some of the design challenges we had as well as some of our goals for implementation. While implementing local live migration, we wanted to keep the machine ABA unchanged. Specifically for us, this meant that we had to handle the issue that Livevert expressed the name and UID of a VM in a particular host to be unique. Clearly, this is not true in local live migration because there will be a source VM and a destination VM sharing the same name and UID. Therefore, we have to account for these conflicts and handle them appropriately. Some other dependencies on name and UID include the absolute path dependencies Livevert has on for some of the files. For instance, a monitor socket path depends entirely on the UID of the VM. Therefore, this ambiguity is needed for this case as well. Once you have addressed the issue of keeping the machine ABA unchanged, we have to make changes to the existing Livevert migration workflow to enable local migration. The current Livevert workflow has five phases for migration. Begin, prepare, perform, finish, and confirm. Of these, begin, perform, and confirm, work on the source host, and prepare and finish on the remote host. However, for local migration, the remote and source host are one and the same. Therefore, the remote steps must be made to be working on the same host as the source during local live migration. Finally, we also have to resolve the correct domain object and use the correct domain object. This is a remote object for the remote phases and the original object for the source stages during migration. One might ask what the benefit of local live migration is if you are still incurring the memory copy phase. This is the most expensive and the most unpredictable phase of migration. Therefore, we really want to avoid the memory copy. We will see how we achieve this in the next slide. As I mentioned, avoiding the memory copy was a primary goal for us while doing this design. To allow bypassing memory copy, we implemented an FD transfer mechanism. Through this, the source KMU uses live work as an intermediate to pass the FD of its memory backends to the destination KMU. The way this works is as follows. First, we change KMU to add a new QMP command called fetchbackingFD. This fetchbackingFD command will send the FD for the memory backends, all the FDs for the memory backends to the client using SCM rights. This is invoked during local migration in the begin phase. Libvert queries KMU and receives all the FDs. Next, Libvert prepares to start the destination KMU process, the destination VM in the prepare phase. At that time, we use existing Libvert FD passing workflows to send the received back end FDs from Libvert over to the destination KMU. Now, we again have to make changes for the destination KMU so that it accepts this FD and knows not to open the file or allocate memory, but instead simply use the already opened FD that it has got, that it has received from Libvert as far as memory backends. Now, when the iterative phase is supposed to start during the perform phase, the destination VM already has the FD and all the memory that it requires. Therefore, we can mark all the pages as capable, and there is no memory copy, copy, there are no memory copy iterations at all. In this way, we avoid the entire copying phase and improve on the migration time as well as improve reliability since the failures during copying are not going to happen for this case. The every transfer mechanism is not restricted to memory backends alone. It can be extended to other devices such as block devices and character devices. This will improve the migration times further. However, every transfer does not solve all the problems related to enabling local migration in Libvert and KMU. Just in cases, we can see that every transfer does not work. For instance, with a monitor socket path, it is already open in the source VM which keeps running during the migration, and therefore, the FD of the socket cannot be passed to the destination VM. However, the monitor socket path has a dependency on the VM new UID. For instance, it is some location, then domain-vm1uid-monter.sol. When we do local migration, we'll have two VMs with the same UID. Therefore, this will cause a conflict. Instead, we disambiguate between these two VMs using the numeric domain ID that Libvert assigns for each VM. For instance, say VM1 has domain ID 1. We will append the domain ID before the VM1 UID, so the path will become domain-1-vm1uid-monter.sol. The back-end compatibility will create a sim link from the expected path without the domain ID to the new path that has been created. This way, the expected path is always available and pointed to the correct path in the initial fields. In the prepare phase, we launch the new KMU process, and for it, we create a monitor socket with domain ID 2, which is the domain ID for VM1 hash. Now, the VM1 hash UUID and VM1 UID are the same. VM1 and VM2 are the source and remote counterparts, respectively, and therefore, they share name and UUID. So, the domain ID is the only thing differentiating between these two paths. In the perform and finish case, there are no changes, but in the confirm phase, we kill the source VM and we also remove the monitor socket. Now, the currently running VM is the destination VM-VM1 hash. Therefore, we switch the sim link over to the VM1 hash. We can see that at each step of the migration, the expected monitor socket path always points to the correct path in the new format. In stages where the source VM is running, it points to the source VM, and when the switch over takes place and the destination VM takes over, it also switches the sim link to point to the destination VM. This maintains backwards compatibility, while also allowing us to disambiguate between the two VMs, which we required for enabling local migrations. We'll see a similar example for the case of modifying the migration phases. For the migration phases, the main challenge we have is to use the correct domain object, as well as ensure that the data structures Livert stores do not have any conflicts. Livert has hash tables indexed by the UUID and name of the VM. In normal operation, this is not a problem since these are unique, but during local migration, a query for a VM and UUID may fetch two different VMs, the source and the remote, if we do not make any changes. In order to separate these two VMs, we create a remote hash table, which is initially empty and is only used during the remote phases of the local migration. We name the existing hash tables, source hash tables, as they will be used in the source phases. The source phases have a little change and in the begin phase, we query, if you want to query for VM1, using the VM1 UID, we'll get the source VM VM1. Before starting the prepared phase, we need to populate the remote hash tables, since that is the hash table we want to use. So we create an entry for the same UUID, that is VM1 UID, which is the VM being migrated, but we create a duplicate of VM1 called VM1 hash, which is a remote object corresponding to the source VM. This way, when we query on the remote hash table, using VM1 UID, we'll get VM1 hash, which is the correct object to work on for the remote phases, but when we do it on the source table, we'll get the VM1, which is the source VM, which is the correct object for the source phases. We can clearly see how we are separating these phases and ensuring that they work correctly by adding this extra table. So prepare works on the remote hash table, perform on the source hash table, finish on the remote hash table. Now, after confirm base, the VM1, the source VM, will get killed, and VM1 hash will take over. We have to make sure that this is also correctly reflected in the source hash table, since for all limits, interfaces, and queries, they'll be on the source hash table and remote is only used for the local migration work. In order to ensure this, we copy over the entry from the remote hash table over to the source hash table. Now, the VM1 UID queries will turn VM1 hash, which is the correct VM at this stage, since this is the destination VM that has been successfully migrated. Note that again, at each step of the migration and at each phase, the source hash table always contains the correct VM that is running. Therefore, commands such as worst list and other commands will always return the correct VM and they will continue to work at the end of the migration as well. So in summary, we wanted to avoid the memory copy for local migrations. Therefore, we implemented and extended the every transfer framework from the KMU, using from KMU to the destination KMU. This avoids the iterative memory copy and improves our performance. For cases where we could not use FD passing, we used a sim link approach. We created entries for the paths for soft monitor socket and log files using the domain ID for differentiation. And then we used a sim link that we switched over to point from pointing to the source path, then to point it into the destination path at the end of the migration. Finally, for changes in the migration flow, we added a shadow remote hash table, which is used during the remote phases and then switched it over to the source hash table once the migration was completed. We can see how we did not break any of the existing interfaces in LibBot. However, we were able to implement local migration while having the while having two VMs with the same name and UI. We'll now go over to a brief demo and then look at some of the results for our local migration implementation. Prior to the demo, here is a high level approach of the workflow we expect. Using your favorite package manager, for example, YAMD, one can update the KMU binary on the host. This will move the KMU file in the file system from an old version to a new version. However, the VMs running on the old version continue to run on that version itself. Therefore, we have to locally migrate the VM to the new version. This is done using the local migration command available from BERSH. We add the minus minus local parameter and the command list BERSH migrate minus minus local domain id. Other fields are not needed for local migration. After the completion of local migration, all the VMs should be able to run on the updated KMU binary. We'll see this workflow in action in the demo. On the left, you can see that there is a guest VM running Nubuntu. The right is the host on which this guest is running. Now, we can check this by issuing a BERSH list command which should show this VM. Next, we want to update the KMU binary. So let us first look at what version of KMU the guest is running on. When we check this, we'll see that the version is 6.1.0. So, of course, we have a version 6.1.1 with some fixes we need to, we want to bring into this host. And therefore, we install it using a package manager, for instance, using the package in this case. Once this is installed, the KMU version on the host has been bumped up to 6.1.1. However, the VM still remains running on 6.1.0 and I'm missing the changes between these versions. Therefore, now we need to issue a local migration. Prior to this, as an example of the impact of local migration, we want to run a program that has a very high write-show group. The program we're going to run writes to each page and increases the dirty rate at a rate specified by us. And a higher dirty rate leads to longer convergence times since the iterative copyface keeps on going until all the pages are left to copy over. Now, we'll execute this program and issue the BERSH Migrate local command. Note that the dirty rate we have specified here is 8GB, which is which means that it is dotting the entire size of the VM. This is also 8GB in size. We perform the local migration command and it finishes within a second. Note that here there is no dip in the dirty rate at all during the migration. So in terms of impact of local migration using every transfer, it is basically zero since there is no iterative memory copy and there is no auto converge algorithm which will throttle the throughput of a process. Therefore, throughout the migration, the time the dirty rate is the same as before. Now, we can take a look at the metrics, specifically migration time as well as downtime that was incurred for this migration. We show the downtime for command on the VM and we will see that the total time taken is 830 milliseconds, which is very low for a VM running a high throughput workload as well as the downtime is only around 30 milliseconds. We'll elaborate on these results in the following slides. The first metric we'll take a look at is migration time. The graph on the right shows the migration time for three algorithms for four categories of VMs. The yellow line shows external migrations. The blue line shows local live migrations using an iterative memory copy and the green line at the bottom is local live migration using every transfer that is the approach we propose. So VMs are in increasing order of size. VM 1 has 8 GB of memory and VM 4 has 32 GB of memory. All of them are running a high throughput workload, the same we showed before in the demo and the workloads are darting the entire memory in a second. So this is a very pathological case. We see the impact of this on the external migrations in the yellow line. The time taken grows from 20 seconds to over a minute for the largest VM. And this will keep on going and some migrations for even larger VMs may fail to converge. Or iterative memory copy it is better, but still there is a linear increase and the largest VM takes over 45 milliseconds or 45 seconds of time. These are not viable approaches as they are affected by the size of the memory and the workloads. In contrast, the FDA transfer graph, the green line is almost flat. If you look at the left, we see the same graph of every transfer in milliseconds. The time taken varies between 800 to around 850 milliseconds for all the cases. Clearly, this shows that there is no dependence on the size of the VM or the workload that we are running here. Even for larger VMs, the time taken should not grow. And this is a clear benefit of using every transfer mechanism for an implementing local live migration in this fashion. One can migrate any workload of any VM in a very low time, which will reduce the maintenance windows required as well as the reliability will also improve since there is no chance of failures during the iterative copy case. The next metric we'll take a look at is downtime or the time the guest is unavailable during the migration. We want to reduce this as much as possible to improve the upgrade experience. We see a similar trend for downtime as we saw for migration time. The downtime is of the order of less than 50 milliseconds for all cases, respectively of workload or VM size. For iterative memory copy, the round time is around 150 to 200 milliseconds. And for external migrations, it varies widely from 400 to 300 milliseconds, depending on several factors. For instance, the network bandwidth at the iteration prior to the downtime starting. Again, we see a similar trend and this proves that the fact that we do not, the size of the guests or the workloads are not a factor for this approach, which is obvious since there is no memory copy phase in part. We also note that the guest penalty is zero as we saw during the demo. Therefore, we have improved the performance as well as the reliability since we have avoided the most unpredictable phase of memory copy. And this should significantly cut down on VM upgrade times. In conclusion, we have enabled chemo upgrades using local migration. We have made changes to the existing network migration workflow and have extended on the every transfer flows in chemo and live code to allow a local migration without memory copies. Using our approach, our migration times are around one second and our down times are less than 50 milliseconds, irrespective of the size of VMs or workloads or other metrics of what is running on the VM. In terms of future work, we want to extend the every transfer framework to all types of devices. We have tested with a subset of devices. However, we want to explore more and look at a wider set, including looking at the case of past devices as well. Finally, we want to continue upstreaming our patches. We have sent out a prior version of local migration for review upstream last year. We want to update this with the latest chemo and liver changes we have made and want to want to continue the discussion upstream. We look forward to the comments and and discussing our approach. Thank you for listening to our talk. Please feel free to reach out to us at our email id shared below.


pkgnzone. Basically, what we want to do is to generate an RSA encryption with OpenSSL and then we will do it with a NWB55. That means I will create a project from scratch on this platform and we will try to reproduce an encryption that I have been done thanks OpenSSL. So for this nzone, we will use the STM32WB55 Nucleo and I propose to start a new project with STM32WB55. I will select the MCU selector, one that we expected is the one from the Nucleo. So this one. Okay let's test the pkgnzone. That's it. So the only thing we have to activate will be here. I want to activate the pkgnzone. As you can see, we can't configure it inside STM32WB55 at the moment. So we will do it this manually thanks HL. Let's generate the code. Okay, if we check what we've got in the main, it's quite simple. I will say we've got the pointers on the structure needed by the HL. We've got a pkgnzone. But in this pkgnzone, here you see no parameters. So we will need to do a new initialization on a proper one. So for the moment, I will say that we've got the infrastructures to put our code. Let's switch to OpenSSL to generate our test vector we will use now. So now I launched OpenSSL on my windows and we will generate together the vector test or what we want to reproduce with our pka on the STM32WB. So first thing, let me show you the version that I'm using just for your information. And now let's generate the RSI key. So my private PEM and we could select a size of 1,024. Okay, we could check the value generated, my private key. Okay, and we would like to see it in text form. So here I think now we are familiar with the structures. We've got the modulus, the public exponent, the private exponent, the 2 prime number, the exponent 1, exponent 2 and coefficient that could be used to do the CRT modulus exponential. Here we've got all the fields needed and we will just have to convert this to put them in the C structures for our test. Let's continue to, I will say, encrypt a file. So now I will extract the public key from this private key structure, I will say. From the public key, I want to have the public key and it will be put in my pubkey.pen. Okay, so now we only have the public key in the my pubkey.pen. I prepare for you a file that we will use as input. I propose something quite simple as you can see and align on the 1,024 bits for simplify the example. Okay, so this is our plain text and I want to encrypt this with the public key that we will have here. Okay, so the command is open ssl, eresa, and we will do a row conversion. We will use a public key as input, the in key with my public key.pen. We will do some encryption and we want to encrypt plaintext.pen and the output will be in plaintext.pen. Okay, if I just check the content of plaintext.pen, here we are. So now we've got our victor test. We've got the input or what we want to encrypt. We've got the output, the encrypted messages, and here we've got all the value public and private to do the encryption. In fact, we will only use a public key here. Okay, so what I propose is we will copy all this stuff in our C file and then we will put it in a command at the beginning. For example, in this part. Okay, so I would say what are we interested in. Here we've got the value for the key, for sure. We're going to keep them. This format is not interesting for us. We don't need the command. The plaintext is here and the encrypt text value is here. Okay, so let's have this first. I would say this is our victor test now we will use. Let's declare the C structures with our key value. So we'll take the mobirus, public and private exponent. Let's declare private variable with those values. For sure. Then I will probably convert all the value. Okay, for the public exponent we can do it directly. Exponent in the same way. Okay, to convert this I will just use text edit for example. Just copy past this. We can search and replace. So let's replace each point by this and then we've got replace. It's okay with this one. It should be okay. So this is for the modulus. Okay, just one remark. It started by zero and with open ssl there is a coding in isn1. When you've got zero zero to start that's just some things to indicate the sign of the value. So it's a positive one and you have to remove it if you want to have 128 bits. So please if you've got zero just remove them. Okay, it's a boring task. I do it with you that way you can see how I do it manually. Not very interesting but it has to be down. So I do the same for the private. There is not started with zero zero so no modification to be down. Okay, let's align the structure. It will be better. Okay, we are aligned on the size. Good, so key value are now defined. Let's define our plain text location. So the test victor we've chosen on this one. Okay, quite simple. So let's initialize it by ourself. Okay, so now we are I will say ready for the structures. Let's define also the output structures. So we will get the result which will be encrypted data and eventually decrypted data if we want to double check this. Okay, I will just try to compile just to check if I haven't made an instant X error. Good, so now we can have the happy high. So let's move to the main. Okay, we've got the Pekka and it's just done here. If we check in the code what we've got here. Pekka in it is done for the hash error. Good, let's come back to our main. And let's see what I want to do. It will be a module's exponent action. So module looks exponent. Okay, the structures we've got an input structure that we will see together. And here let's put some time out value for example. So let's check what is this input structure we have to use. So number of side of the exponent array, operation one number of element and modules. Okay, so I won't say it's trivial but it seems quite obvious regarding what we've got as input. So let's declare this variable here. So our input parameters. Okay, then so exponent size equal size of our public exponent addition to the job first, sorry. Then in that, hope the size is the size of the modules. Next one, our exponent. So here it's our public exponent, the modules and the last parameters. Oops, sorry. Our operand which will be our plain text. Okay, that's I think it's okay here. It's a pointer on the structure. So here we'll do the exponentiation to the operation will be down here. Let's check maybe the results or first if the status is okay or not. If it's different from a shell. Okay, add a hero here. And if it's working fine, we will need to get the results. So here we've got another happy eye for that. A mode X catch results. Okay, here the pk and we will put this in the encrypted encrypted data. Okay, let's compile this. I think it's okay. So now we need to debug. So I will launch the debug. Sorry, I'm plugging my board on a USB help, not bad idea. I will unplug it. I propose to add now the description. So it's exactly the same operation, but we won't use the same parameter. Here we will use the private exponents, private exponent here and we will use the encrypted data. We need here to increase because one second won't be enough for this description. Let's take five and should be okay. And let's put the result in decrypted data. Okay, I hope I haven't forget anything. Tim note and below. It will build my sources. Still have breakpoint on the white one. So here it's finished. Let check the encrypted data now. And we've got what is expected. So we have encrypt and decrypt. Okay, so it was exactly what I want to show you here with the pk. I hope you will like this hands on.


We are about to lose signal through the tracking data relay satellite system, so we'll be losing these downlinked television pictures. This is Mission Control Houston, this television coming from the Spacehab double module in Atlantis' cargo bay. Comments on that change we just read up. The pictorial later on in the deorbit prep on page B3-24 already correctly shows the power system switches off. And also INCO says that he does not think you'll actually get the SPEC-62 message, so if you don't get it, don't worry about it. Affirmative? Spacehab pocket lanus for RMD. Go ahead. Go ahead. We copy. Floating into the field of view now in this live television picture from the Spacehab double module in the cargo bay of Atlantis to join Apton Walls is Mission Specialist Shannon Lucid on the eve of her return to Earth after six months in orbit aboard the Mir space station. Again, her colleague John Blaha, veteran U.S. astronaut, is aboard the Mir at this hour with Mir 22 commander Valeri Corzun and flight engineer Alexander Caleri beginning his scientific research over the course of the next four months. Blaha will be returned to Earth in January aboard Atlantis on the STS-81 mission, which will carry astronaut Jerry Leninger to the Mir as the fourth U.S. astronaut to conduct research on the Russian space station. Good morning. Shannon, I want to talk to you first if everybody does in mind. First of all, you guys are, we're mid-deck now, right? So you're pretty much cramped in. All kind of wedged into the portion of the flight deck where we flew the rendezvous from. Shannon, let's talk about your time aboard Mir. I know it's difficult to prepare for six months in space, but was it what you expected from your training before you went up? Well, yes. I think everything was pretty much as I expected, except I think I had a better time than I really... I mean, I expected to have a good time, but I think it was even better than I expected. Well, I'll be happy right now. Now, how sad are you about leaving space? I mean, obviously the anticipation must be great before you go. Is it time now or are you ready? I do feel just a little bit sad because I've had a great time. I've had great friends, two different crews when I was on Mir, and everything just worked out just really well. And yeah, I feel a little sad, but all adventures have to come to an end and you have to go home sometime and I'm ready to go home. He says compatibility and good humor are two of the characteristics which come through in all of your interviews, Shannon, especially over the last six months. So for long duration missions and eventually flights to Mars, how important will it be to assure good compatibility among all the members of the crew? My flight, such a great flight, was the fact that all the crew members got along real well. I could not have asked for a more compatible group of people to work with, and I think that is of the very primary importance. I mean, that is the prime, the most important thing. Everything else falls underneath that. But if you have people that you can work with, that you can get along with together, and that you laugh a lot with together, things just tend to work out pretty well. Shannon, is there anything about women that may make them psychologically or physiologically more adept at being astronauts, especially over long periods of time? I think it just depends on the individual. I don't think it has anything to do with whether you're male or female.  I mean, that's what makes human beings so wonderful is that we all like different things. Can I ask about our partners in space and the International Space Station, the Russians? There seems to have been a real problem and a continuing problem with them, not so much as far as their technology is concerned, but certainly as far as cash flow is concerned for them. Are they going to be an OK partner for us to be able to go forward with, or is this going to present a problem for us? I'll start with what you said first. I think they build marvelous hardware, and most of what we saw over aboard the Mir station is just elegantly simple and robust, and if you think about the fact that their station's been in orbit now for over 10 years, it's really remarkable. I think we were all impressed with the volume that you have available to do work over there and just how magnificent a facility it is. And we're going to move off using the Mir as kind of our prototype space station, sharing it with the Russians and using our shuttles to go back and forth, and then we're going to build the International Space Station, and the Russians are contributing launches and they're contributing modules. I think their finances are their business, but I think when it comes time to produce, that they'll produce and they'll produce quality hardware. Shannon, for you, being up that much longer, has it made a really large change in you? I can tell. I haven't changed. I'm the same person I've always been. I guess you could ask these guys, so they thought I was changed, I was different. I think I'm the same person, but a lot of times a person's not the best judge of themselves, but no, I don't think I've changed any. All right, we're really quickly running out of time before we go. Shannon, tell me what you expect physically as you come back to earth and gravity, because I know you've flown a number of times before, but there is a distinct difference in this flight, and we have very little time. There's a lot of muscles we haven't used for six months, and so it's just going to take a little time to get back used to using them, and I just think I'll be tired for a little bit, but I'll rapidly be my normal self. Okay, we copy page 1-45, step 45 report condition of isolation plate to MCC. That's firm. Sequence look good. Good config, minute to the ZOE. We'll pick you up at one minute after the hour.


That's part of why I change jobs all the time. So yeah, I'm going to talk about validating big data and machine learning pipelines. I think it's a super important topic, and I wish more people validated their pipelines. Oh, I need to speak up. OK, cool. I'm going to try yelling, and this is going to be a long day. So if my voice starts dipping and you can't hear me, let me know, and I will start yelling some more. Yay! OK, my name is Holden. My preferred pronouns are she or her, and I'm a developer advocate at Google. I'm on the Spark PMC, which is why most of my examples involve Spark, even if it is a really bad idea. But that's OK, because you can do the same tools with other things. I'm a co-author of two books on Spark. Neither of them talk about anything in this talk, but that should not stop you from buying my books. That is the more important part. One of the things that I've started doing that I think is kind of cool, if anyone is particularly interested in how projects like Spark do code reviews, is I've started doing code review live streams, where you can watch me review open source pull requests live and try and not swear. I fail at the second part, but I still think it's fun to sort of watch and join in. OK, cool. In addition to what I am professionally, I'm trans, queer, Canadian, and America on an H1B work visa, which they're debating whether or not they want to keep. It's a really great feeling to not know if you can renew your work visa or not and part of the leather community. And this is not directly related to big data or machine learning. However, I think for those of us who are building machine learning pipelines or even just tools with data, it's really important that we try and make diverse teams. And this includes us in the open source community. If we don't talk about where we're all from and if we don't build diverse teams, we're just going to recreate yesterday's problems more efficiently. And that is not really what I want in life. I want us to find new solutions to new problems. And I think diverse teams will help us do that. OK, that being said, not going to talk about that. So we're going to talk about how to avoid having everything catch on fire. Why you should do it, given that you're here, you might be fairly convinced. And I promised at least one cat picture. And at least one picture of my scooter club, which is only tangentially related. But I do wish to try and expense my gas. So I'm working on making it more related to computers. So hopefully you're nice. Maybe you're familiar with, is anyone here familiar with Scala? Yay, friends. Thank you. OK, cool. And if you're not familiar with Scala, that's totally fine. How many people are familiar with something like Spark, Beam, Flink? OK, that's a good number of people. If you're not, it's OK. These same techniques apply to other systems. Generally they're a lot easier to do in non-distributed systems. So if you happen to be working locally, your validation tasks become way, way easier. And hopefully you're here because you want to make better software. And if not, I can't convince you. OK. So validation is really important. Driver's license tests can be very similar to how we test our software, which is to say, they're better than nothing. I would not want my friends to start riding a motorcycle without a license, as my friend is doing here. But at the same time, even if someone has passed their motorcycle license test, they're probably maybe still not the safest driver. And it's just like your code. Your test covers the basics, but it's not going to catch everything. There's always going to be that strange SUV operated by some random drunk person, or you're going to have some null records in your CSV files randomly. One of the two is going to happen, and your pipeline is going to fail or you're going to crash. And ideally, you want to know when something has gone wrong so you don't make it worse. So our tests are not perfect. We are eventually going to get on the fail boat. And at some point, you want to minimize the impact of this. Does anyone here have to carry a pager? That is very, very few people. You are all very lucky people. I am kind of jealous. Someone else in your organization may be woken up at 3 o'clock in the morning to do a rollback from your data pipelines.  So it is still worth saving other people being woken up at 3 o'clock in the morning, even if it is not saving yourself being woken up at 3 o'clock in the morning. So I did a survey for how many people have had spark jobs, causes serious production outage. 15% said yes, 50% said no, and 30% were like, I didn't have to update my resume, but we did lose a few million dollars, also known as depends on what you mean by serious. And so you really don't want to be in the 15%, and you don't want to end up in the 30% either. You don't want to have your pipelines cause really bad failures. And I have a survey if other people want to give feedback on this. And as more people deploy their pipelines automatically into production, and as more people start doing streaming data, you don't have the same time to do manual verification and validation that you used to. So when scooters, it could be going home after an accident rather than checking on your bones to see if they're all put together, a bit of a stretch. You'd probably check notice if your arm was broken. But with computers, we might not notice that we broke a feature and cost our employer a few million dollars. I've done that. It was a really awkward one-on-one the next day. Like I updated my resume. I did everything right. But I was really stressed about that. And you don't want to have that experience. There was another time, much less stressful. I just assumed that everything was a coffee shop. And my only test query was the word coffee. And then my boss got upset that we were returning Starbucks for when he was trying to find a steakhouse. And that was an awkward call, but not that awkward because we were a startup and we didn't make any money. So it was just like, man, whatever, it's jokes. And then there's other ones. Some words can have multiple meanings. And those meanings can be really awkward to have to explain to people. And if you have tools which do a good job of keeping minors from seeing inappropriate content, you really don't want those tools to break. And it's really easy to have that happen when your data changes. With money things in America, the Veterans Affairs Agency couldn't pay a whole bunch of people because of data validation pipelines, Bank of America for clothes, sorry, because of not validating data. Bank of America for clothes on a whole bunch of people's mortgages. And like other generally really bad, terrible things have happened to specific people from us not doing a good job of catching data quality issues. So hopefully, this is enough that you care and you'll pay attention. And if none of these problems are things that you care about, it's OK. The internet is working somewhat. So let's do some validation. Yeah. OK, I'm going to take the unicorn horn off now. So another thing that we might want to validate is, are our slurpees drinkable? Do you have the concept of slurpees here? It's an ice drink with more or less sugar and some food coloring and something resembling flavor. And occasionally these machines are at gas stations and maybe not the most well maintained machines. And you might not want to drink something from a machine which is fundamentally just a giant Petri dish when it's not operating correctly. So a good validation rule that we might have for consuming slurpees is that our slurpees should have the food coloring in them. If they don't, something has gone wrong and I probably don't want to get sick. Maybe that could happen. But the new slurpee ghost white gummy flavor is going to break our validation rule because now there is a slurpee flavor without food coloring. And this is OK. I think it's totally fine to have validation rules that break occasionally. And I think this is perhaps different than tests. We really don't want tests to break occasionally. We want our tests to be deterministic. But with validation rules, since they're being used as sort of a second last catch, it's OK if they're sometimes wrong. They can't be so wrong so often that people just start to ignore them. But if they're occasionally wrong once a month and people don't start turning out their alerts, it's OK. So how do we make validation rules? So hopefully, at some point, you've had software that worked. If you do not have any software that works at all, validation is not your problem. It is time to go and fix your software. But you've had your software that works and maybe you can collect some metrics about how our software is working. And then we can look at future iterations of our software and if it's not looking like the previous iterations, maybe that's a thing that we can do something about. And we can also do similar things for our inputs. We can look at do-todays inputs, look similar to yesterday's inputs, or is the rate of change between the days of inputs within reason? Right. OK. Does anyone in the audience have something equivalent to this where you load some data, you try and parse it, but maybe the schema doesn't apply, maybe it's missing a field, maybe something set to null, and whatever, just throw away the bad data. It's OK. We'll just keep the good data in our pipeline. Does anyone do this? Am I just the one bad person here? OK. So a lot of people are raising their hands and they're only half raising them, which makes me think more of you do it, but you don't want to be caught. And this is OK, right? Especially for anyone who has to work with JSON data, if you require that all of your inputs were completely correct, you would never produce a result, right? That's just not going to happen. JSON data is garbage. So we're going to have something like this. And that's OK, but the problem is we might throw away 99% of our data when, historically, we've only been throwing away 1% of our data. And then if that 1% now is no longer a representative sample of my users and I'm training a recommendation model, I might make some really bad recommendations or other kinds of decisions based on this. So we could go ahead and we could write a check. Is it valid? And then we can go ahead and we can count. Is it valid? And we can say, if we have less valid data than bad data, we'll put some special business handling logic in here.  And just do something. And this is technically a validation rule. It's not a very complex one, but it's a good start. And it's better than nothing. And at this point, right, your special validation rule can trigger, apparently, all of you are very lucky and have operations people's pagers. And they'll come and take a look at it and then yell at you the following morning. It's less fun, though, right? Like this code is less nice to write. And that's the thing I care about. In terms of working in Spark, there's some performance problems with doing these two counts. It's a little sad. And it's similar in other systems where essentially triggering two actions can be kind of not great. Beam is different because it has a sort of whole program optimizer to a degree. So if you're in Beam, this last part doesn't apply. But yeah. OK. But we could use counters. Yay. Counters. And the other thing is we don't have to define all of the counters, right? All of these systems keep track of some metrics themselves already. And these metrics can actually have a lot of really useful information that can tell us whether or not our job is operating normally. We can look at the number of bytes that are being shuffled around. We can look at the number of records that are being read. Execution time is super basic. But if you've got a machine learning pipeline that takes three hours to train, and today it takes 15 minutes, you do not want to push that model to production. There is something going on.  But maybe it's time to spot check this one. And we can add counters for things that we ourselves have either had catch on fire or suspect might catch on fire. And we can still pretend it's nice functional code. We can just hide all this mutation and counters just inside of the underlying systems and pretend that we're writing functional programming code. For those of you who care about writing functional programming. And right, yes. So counters solve our problem in the same way that regular expressions can solve our problems in that they produce a new problem for us to solve. But it's different. So it's like they solved their first problem. So we have a happy counter and a sad counter. And we see how sad we are at the end of our job. And if we're really sad, we won't do anything. We'll just go to, like, you don't have Taco Bell. Pizza Hut? Pizza Hut. We'll go to Pizza Hut and we'll get some pizza and we won't push our model to production. We'll take the rest of the day off. And that's better than nothing. So OK, this is still really tightly coupled to our code. And this kind of is less than great. Oh, right.  Some of the problems are that beams counters are implementation dependent. So you can change the runners for beam, but doing that will change the behavior of your counters. So if you've got something that's working locally and start using it on a cluster, it's totally possible that the validation rules that you've been working on locally will just stop working. And that'll be a great experience. Or if you're trying to change runners and you want to use your validation rules to validate that your new runner is performing reasonably, it might just not be and you won't know. And Spark counters have their own problems, too, with data properties. But where do we put our counters? Fundamentally, we have to understand our problem domain. And this is very similar to the problem domain of running a queer scooter club, which is everyone loves glitter and bubbles and scooters, but it turns out that putting a bunch of soap on the road was a bad idea. Who could have foreseen that having a bubble machine would cause accidents during a parade? No one, obviously. But if we spend time to think about our problem domain, we can add some tests like, hey, or sorry, validation rules like the road should not be more slippery than I'm used to. If it is, maybe let's turn off our bubble machine and stop writing for a little bit and think about what's going on. So what do people do in practice for making their validation rules? Really depressingly, I ran a survey and it turns out that the only thing most people validate is execution time and the number of records they read in. They're like, you know what? If it took three hours today and it took three hours yesterday and it read six gigs and today it read seven gigs, that's fine, right? Everything in between, whatever. It's probably the same. It took the same amount of time. How many people think that's enough? What, oh, please tell me you don't work at a bank. Oh dear, oh dear, at least not my bank. Okay, well, whatever, it's not like I have a lot of money. So not a big problem for me anyways. Maybe you'll lose my mortgage, that'd be great. Okay, so Spark Validator. So I did a proof of concept and I actually have a second proof of concept as well. This one is integrated into your SparkJob and I have a second one which integrates into your Airflow Pipeline and then you can just go ahead, put in all of these counters and then you can define historic rules on your counters and say how they should be related from today to yesterday over time. And then it's nice and you can have sort of decoupled stuff. There's input schema validation is really cool. We can write our own input schema validation. We can look at the percentage of data that's changed and there's also, come on. Okay, validation rules can be separate stages. Right, and this is important with the difference between Spark Validator and the second proof of concept that I made. To an extent, we can do data validation in parallel in a separate process. And that's to say, not all of our validation rules have to be about how our program is behaving. It can be about do the summary statistics of our data look similar to today as yesterday because that can be a really good sign that something has gone wrong. And then we can run that in parallel without slowing down our main job, which is really cool too. And in fact, there's a tool to do that that runs on top of Beam. It's called TFDV. Despite being called TensorFlow data validation, you can use it for things other than TensorFlow. It's also open source and you can use it to compute some basic statistics about your data and compare it to the previous statistics and also find anomalies based on your schema. And this is a really useful tool. There's some limitations with the places where you can deploy it right now, but it's open source and you can take the ideas and apply it to your own system. Software changes too. You should, when software changes and your data hasn't changed, run your old software against your new software and see if they look the same. It is really simple and can save you a lot of time. Even when you think that only unrelated changes have happened because your unrelated changes might change your FORTRAN libraries, which, oh dear God, are somehow related to Spark. Okay. And we can put it all together in Airflow, yay. So if you use the second proof of concept that I have, you would put your Spark submit operator as a data validation task and you can define it as dependent on the important business logic task. If you were doing parallel stuff, you wouldn't define the dependency graph that way. You would instead just have it run parallel and then you can use the batch operator to just call the TFDV stuff that we showed you here. And that's all kinds of fun. Some ending notes. You don't have to be perfect. Just do something.  That's okay. It's better than not doing anything. Just start somewhere. Here's some related links. Here's some books. Here's some books. It's unrelated, but that should not stop you from buying this book. I also have a project to teach distributed computing to children. Probably not the children that you like. But the children who don't have your cell phone number, but you still wanna convince them to join us in this wonderful adventure that we call programming. And this is not a joke. It sometimes gets confusing. Okay, so this is, okay, thanks, bye. I think you said there were five minutes at the end for questions, right? Cool, so I've got time for a few questions if anyone has them. Yay, question. So the question is, I've been talking a lot about machine learning being more and more integrated into distributed systems. What about algorithms which are not easily parallelizable? Do I see distributed systems fitting into that? And I think that, yes, even for algorithms which are not easily parallelizable, my friends at a company which will remain nameless, because they know where I live, use distributed systems to do all of their feature prep and they also do down sampling on their large data and they construct representative samples and do their training locally with non-parallelizable algorithms.  In real life, I tend to spend more time doing feature prep than cool machine learning stuff.  Because there is no magic wand of parallelization. If there was, I would make a lot more money. Or less, I don't know actually, if people wouldn't need me. One of the two. Okay, question number two. Oh, yeah. So I should have, sorry, I was rushing because I got the like, you're at a time thing. But I'll come back here. So this data validation task, oh, right, sorry, okay. The question was, I showed an example in Airflow. Why was I doing the validation after the important business logic rather than in parallel? So the Spark Validator example that I have, it looks at the counters that are output by your Spark job to do the validation on it. So it doesn't like look at the data independently. TFDV looks at the data independently. The Spark Validator tool that I made looks at the metrics produced by your Spark job. And that's why I have the dependency chain in the way that I do here. Awesome. Cool, I think I'm probably out of time. Thank you all. Um.


This week I'm starting to feel like a real mumma-ger. I've put the brand development on hold because Luke spoke to me and Winter to do a paid social media post. So me, Ben and Winter are in Manchester to meet Erin, who has sorted a location out. It's for a very glamorous swimwear company. Truth say it wasn't the best swimming pool but with my visuals, the way I visioned it, I thought we could make it pop in and we'll get the post. Being a celebrity isn't all glamour and champagne. Sometimes it's really hard work and if Natasha wants to make a success of this and make a million, she needs to get on with the job in hand. You are having a laugh in him. You are actually having a laugh. It's a different vibe to the usual. Do you know what I mean? I like it, it's nice. Oh my God, oh my God. Just trying to see what I bought to make it glamorous. Peggy the Pegasus. Seriously, what's a Pegasus? Pegasus is one of the best known creatures in Greek mythology. Greek and Roman poets write about his ascent to heaven after his birth and his obesity diseases. What's with all these long names? Why have everyone got such weird names? How is a Greek horse going to improve this location? Like seriously? Look at matches. I've got your bikinis ready. It's matching. Good. That is stunning. And it matches. Mine is really nice. Oh my God. Luckily the bikinis are actually stunning and they look lush on. This might not be my dream location but I have to remember I'm doing this for winter. To be honest, I can't see us getting another job at the back of this job but I'm happy. We've made 500 quid. That's like a huge supply of nappies. At least we've got something good out of it.


I just wanted to show you how we're defining gender ambition at IUD, so hopefully you now have a new slide that we can refer to. So at the very minimum, we're talking about gender sensitive or gender aware projects. These are projects that acknowledge that gender plays a role in specific development outcomes of a project that it identifies women as a stakeholder group. So gender sensitive or aware projects would aim to ensure the participation of both men and women in its activities and it should disaggregate data for biceps. That's what we would call a gender sensitive or aware project at IUD. A gender responsive project would aim to promote equitable sharing of benefits between women and men. It would recognize how project implementation may be influenced by the differences between men and women, but it does not address gender relations or underlying societal or power dynamics. That is the realm of gender transformation or gender transformative projects, which we see at IUD as a project that goes beyond promoting equitable sharing of benefits that actively seeking to shift power dynamics by promoting shared power, control of resources and genuine decision-making power. In doing so, the project should seek to understand and address societal dynamics that are gender biased and other structural constraints to gender equality. And on your screen you'll see the three dimensions of equity. There we see a truly gender-trimmed, transformative project as a drafting for three dimensions of equity.