Seriously, I'm melting.  Excuse all the bugs. First time coming home to the new shop. Haven't organized anything yet, but no time for that. I just had to print off all of my tax papers because I still haven't handed in my tax papers from quarter one. So I'm a little bit behind. It's pretty much the end of quarter two already. So we've got to get those paper, all that paperwork together too. And send it away to the tax people, AKA mom, she runs an accounting business. It's very nice having an accountant for a mother. I still need to learn how to file all of my own taxes. I'm pretty sure it's easy, but she's always done it for me. So one of these times I'm gonna sit down with her, she's gonna teach me how to do it. But she is my accountant. So I gotta get all that paperwork to her. Old blue. You take care, okay? Your new home, nice and cozy. You be good, all right? Good morning, everybody. Afternoon, it's already afternoon, isn't it? We're here in the shop and old blue is sitting here. Gotta give her a rinse down probably tomorrow or before we leave. It's dishwasher install day, installation. And dishwasher installation day. So I gotta get this thing working for my wife so that this week she has the dishwasher she can use because we don't have one in our house. And there's a lot of dishes to do when you have a baby. And we just have one. I mean, eventually we're hoping to have two, three or four maybe. So there's gonna be lots more dishes to come. We're probably not gonna be in this house much, much longer. Like by the time we have three kids, we'll be in a new house already. But we'll be here for another year or two anyways. And it's a lot of dishes to do by hand all the time. And figure, you know what? Let's get a dishwasher installed in there. And then we're gonna redo the countertops. I've actually got the countertops over here. That is not a project for today. This is a project for, who knows when, got three of these here. And we're gonna be redoing the countertops with these.  Never done countertops before, but hey, I got signed up for a project I'm gonna deliver. I'll figure it out. I'll get my dad to help me or a couple of my uncles, they've done finished carpentry as a career for decades. So I can always give them a call and get some pointers and exactly how I should get it done. So I have the resources. We'll be okay. I came here for a reason though. What did I come here for? Why are we at the shop? Yes, screws and washers. So what I'm gonna do since we don't have the countertops over top yet, the dishwasher is gonna be standing free for now or beside the old counter. I gotta build a new one over top of it. So that we can use the dishwasher now in the meantime while we're waiting for me to figure out how to do countertops. I'm gonna secure the dishwasher to the floor, to the subfloor so that it can be used and stayed. So I gotta get screws and washers for that here right now. And I've already got the drain pretty much all set up at home. I just gotta hook up the hot water and secure it to the floor and turn her on. Go on, hook it up to the electrical. But that's pretty much all done as well too. That was the part that I was a little bit most worried about, but we got that figured out. I haven't been able to get in there. But at least I knew where everything was there. Now that I've moved everything into this shop, I sort of rushed it. I just wanted to get everything in here, get it done. Cause I didn't have a lot of time, right? So I just, I piled everything up and I sort of know generally where everything is. And I just piled it all in here and I haven't had a chance to go through it all yet. And it shows, I have no idea where my screws are. I'm looking for specific screws where I can screw it down into the subfloor. I know I have these and I looked at them. I'm like, oh, those will be perfect for what I need them for. Let's remember where those ones are. No idea where they are. No idea, they're in here somewhere. Okay. Here's my floor tape. I know where that is. I left that right out in the open. So I can put new lines on the floor. So I back all blue in here straight. So the important things we're taking care of, you know? Make sure you're straight. I'll find it. I'll find them, yeah. Later on, here's some. I gotta, oh, one of these days. Everything's so busy right now, you know? I've got a baby and everything's, it is. So at least everything's in here. Thought I'd put it here somewhere. Hmm. Well, this is embarrassing. I shouldn't even be showing this to you. But whatever, I'm a vlogger. I'm a little bit too honest with you. Look at how messy and unorganized I am. Drives me nuts. I like everything to be in its place, organized, neat, and tidy. Is it ever that way? No. Not even close. Even after I organize it, it still doesn't feel like it's organized. Where are you? Where did I put these things? Oh, an old blue, you need a bath, too. I'm so sorry to leave you here like this overnight.  So sorry. Yeah, almost dropped you guys.  I have no idea. Oh, I'm gonna keep searching for a bit here, and I'll find them. I'll find them, no worry. No idea what I'm doing. No idea what I'm doing. But it's amazing what you can learn off the internet, right? It's very straightforward, very straightforward. I can't mess this up. It's impossible. You hook the hot water line up to the dishwasher, done. Hook the drain up to the drain under the sink, done. Hook the electric up. It's the hardest part. That's pretty much done. Now all I really need to do is secure it and turn it on and get the approval from my wife, obviously. The coveted approval, the wife approval. Without that, I may as well just come back here and sleep in here. I'm just kidding. She's not like that. Everybody learned somewhere. Everybody did whatever they're good at for the first time. Sometime. They're the first for everything. I come to you from the future. So today's video was actually Friday and Saturday that I sort of put together here because I didn't film much. I was very busy when I was at home. But yeah, you saw what was going on there. We got home. We got old blue in the shop. Had no time to organize the shop. But I did get that dishwasher installed. I learned how to do something new. Thank you, YouTube, for explaining it. It was actually quite simple. There was a lot involved in it. I had to cut the drain pipe from the sink and install what they call a Y connection in there so that the drain from the dishwasher could be stuck on there and it drains down the sink. I had to connect a T connection onto my hot water line for my water under the sink. Wrote one to the dishwasher and one to the hot water on the sink. Got that worked out. And then I had to hook up the electric. That was a bit tricky, but we got it done. So the dishwasher was working just great. I'm sorry I didn't have any footage to prove that to you. I'll have to show you next time when I'm at home. It took me a little while to get done and I didn't film myself while doing it because when I film things, it takes a lot longer to do things, especially when I'm learning something for the first time. I'm not always comfortable like setting up a camera, filming everything. I just want to make sure I get it done right. Half the time I'm like looking at my phone, trying to learn the next step. The other half I'm trying to figure it out. It just took, it took all day. I know an experienced plumber could have gotten it installed in probably like two hours. But I wanted to make sure it was done right. I had to run to the store to get a few more parts that I had forgot. I'll show you when I get home next time. I was just exhausted by the time it was done. I was frustrated. I was glad it was finished. I had secured it to the floor. Yeah, and one of you said, make sure you screw it down in the back as well. I absolutely did do that because obviously when you open up the dishwasher door, when you have dishes on there and you roll it out, you don't want it to tip forward onto you, right? I gave it a good test run. It worked well. And since I've been on the road here, this is Future Josh. Britt has been using it at home and it's been working great. So it was a great success. I'm happy with that. I learned, like I said, learn how to do something new. Next time someone needs a dishwasher installed, hire a plumber. I'm busy. I know how to do it. But thanks for hanging out with me. This was sort of just like a weekend update. I went and picked up that load in Canora and brought it back. That we're going to be delivering in a couple of days. Obviously they're not open on the weekend so I have to go there on Monday. Yeah, by the time I'm filming this, I've already long been and gone.  There's a bunch of reasons why, but it does help me keep up with everything a little bit easier and I like it better this way. But we're not too far behind. Anyways, thanks for watching today, everybody. I appreciate that. If you mind doing me a favor, go down below the video that you're watching right now. Make sure you're subscribed on YouTube. Hit that like button. Leave me a comment down below. And next time I'm home, I'll show you my handiwork. I wish I would have filmed it. I just didn't think of it. By the time I was done, I was like, I don't want anything more to do with this thing. This thing took up the whole day. I want to go sit down and hold my baby. That's all I was thinking pretty much. Hold my baby, watch a movie, and then go to bed. But on the topic of the baby, he slept great through the weekend. So when I go home, or when I went home this last weekend, I'm on baby duty since Britt takes care of the majority and all of the responsibilities regarding Theo when I'm at work. When I come home, I want to try and give her a break. So I'll let her sleep as long as she wants. I'll wake up with him throughout the night. He actually did really good. He slept for four and a half hours, both nights, twice. So we went to bed at like, by the time we got to bed, I think I was like midnight. He slept until 4.30. I got up at about 4.30, 45 minutes. He was changed, fed, sleeping back in his bassinet, went back to bed and from 4.30, that would have been four, five, 15-ish. He slept until like 9.30, 10 o'clock after that again. So I actually got a really good sleep both nights. I got lucky. Sometimes he's a little bit more fussy than that, but it seems like night time is getting a little bit better with him and we're learning as we go. New parents, it's our firstborn. And we're not planning on stopping at just one. We want to have more. So no news on that front yet, other than we have begun the process of trying to do what you, trying, doing what you do to bring another life into this world. We have another egg or another embryo that we can use yet from our last IVF and Britt is willing to go for another egg or trailer yet. We'll see how that goes in the future, but for now, we're so happy with Theo. He's been such a blessing and perfect health. Nothing wrong, it all went perfect. She had a little bit of preeclampsia during pregnancy, which sort of went unchecked by the doctors in Steinbach. That's a whole story for a whole video on its own, but when we went to Winnipeg, they caught it pretty quickly and that's why we had an early C-section. Maybe I should explain that in another video sometime. That's a story I would like to tell for other expected parents, expectant parents because it's an issue we brought up with the doctors again and again and again, Britt had really high blood pressure, really high blood pressure and we were very concerned and the doctors kept brushing it off and finally we went to see another doctor in Steinbach in Winnipeg, it's a Boniface and as soon as that doctor found out the situation and saw the numbers, she's like, you're having this baby today. Like we need to do an emergency C-section right now. So it was a little bit of a scary time when he was born, but everything turned out great and we're as happy as can be. Can't wait to get home. I miss them so much when I'm on the road, but you know, I'm out here making sure that we have a roof over our head, making sure we have food on the table and I have to keep reminding myself of that because otherwise I get into the part of my brain which is like, why am I, why am I here all by myself? It gets lonely. It's a good thing I love my job. I really do love my job, but I love my family a lot too. I love my son, it's still really cool saying that my son, I have a son and I miss my wife and our dogs, but man's gotta do what a man's gotta do. You know, we gotta eat. The prices of things are skyrocketing, crazy, out of control, so you gotta keep working harder and harder, make ends meet, but we're doing okay. We have a, like I said, we live in a tiny small house. We have a good budget. We wanna get into a bigger house eventually, but with interest rates constantly going up, with inflation going up, with the government printing money, like it's going out of style, I'm a little concerned about what the future's gonna bring, and I don't wanna get my family in a position where we buy a house that we're happy in and then all some boom, interest rates go up when we renew in five years from now, and then suddenly, wow, we can't afford our house and we're bankrupt. That's a real possibility for us and for a lot of Canadians. Buying houses is very expensive now, and I don't wanna put us in a situation where we could be bankrupted when our mortgage goes up for renewal and interest rates have skyrocketed. I really wanna see that the government is able to bring inflation and interest rates down and stabilize them before we make any big moves towards the house, another house. Like we own the house right now, but it's small. We're gonna need a bigger house eventually. And I'm just hoping that by the time we really, really need a house that we'll be ready with a huge down payment because we're taking this opportunity to save up a bigger, bigger, bigger down payment. And I'm hoping that by the time we're ready, that the government will have figured things out and gotten a hold of everything and that the economy and everything in the financial sector and everything will have somewhat balanced out and just come down a little bit because right now it just looks like things are gonna get worse. For eight years now, things have just been getting worse and worse and worse. So, you know, hopefully we'll have a change soon and it'll start going in a different direction and we can feel more comfortable buying a house. It's the last thing I want us to buy a house to be happy for five years and then all some bankrupt. Can't afford the new interest rates. So gotta be careful. Anyways, thanks for joining us today. These are all topics I could put in all of their own videos. I'm rambling on here, I know. But hey, we're back in the truck in a couple of days. The next video will probably be released tomorrow so I'll see you in my next video tomorrow. Thanks for watching everyone and take care, be safe out there.


Hello everyone. Thanks for joining me this afternoon. My name is Steven Gordon. I'm a senior technical product manager at Red Hat primarily for OpenStack Compute and its interaction with related technologies in the underlying hypervisor. Today I'm going to be talking specifically about the features in the compute service for moving instances around an OpenStack cloud and primarily some of the differences between them from a user-facing point of view and also some of the things that have been done in Liberty to improve these facilities and also some of the things planned and being discussed from Ataka at the moment. So in terms of that I'll be focusing first on defining what we're actually moving. So you know the spoiler alert at the bottom there is instances so compute instances we're moving around but what does that really mean in terms of the components of those virtual machines? Why we're moving them? How we're moving them? So in terms of those user-facing APIs to actually initiate the moves and then also the new enhancements that are coming both in the current and upcoming release. So in terms of what we're moving if we talk about our instance or our server as it's referred to in the NOVER API we have our guest configuration which can be the simpler or more obvious things like how many CPUs, how much RAM is associated with the guest but also all the way down to the device profiles that are actually exposed by the hypervisor. So today in particular I'll be focusing largely on the libvert slash KVM driver and the way migration works with it and it includes as part of the guest configuration a lot of different things and options for things like the disk device drivers that you should be using and so on. There's also the guest storage primarily here I'm talking about the initial image or volume from which the guest was booted and I separate that out from the disk part of the guest state so you're a ephemeral storage. So in terms of guest state we're talking both about the in-memory state of the guest but also what's stored on that ephemeral disk. All of the paths we have for moving instances around a compute deployment involve moving some subset of these elements but which mechanism you use for moving instances will vary depending on which of these elements you care about so you may for example not necessarily care too much about the on-disk state for ephemeral or the in-memory state if you have particularly cloud-ready applications and you may be willing to throw that away but you just want to know that the instance is going to come back up somewhere else in the initial state that you had in the guest in the image. So in terms of why we're moving instances so in the open stack world at least all of the APIs are going to talk about are marked as admin APIs in open stack parlance so that means they're not necessarily exposed to normal users but they are available to the tenant administrators and that that becomes a little bit important shortly because even though they're only exposed at the tenant admin level it still exposes in the current form back-end details to that tenant administrator that they shouldn't necessarily have to know so why might we might might we be performing these moves so we may be doing them proactively so in advance of node maintenance so that maybe we're adding or removing hardware or we know based on our monitoring tools that there's an imminent hardware failure coming so we might preemptively try and move our instances off of that host it may be reactive in terms of the node has already failed for some reason it's lost power or there has been a hardware failure I'm also also for potentially capacity management reasons so in the way the schedule works in Nova is we basically place once and then once the instance is running somewhere there's not really a concept of dynamic rescheduling in reaction to the way capacity needs may change or something like a noisy neighbor effect so the admin of the tenant may want to move instances around either to spread them out or consolidate depending on what their goals are with their open stack deployment so getting into the meat of it and how we're moving instances so I mentioned there are a number of different API's in Nova for achieving a similar goal and that is in fact part of the reason for the confusion which I guess it spurred me to do this talk in the first place so even in the development community this is little understood in terms of the API documentation around this isn't great and we provided a number of similar things that were exposing to the user back-end details that they probably shouldn't have to know about how they move is actually happening so if we look in the command line help we see that there's seven different actions in the Nova command line to client that relate to moving instances one of those is the list so we kind of exclude that but that still leaves us with six different ways we could move an instance from one host to another potentially so the the issues with this range from minor knits like the fact that we use migration rather than migrate when we're talking about the live migration down to the fact that we include servers in the API call in one case but not the rest or even in the last example the case where we conflate live migration with evacuation which are actually completely different things and the user looking at this doesn't necessarily know whether you're doing an evacuated or live migration when you do a host evacuate live and we'll get to that in a moment. So trying to step back a little bit the primary three mechanisms evacuate migrate which is sometimes called cold migrate as a way of trying to be a little bit more specific about this or live migration so evacuate specifically rebuilds an instance that is currently on a compute node that is down on a different compute node that distinction is important because in many ways it acts in the same way as migrate but migrate only works when the source host is up. It's a little bit more nuanced than that and we'll get into that in a second in that by rebuild in the migrate case we really mean resize so the migrate command is in and of itself a path through to resize which is originally designed in the Nova API for resizing an instance between two flavors and coincidentally as part of that operation it puts it on a different host so migrate is abusing that functionality in some ways and it also results in some other oddities which we'll see in an example in the moment. With live migration that's the case where we're moving an instance keeping all of its states and not just the undisstate but also the in memory state and trying to do it with as little downtime as possible to the point where that downtime should be unnoticeable to the guest operating system applications. In terms of the other commands that are listed so host evacuate host servers migrate and host evacuate live these are all actually helpers and don't necessarily map one-to-one to a Nova API call at the back end either so some of them are actually implemented in the client so host evacuate does a rebuild of all of the instances on the specified host and puts them on a new compute node. Host servers migrate does the same with the migrate command so again that distinction between whether we're talking about a host that is down or a host that is up and then finally host evacuate live doesn't actually do an evacuation but live migrates all of those instances on that host to a new place. So drilling down on each of these individually when we talk about evacuation so we have the Nova evacuate command we have optional password and on-shared storage arguments and then the mandatory service of the instance name or ID that we're trying to try to move and optionally a target host. So as I mentioned evacuation only works when the compute node hosting the instance is down or recognized as down by Nova more importantly and it will rebuild the instance on a new compute node. The main so obviously when we're doing this rebuild the node is down so we can't copy for example the in-memory state or even if we're not using shared storage the ephemeral disk state to the new location but there is still some benefit over just starting completely afresh and that you do get to keep the unique ID and the IP and a couple of other details about the instance it's the same when you use this mechanism so that's the benefit you get there. If we are on shared storage so if we specify that flag we can also get the ephemeral disk across as well. The one other thing is because we are doing a rebuild there is the opportunity to inject a new admin user password at this point if we don't specify one it will be it will be randomly generated for us anyway so in the example here I'm just doing a Nova evacuate without shared storage you can see it generates me a new password and prints that out on the command line for me. The one other thing I should should mention is evacuation does allow us to bypass the scheduler so if you don't specify a target host the scheduler is going to pick it for you which is the default behavior but if we specify a target host we are bypassing the schedule completely and that's one of the reasons this is an administrative API it's not exposed to the normal user. When we talk about cold migration it's a little bit different so it doesn't have nearly as many options available and part of the reason for that is because as I mentioned before it's actually going through the resize API behind the scenes and that resize API is actually available to normal users so that's why for example it doesn't allow bypassing the scheduler and it doesn't necessarily require you to know what storage is involved either. So as I mentioned before it only works when the compute node hosting the instance is up or still up and it rebuilds on a new host selected by the scheduler so that involves actually shutting down the instance copying the disk and then starting the instance on the new hypervisor and after it's successfully done that it'll also remove from the original hypervisor. One of the weird things about it is because we're using the resize path the resize API call has a manual confirmation step so someone has to manually confirm that the resize worked before the instance will go back to its normal operational state. The same applies to migrate because it's going down that path which is a little bit of an oddity in the current way it's implemented. So in the shared storage case migrate doesn't actually know that so it will do the copy anyway which is problematic in and of itself so one of the weird things about so this is where the I guess information that a tenant admin shouldn't necessarily have to know is filtering up through the API is that NOVA is not make but not even trying to make a determination in these cases at least as to whether it's on shared storage or not. So for that reason that tenant admin trying to initiate this command or not the specific command with the others has to know and tell it and that's one of the problems with the current implementation that we've been discussing a little bit because as a user of the cloud obviously you shouldn't necessarily need to know or care what back-end storage is going there even even as a tenant admin you know because you may just be administrating for your particular department or whatever it happens to be. So in the cold migration state here I used the poll option and that ticker goes gradually from 0 to 100 percent as the migration completes I do my NOVA list and you know oh that's weird I have this verify resize status rather than just up which is what you might expect running this for the first time. So then from the command line I'm just running that quick resize confirm it is also available from the horizon dashboard so if the instance is in that verify resize state there'll be an extra button or option there to do that as well. So I'm going to move on now to live migration which is I'm probably what I'm going to spend the vast majority of the rest of the time talking about both in terms of the fact that it is I guess one of the one of the ways of moving instances that people find more interesting but also that because of what it's trying to do it has more complex prerequisite requirements to actually get it working. So live migration as I mentioned before moves the virtual machine from one host to another without any noticeable downtime we'll get to what noticeable means in a moment but I say that to highlight the fact that there is technically actually a brief outage as we're completing the copy. So there are two approaches to live migration supported both at the QIMU slash Libvert layer but also via OpenStack Nova. So there's using shared storage and I include in that volume based so that means that either you're using shared storage to back up the Nova instance or share the Nova instances path on each of your hypervisors so that any individual hypervisor can access that's those same set of disk images or alternatively if you're using boot from volume then you effectively have shared storage in that case being supplied by Cinder. We still obviously have the need to sync the memory state while we're doing this but we've obligated the need to copy across the disk storage because that's already there. The other alternative is using block migration which in previous or older versions of QIMU was kind of a little bit shaky. It has a completely new implementation and newer versions of QIMU. Quite a few people use it and it does a direct transfer of not just the memory state but also the disks. So the trade-off you're making here is that it does take longer you're transferring a lot more data between the two nodes but on the flip side some operators prefer this approach because either or they don't use migration that often and they don't want the overhead either performance or administration wise with running shared storage if they only need it for these occasional maintenance events. So that's that's kind of the trade-off that you're making there. Shared storage migration will typically be much quicker but it does have that extra overhead to actually have it set up and working. So in terms of how it works kind of at a high level. So initially the Nova scheduler selects a destination host although again I should mention that like some of the other commands you can actually override it by specifying a target host. I should also mention what I'm saying that that again here if you're using block migration you actually have to specify that so the default is to try and do shared storage migration. So the scheduler selects a destination host unless you specified one yourself in which case again it bypasses a scheduler. Weirdly though in this particular case and this is an oddity that's probably worth highlighting it will come up again in the end when we talk about issues that we're still working on. There are additional checks done in the Libbert driver on both the source and destination host on disk, RAM, CPU model and also the mapped volumes that may or may not be connected to the instance. RAM in particular causes some issues at the moment so for those familiar with the concept of over committing memory by default or in the default configuration Nova over commits memory by 16 to 1 operators tend to change that substantially but regardless a lot of people are using OpenStack with over commit of some level enabled. This calculation is happening on the compute node and over commit is a scheduler side setting so when we do this calculation on the compute node at the moment we're not factoring an over commit at all so that means that although by your over commit calculations the destination host should have enough space to take the instance it may actually fail anyway that's an issue we're currently working to resolve. The mapped volumes comes up because when we're using or when we have an instance at the moment with kind of a mixed storage model so where we've used an image to boot it and then we have attached volumes associated with it the migration of those mapped volumes will currently try and copy them over themselves also not good so that's again on the list of things that we're trying to resolve that I'll get to particularly for the metaka piece of the discussion. So anyway assuming that our source and destination host checked out we move into stage three which is what we call the iterative pre-copy so what this means is we start copying memory pages from the active virtual machine to a new virtual machine that's in a paused state that we create on the destination. Obviously while we're doing that our source virtual machine or the virtual machine on the source host is still running still dirtying pages as it accesses memory so while we're doing this we take one big block copied across keep going and then eventually when we get to what we think is the end we take another look and we say okay we have more dirty pages we keep copying and gradually the idea is that the delta should get smaller and smaller to the point where QMU calculates that okay given the transfer rates I'm getting I'm going to be able to copy all of these remaining dirty pages in one step and that's when we pause the source VM copy that last step typically in a matter of milliseconds and then fire it up on the new host and then finally once that's worked we clean up the source. In terms of gotchas or how it doesn't work I mean I mentioned a couple of things where we went through there one of the big things that causes a lot of questions so I'm fairly active on ask.openstack.org for example and there's kind of like two questions two nova questions that are just recurring and this is one of them the other one is why did I get a no valid host error gets a lot of free calmer points for answering those so in terms of the CPU mode or model compatibility we have this idea in at least the liver and KVM based hypervisor of exposing to the guests some virtual CPU which may or may not match what's on the physical box so the maximum performance we get is by exposing every single CPU feature that exists on the physical CPU die the downside however is that live migration is going to fail if on the source and destination host we don't have that same exact set of CPU features and the problem here is that even within certain named model ranges of CPUs which you know as a consumer of the of the hardware you would expect to match exactly occasionally there are changes between them which can cause this problem so effectively what you have to make is a performance versus flexibility trade off so you want to determine how much of that performance you want to trade off for the ability of increasing your effective live migration domain or the size of your live migration cluster so where you can live migrate to from any given host so Nova provides exposes some configuration keys to influence this choice a little bit so the host pass through option is kind of what I talked about first the idea of just exposing through every single feature that's available for maximum performance but minimal ability to migrate basically alive my great I should specify in particular we have the idea of host model so where we can pick from anyone of a number of host models that are already known to live in QMU predefined so we can use one of those and those kind of best fit for that sorry what I was referring to there is actually custom so I'll go back a step so host pass through is all the way through all of the features host model is an approximation of the host CPU model so what I mean by that is it'll take a look at the the physical features being exposed compared to the list QMU knows about and do an approximation which is kind of generally relevant to the lowest common denominator within that model and that allows you migration at least if all of your CPUs in your cluster are based off say sandy bridge then you can migrate between all of those now where you can hit trouble obviously is where you have heterogeneous hardware then maybe they're not maybe you have some west mere some sandy bridge some something else and you have to find some lowest common denominator for those if you want to migrate between all of them and that's where custom comes in so this is where we have the ability to actually specify a specific CPU model that we want to use so what one of the example questions was came up with someone was trying to use a I3 at six box or I686 box and a XX664 in the same cluster and to get that you basically have to go down to like a Pentium 2 so that's probably a little bit of an outrageous example but that's the kind of thing where you really have to trade off the performance if you want to migrate between these machines that are you know pretty vastly different different although even if they're both you know x86 architectures so to find out what models are available we can use verse or QMU KVM CPU help just to get that list and then at the bottom there we just drop that's pretty much straight into our Nova config in terms of other ways to fail I'm not going to go into great detail on these so there have been a couple of good presentations particularly in Vancouver covering a lot of the details about how to set things up to avoid these specifically so I have that linked on the reference slide at the end for people who want to dig into those if they missed those but similar to CPU models we have a concept of machine types in QMU which effectively defines what hardware we're exposing to the guest the machine type in use has to be available both on the source and destination hypervisor typically the distributor if you're getting packages from one will deal with that for you it's mainly an issue mainly becomes an issue when we're trying to upgrade between distribution releases and need to migrate to do a rolling upgrade inconsistent network in configuration so but the source and destination destination hypervisor have to be able to talk to each other on the specified network in inconsistent clocks can cause a lot of issues which trying to sync this up as we go through VNC listening addresses so when we if we're too specific in the way we're binding VNC on the source host and it may not work when we move to the destination host and we may not actually be able to connect via VNC and if we're doing security or a secure live migration so using SSH tunneling we need to make sure that that's set up correctly and we've distributed keys correctly in terms of other operator issues which are maybe a little bit more open stack specific so that the list of issues on the previous slide is pretty general to if you're doing migration with the Burton KVM those are things you need to be aware of as a deployer but open stack specifically so migrations can take too long or fail to complete so if we think about what we talked about before we're basically at the whim of how big the virtual machine is and how active it is in terms of whether we're going to be able to migrate or not if we can't keep up in terms of network throughput and we're not doing some tuning around the edge to try and compensate for that then we're not going to get anywhere until very recently Nova wasn't doing any of that tuning so I'll talk about that some more in a moment we also need to use verse bypassing Nova to do a lot of things so you can't for example by the Nova API throttle migrations cancel migrations monitor how they're going so there's very little to no feedback tune the migration max downtime so what that means is the finalization step I talked about where we pause the source VM to copy those final few dirty pages across and restart on the new host the there's a value associated with that called the max downtime and that's what QMU is evaluating against when it makes that calculation of can I finish the migration so it's looking can I finish the migration in this time if you have a particularly large virtual machine you may want to extend that time a little to give you a higher chance of actually finalizing the migration so Nova hasn't been tuning that up until this point so that's something we've been missing out on there are certain instance configurations that can't be migrated at the moment so mixed storage is probably the most glaring one which I touched on before so if you're mixing image and volume based storage if you have a config drive attached all these things are currently not going to live migrate that's something we have to work on and also the use of pass-through devices so this is less likely to go away anytime soon but if you're using physical SROV cards GPU pass through anything like that then at the moment there's no live migration available with that as either there are some things we can potentially do in the future with SROV particularly if you're using a Mac V tap device there's a performance trade-off with that as well so in general we can probably assume that for pass-through devices live migration is going to be not in the question anytime soon I mentioned live migration doesn't correctly account for over commit so that's also an issue for operators currently effectively you know if they're doing a lot of migration it really impacts them in terms of the memory utilization of their cluster and then also I mentioned the tenant admin currently needs to know shared or block storage is available so what have we done about that in terms of the Liberty release so long-running live live migrations so I mentioned the factors involved here are the amount of guest RAM speed with which that guest RAM is being used or dirtied and the speed of the migration network so we were using a fixed maximum down time with QMU as of Liberty we have the ability to scale up the downtime to allow a better chance of completing live migration we also have a limit on the number of outbound live migrations that are going at once so if you think about if I have a host and I have say three live migrations all going outbound at the same time they're all using you know approximating a little bit you know 33% of the network traffic give or take theoretically by doing that I'm limiting the chance or decreasing the chance that any one of those three is going to finish because they're all getting less network traffic than they would if I did one in time so that's what we're getting at there kind of it was related work which you know is indirectly related to migrations we're also now limiting the number of inbound build requests that a single host will try and take on so the default for that is a little bit higher talk about that more in a moment and also you know just talking about all these changes when combined the idea is that we're maximizing the chance that finalization step will actually finish particularly for large VMs we also have a lot of new configuration keys to influence this behavior because it can depend a lot you know on how much throughput you're getting on your host what type of network cars are using and so on and how frequently using live migration for that matter how you want these things to work so in terms of the scaling steps so we have the live migration downtime so that is that maximum length of time for the finalization step and that's effectively saying what's the maximum amount of time that the VM can be paused to complete migration on the source the number of live migration downtime steps so we take incremental steps to reach that value so we don't initially set the downtime to the maximum allowable we try and start optimistically and then gradually work back to that more pessimistic value which is obviously going to be longer finally the live migration downtime delay so the time to wait before we increment through those steps and all three of those work in unison to handle the way we scale that downtime value and I'll walk through an example in a moment because the way those combine is kind of a little bit non-obvious to explain in terms of timeouts so we also have some overall totals so these are kind of outside of the scaling process but how long can I allow our live migration to go before I assume it's not going to finish so we now have a value to set that time out value it is scaled by the amount of guest RAM just in number of gigabytes so if I set live migration completion timeout to 800 for example which is the default that's actually multiplied by the amount of guest RAM to get to the value that's actually going to be used in terms of live migration progress and the timeout there so what we're saying is if we don't see progress in terms of the data being copied for a certain amount of time a default 150 seconds then at that point we again we assume live migration has failed catastrophically and we try and clean up in terms of concurrent operations so I mentioned you can control the number of max concurrent live migrations we default to one which is pretty pessimistic but you know it's recommended that if operators want to run concurrent live migrations then they test that first before they increase that and max concurrent builds and new instances being built on a single hypervisor with a default of 10 so in terms of the stepping example so here I'm using a 400 millisecond max so again that's that maximum amount of time for the completion step with a VM on the sources paused 10 steps a 30 second delay between each step and a 3 gigabyte guest so first of all the delay between the steps is set to 30 by the number of gigabytes of RAM in this case 3 so you can see that I go from 0 to 90 to 180 increments and we scale exponentially across those steps until we hit 400 so you can see a start off with 37 milliseconds 38 milliseconds and then gradually we go up all the way to 400 milliseconds and it's a little bit hard to read on the chart here but the blue line going up on the diagonal is the delay and the red line towards the bottom is the max downtime as we go up and try and give that guest a chance to finalize so what we're effectively saying is that at the end of the day if the guest can't be fine or the transfer can't be finalized in 400 milliseconds then it's going to fail and we fail out the migration when that happens but we're giving those larger guests the best opportunity we can to actually finish the live migration one other addition in liberty which is relevant more to evacuate the live migrate so we have the ability in liberty to report from external tools into Nova that a host is down so the reason we might want to do that is that as I alluded to when discussing evacuate Nova is not going to immediately notice that a host has gone down it does it virus here is a periodic task which means you can depending on how you set things up you know host failure detection can be kind of one to two minutes for Nova to pick that up in the database we have external tools like pacemaker which are much faster at picking these things up and have existed for a long time for that matter so there's now the ability for those to call into Nova and say hey this host is down which means Nova then puts it in the down status and then we can immediately initiate and evacuate at that point rather than having to wait that extra piece of time in terms of metaka and beyond for that matter so that discussion is kind of obviously going on this week but going into the summit there's a group of people in the Nova community coalescing together to work on life migration and their issues related to that and there's an etherpad associated with that and I think in the session after this we're going to be discussing this potentially in the unconference session but in the short term so CI coverage so it's only been fairly recently that we've had a job somewhere in the open-stack infra that can do multiple hosts which is obviously a prerequisite to being able to test in the CI infrastructure doing a live migration or any kind of migration between those so in terms of CI coverage the goal is to get that job voting but also to expand the test coverage to ensure that not just the basic parts that being covered but also things like boot from volume etc so we have full coverage to agree on and improve the API documentation so one of the big problems in this area and one of the reasons that I wanted to talk about the differences between evacuate migrate and live migrate from the API perspective in the first place is that there is very little documentation on what those differences are available outside of the code base basically so within the development community there's a desire to agree on what these are and improve the documentation of what currently is before we try and move to what we might want to do in the future. Support for migrating instances with mixed storage and in particular I think the config drive case is the big one that we really want to get fixed as soon as possible so that you can have an instance with a config drive attached and actually migrate it and then eventually have an instance that has volumes associated do the same. Support for pausing and potentially canceling migrations. Pausing in particular is a very helpful one if we can implement it. The reason for that is it gives the admin the opportunity to effectively stop the guest dirtying pages and based on our discussion about how live migration works if they pause then obviously it makes that finalization step you know much easier to complete you're no longer constantly putting yourself behind so if you have a really big VM and it's taking a really long time to migrate it gives the admin the option of pausing that VM to make sure they actually get it where they want it to go. Better resource tracking in this area in general is needed so we have a number of kind of edgy and race type conditions that can cause a long-running live migration to fail for various reasons. Using libvert storage pools instead of SSH for the migrate or resize case so that's an enabler for migrating suspended instances in particular so at the moment if you have a suspended instance you can't actually migrate it you have to bring it up and then migrate it which is a little bit strange because from a theoretical perspective it should actually be easier to migrate a lot of suspended but anyway and as I mentioned correcting the memory over commit situations the way that's tested when we're doing the source and destination will a migration work check. Medium to long-term things like not just using the current tunneling system for securing live migration but also doing TLS encryption as there's work underway in QAMU around that which is obviously a gating factor for this to actually be done. Auto-convergence so this is semi-related to some of the concepts we talked about but there's also this way through auto-convergence to adjust effectively the amount of CPU the guest is getting so not having to do a complete pause but just effectively giving it less cycles to slow the speed with which it can dirty pages and therefore again make that finalization step easier to complete. And finally post-copy migration so the idea of actually kind of flipping the way we do live migration on its head and instead of putting the destination instance on a host and putting in pause state actually bringing it up straight away and then copying the memory across as it's accessed which is a little bit of an inception type idea but something that's being looked at as well. So at that point I'm pretty much done so I can take questions but just in terms of some general information stuff so the slides I've already uploaded on SlideShare I believe informed you can submit anonymized feedback or abuse using the Summit app and for pictures of cats with stuff on their head etc I think you still have to use Twitter and email so those there as well. For people who want to get involved there is that etherpad there at the bottom and in terms of references I mentioned that there have been a couple other talks in this area there's a long list of bugs in a Google Doc spreadsheet etc so there is a references slide that is going to be on the SlideShare page as well if you had there so that's a lot available so questions. Yes basically well I think one of the things that's actually been discussed at the moment on the mailing list as more and more developers realize that themselves realize that evacuate and migrate are different things is actually trying to at least condense this client side of all this if not the API side and make like a generic move API because the users shouldn't necessarily care. I think for the most part I don't see the SSH requirement going away because for most real use cases people actually want some kind of encryption over it. Nope, taking that as a no. Migration might fail and still leave me with the most VM. So you mean the source VM? Not many effectively so it will in the cases where the VM is still up so live migration it should stay there. Where it gets complicated is primarily the migrate case I think where you can end up and I actually did this in my testing with like half the VM on one host and half on the other and that's not a good place but I think live migration you know if we find instances where you try and live migrate it fails and you can't get back to the original instance I think we treat those as bugs but migrate is the weird one. Does that make sense? The networking causing issues you know it can't recover the tap on the source one or now I don't know. Most of my failures that I've seen were exactly in that pattern. It would create an instance on the destination. Something would time out or whatever reason it felt that it should fail it but it will never get back the network tap on the source. In theory the network part is supposed to be one of the last parts we do for that reason but still because of some of the other foibles in this area of the codebase it wouldn't surprise me if there's race conditions in there that need to be fixed so yeah. Resize methods on the passenger driver? It's a little bit special because migrate does use the resize API. Evacuate uses of rebuild I believe which is different again just to confuse people. I was going to ask is there a set that you get free if from the point of view that no driver implements resize is there a lot of extra work to do? Yeah so if you look at the hypervisor support matrix which I originally had a slide in there on this and I actually pulled it because it wasn't particularly useful in a lot of cases but one of the reasons for that is that migrate is not actually treated as a separate call on that chart. So when you look at that chart you actually want to look at resize because if resize is there as supported for a given hypervisor then effectively migrate should work as well. It is actually the same code path. It's like the migrate bit in the API is like a really thin shim which is basically just passing the parameters straight through to the resize call. If you configure to resize on the same host does migrate then stop working? Yes so the thing I'll say about that is that typically we don't recommend turning allow resize on the same host on in the first place. It's primarily there for people who are doing like a single host test install just to try it out so that's the reason I don't think people are too concerned about that. That was I think one of the asterisks on that slide. Comment it like this. So if you've got ephemeral disk and volume on shared storage, live migration, what would that work? Yes. Yeah well there's a check in there. That mapped volumes check I mentioned in that when it's checking the source and destination host kind of check migrate will work. There is something in there to catch like if there's a mixture of volumes associated it won't let you do the live migrate it will fail out of the API call. Because previously it would really mess that up. I was looking at trunk today when I was double checking that so that's not a good sign. But I think it's been there for a little while because these issues in that aether pad have been percolating for a while so people have already been fixing some of those things. Any other? The fact that there's resize and rebuild stuff. Am I correct that there's no real plan to get rid of that? There's actually so I didn't have this in the slides because like with the metachron beyond stuff obviously I'm just forecasting based on what's being discussed. So there is a discussion thread that originally kicked off on OpenStack Dev talking about the idea of having a migration state machine. But it actually went a little bit elsewhere which was this there's a fair bit of agreement that the way migrate in particular works at the moment is kind of dumb. Like particularly like having a confirmed step where there's nothing you can really confirm either it worked or it didn't. So I don't know there's not an agreed upon plan for how to deal with that from an API or even a client perspective. There was some suggestions on the thread like I mentioned of having a move API which calls evacuate or migrate depending on the state of the host. And potentially reworking the migrate call to be a little bit more specific to migrate instead of just being a pass through. But I wouldn't say there's agreement around those things. There's this agreement that there's a problem. I would say I expect that to go somewhere just based on you know the people involved the amount of discussion that's going on about it in various forms but it may take a while. That's all. Yeah so I haven't looked much at it outside of Keystone in terms of the verbs they're using at the moment. That doesn't surprise me. I know they've tried to do a bit. They've tried to do a little bit of abstraction to try and make those things nicer and not necessarily bind themselves to how it is in the API or how it is in the original client is how we have to have it there. But I can imagine that. I wanted to use it specifically to try and live migrate without specifying a target host. So the schedule to take care and I just couldn't figure it out so I ended up using the API. Yeah. In preparing for this I primarily stuck to the Nova client and obviously the Nova developers at the moment that's what they're still largely focused on. I would say that while there's been a big push in so the Keystone community to move to the open stack client that hasn't that hasn't really happened in many of the other projects including Nova. So that in particular maybe a ways away. Yeah. All right. Thanks everyone for coming.


What is the difference between a print disability and a learning disability? You're all familiar with learning disabilities and all sorts of things, but there's something very different about the word print disability. So we're sort of big for this, but let me just see if someone would venture a... What's so important and what's so different about this new term print disabilities that's very different than learning disabilities, emotional disabilities, any of those kinds of things. What's the big change? Anybody know it's a big audience and it's away in the back? Great. Can you shout? Great. Perfect. It puts the medium as part of the disability and points to the medium and says, hmm, print is part of what's going on here and I forget the second part of your sentence and say it again, your second part? Right. Rather than the internal. It says it's not in the kid. It's in the interaction between the kid and something else, print. In this regard it's much more like allergies, okay? That it's something between you and ragweed or dust or whatever. But it's a very different concept to say you have ragweed because then it... I mean, say you have allergies because there's two things you can do. One is you can treat the internal condition sometimes. The other is to get rid of the ragweed or the dust or whatever it is, okay? And this is what this change is about to say, you know what? Print can be an allergen. There are some kids for whom print is not a good medium at all and rather than say they have an internal problem, they have a problem in there the way they relate to the environment of print, okay? A big change. And I think it's the beginning of a whole lot of changes which are universal design changes which are to say, instead of locating disabilities in the kid per se, we start to locate disability in the interaction between kids and their learning environments. Print in the next three days will often be the boogeyman for instruction materials that are not well designed. New media can be just as badly designed. There's nothing magical about digital media. They can be just as inaccessible and so on. But there's great potential in digital media that we don't have in print to overcome those disabilities of the medium. But those changes are now rapidly happening in the United States. The realization that we need to make materials that work for everybody instead of materials that work for some kids and then calling the others disabled. Okay, is everybody with me on that? So it's a big sea, a small but big sea change. It says the medium of instruction may be where the disability is. And in fact, our talks now, we typically go around talking about the disabilities of the various media of instruction. And we'll have reason to do that over the next three days. What are the disabilities in my curriculum?


Happy Choo Tuesday, ladies and gentlemen. I'm Joe for jaspeaskacebrakes.com and that is 2022 Hit Parade, a Jaspy Exclusive. It's Hit Parade football, Jaspy Exclusive. Dual case break, random team break number five, yeah, really nice price point for dual cases, for two cases, a lot of autographs, mostly autos here, and those are some of the chasers right there as well. You'll find these in about one per case, so trade at your own risk. Also, there's a Hit List promo. If you go Hit List, we'll gather all those Hit List names. Top two will get, remember those Trevor Lawrence, Top Trevor Lawrence Exclusive boxes, it sends you two, or top two will get a box each. So that'll be pretty cool as well. I think the boxes themselves, I think sell for like 60 bucks nowadays, sealed, but if you choose to rip it and you get some of those autographs, those autographs can go for a lot. 500, 600, 700 dollars. I saw graded ones going for $1,000, $2,000. So pretty nice stuff in there, if you're lucky enough. Big thanks to this group right here for getting into the action, I appreciate it. Josh X-Line going ham on this break, and that filler from last night. And nice guy, Josh X-Line. So it was just Kevin and the X-Line, Kevin, Matt, and Josh in that filler we did last night, you can see the video for it. Kevin snuck in with his one spot, Matt did not get in. The X-Line was nice enough to be like, hey, Matt Arnold, why don't you take that, take that 10th spot right there. So thanks to Josh again. Always good to see some nice guys in the hobby. And we just recently did this pack right there, congrats again to all the winners. No matter how you got in, I appreciate you getting in. There's all the teams right there. Let's roll it, let's randomize names and teams one out of five, six times. One, two, three, four, five, and once more, sixth and final time. We got Josh down to Zach. Yeah, that was cool of him. There's still some friendly people in the hobby. One out of five, six times for the teams. One, two, three, four, five, and six. And we got the New York football Giants down to the Tennessee Titans. These Giants got one over on the Tennessee Titans in week one. All right, Josh with the Giants, Santino with the Ravens, Josh with the Jaguars, Bills, and Colts, Matt Arnold with the Jets, Michael with the Patriots, the X-Line with the Chargers, David with the Cardinals, Barry with the Eagles, Mark with the Broncos, Daniel with the Packers, the X-Line with the Texans, Vin with the Seahawks, the X-Line with the Cowboys and Niners, Eastwood with the Browns, the X-Line Chiefs, Carolina, Detroit, New Orleans and my Raiders, Roger with the Vikings, the X-Line with the Buccameers, Matt with the spot from Nice Guy Josh, Steelers, Jarrett with the Commanders, Josh with the Bengals and Bears, Ryan with the Dolphins, the X-Line with the Falcons, Kevin with the Rams, and Zach with the Titans. And Zach has the Titans up for trade. As always ladies and gentlemen, trade at your own risk. Sort alphabetically by team. We're gonna pause the video to allow for some trading time. I'm gonna go grab those cases and when we come back we'll have the break. Stick around, B or B. All right, welcome back everybody. As you can see right there, there were no deals done. So let's gather all these names here. I'm gonna print this out in a second. Actually, let's print this out now and I'm gonna keep track of the hit list names right here by deleting the names of the people who do get hits. And then the hit list names will be, like I mentioned before, we'll get randomized and the top two will get either a Trevor Lawrence Sunbox or a Moonbox, Sun and Moon, like the Pokemon game. I think they're valued the same. Our sorting team will decide who's gonna get which box. So it'll be a little random element there, but top two will get one of those boxes. All right, there's the final printout. Hot off the presses here on Tuesday the 13th, Tuesday. Thanks everybody for getting in, for making this happen. Appreciate it. All right, so it's a dual case break. There's one right there. There's the other one over there. Some really nice stuff. I think at a pretty nice price point too, I'm trying to figure out like where this would land in the hit parade. This will probably be like hit parade, somewhere in between limited, which is the base level and the next level after that. Platinum level, somewhere around there. A little less expensive than the other hit parade collab that we had. The other one featured more graded cards, and I think this one features a lot more autographs. Different little twist, but all good stuff. Thanks everyone for getting in. Another dual case break on the site right now, jaspyscasebreaks.com. So we're doing the same formula, selling 12 spots outright and doing two fillers. jaspyscasebreaks.com, that's where we're giving away spots in those fillers. Helps fill the break, try to win a spot for a fraction of the price. Or if the full spots are moving quickly, we can cancel a filler, add more full spots back in, just whatever helps the break move a little more quickly. All right, to the hits. First one out of the gate, it's Michael Vick. Nice from an upper deck set, Supreme Signatures. Let's back in, what year is this? I don't see the year on the back of the car, but no, 2003, there it is. Nice one for the Falcons, that's going to be for the X-Line. I'm sure most, if not all of these cards, will have some nice grading opportunities as well. All right, next up is, is for the Jets. Zach Wilson, rookie jersey and autographs from Contender's Optic Football. It's Matt Arnold with the Jets, it's 11 out of 25, that's where the spot that he won. We need to see Zach Wilson back healthy and back in the action. That's hashtag good for the hobby. See what he can do this season. Next up is, Player of the Year Contenders. From a 2021 Contender's Optic autograph, that's my guy Derek Carr. 20 out of 30 for my Raiders, that's the X-Line with the Raiders. A little bit of a tough start for Derek Carr, but knocking off some rust. I don't think it's terribly surprising, or at least it shouldn't be, that the quarterbacks or the offenses that really didn't play a lot of pre-season series or snaps, kind of rusty starts. I don't think Russell Wilson played any pre-season snaps. Derek Carr didn't. Joe Burrow didn't. But I think Mahomes did. And he looked really good. There's Matthew Stafford, 10 out of 10. I don't think he took any pre-season snaps either. X's and O's, dual relic and autograph for the Lions. The Lions edition for the X-Line. Then we have Chris Godwin from Origins Football. Rookie, Patch and Autograph. Those Buccaneers paths always really pop nicely in that window. The X-Line with the Buccaneers. That number? No, no number on it, but still nice. That's true. There are exceptions to the rule. Peyton Manning never played pre-season, but I don't know. How are his weak ones, though? Were his weak ones a little less Peyton Manning-y than usual? I'd have to do a little research on that. There's Mark Andrews. Select, Jersey and Autograph. Mark Andrews, 30 out of 49 for the Purplebirds. Santino with the Ravens. I don't know, maybe Peyton's weak one record is a lot better. It's hard to compare, you know, even Russell, led likes of Russell Wilson to a Peyton Manning, though. For the fish, Jalen Waddle. Waddle, waddle, waddle, waddle, waddle. Penn Powell's on-card autograph from the lead football. And that's going to go to Ryan Stewart. Got Miami in the randomizer, won that spot, too. For my rivals, the Broncos. Rookie, Jersey and Auto, Jerry Judy. Had a touchdown yesterday, 21 out of 49. Swatch autographs. And that'll be for Denver. That's for Mark N, won that spot. There you go, Mark. The next box. And we've got a Devonta Smith. Rookie ticket autograph for the Eagles. Nice start to the season for the Eagles.  How did Devonta Smith do? Let's see, a little curious. Let's go look at the scores. The box score. Devonta Smith. Is he injured? No, had four targets, but no catches. All right, a little slow start, but hopefully he'll ease into the season. That's going to go to Barry and the Eagles. Next box. Wow, man, he's built different. So I don't know if everyone heard, but Chris Chasvey did a little research in 17 week ones, 14 and three. But yeah, I wonder what those stats were like, though, yeah. Joe Mixon, green ink autograph, out of origins. There's a seven touchdown performance here from Bay in the 20th grade. You know, in week one, the X line has Cincinnati, but wait, there's more. Oh, you know what? I've not been keeping track of the hit list promo. So sorry, give me a second here. We got Bangles hit, Eagles hit, Broncos hit, Dolphins hit. I know, come on, man, Broncos hit, Dolphins hit, Ravens hit. Lions hit, so did my Raiders, Jets and Falcons. All right, let's play two. Wow, Chad Dodd just shoehorning in a shot at Russell Wilson. Chad Dodd says, but I believe Russell Wilson's oh and one in week ones as a Bronco, right? Really, really shoehorning that one in. Yeah, yeah, hey, I was on Seattle plus, I was on Seattle plus six and a half. Chad Dodd, danting on graves. Chris, I don't know if that's a dangerous thing, Chad. Dancing on graves. All right, we got Marshall Fox, flashback autograph, 48 out of 99 for the Rams. There you go, Kevin was worried there'd be no Rams. There you go, Kevin. I hope so, Chad. Chad saying Broncos, last place in the AFC West this year calling it right now. Wow, that's Chris. Thanks, guys. I mean, that means the Raiders won't be last. I have Raiders second. Wow, I have Chargers third. You have Chargers third? I did not think that, I don't think Herbert's that guy. Wow, he's good. Oh, the Chargers often do chargery things. There's Michael Carter, rookie autograph. Six out of 10. Nice one for the Jets. That's going to be Matt Arnold. Jets already hit in the first case, too. First case as well. Next up, and we've got an AJ Brown, 17 out of 49 from gold standard football. Two-color patch and autograph. Nice, from 2019. Rookie, patch auto. Zach Gwynn with the Titans. I think, was that tried to trade, couldn't trade Mojo? Yeah. Yeah, and he had Titans up for trade. Nice, AJ Brown. Great start to the season for him. Do I still have that box score up? I do. 10 catches on 13 targets, 155 yards. No touchdowns, but a 54-yard long. That's pretty nice. Oh, interesting. Mrs. Chadda made a good point the other night. Mariners get rid of their old sad player, Kyle Seeger, and are going to the playoffs. Seahawks get rid of their old sad player, and then they start the season 1-0 against that very player. There's Tua, Tunga Bailoa. Contenders draft picks, on-card autograph. Kind of see the blue parallel, the blue foil right there in the lighting. That obviously goes to the Dolphins. And I think the Dolphins already hit. Yeah, there you go. Hosting notes here. There we go. Tua and the Dolphins, a solid start to their season as well. All right, nexty next. Ooh, this is a real thick card here. It's a book. That's why it's so T-H-I-C-C-C. It's Jalen Hertz, 55 out of 99. So peel that sticker just so the Eagles can take a look at it. Had the Eagles hit already? They did. Barry Robert. Oh yeah, that Devonta Smith. Now you get, that's my quarterback. Yeah, speaking of Jalen Hertz. Yeah, 18 for 32, 243 yards, 17 carries, 90 rushing yards and a rushing touchdown. Eagles fans are happy. They've got a good-looking team. He's got a good-looking autograph too. Nice Jalen Hertz. There you go. Barry Roberts with the Philadelphia Eagles, the Fly Eagles Fly. From 2020 playbook. That's right, Jay. They should have gotten Corey see here. Next up is, is neon signatures Jason Witton. Seven out of 35. Not a little 82 inscription right there. That's a cool looking car. That orange right there really pops, a little jazzy orange. That goes to Dallas. That's going to go to the X-Line. So we got Rams to carry that. Jets, Titans, Dolphins, Philadelphia and now Dallas. A silver ink autograph. That's Muth, the pat-a-friar Muth. RPA, 31 out of 99. Silver ink on that eye black. That's for Chris Jaspy Steelers. Who has Chris Jaspy Steelers? That is Matt Arnold. That was the nice guy spot. The spot that the X-Line gave you. And you end up with a RPA. There you go, Matt Arnold. Thanks again to the X-Line for being such a nice guy. That's the kind of people we have here at the Jaspy and the Jaspy community. Michael P needs some patriots. Got a few boxes left. Michael, good luck. Here is a Raider, Darren Waller, Shining Stars autograph. 14 out of 15 for the Raiders. That's going to be for the X-Line.  Darren Waller, well-deserved.  Maybe not this season, we might be stuck with it. All right, next up is Adam Thielen. 10 out of 10. Feeling good. Roger with the spot that he won in the filler. Got randomized. The Vikings gets the Adam Thielen. Looks like from a Panini reward store card. Last one. Good luck, everyone. It's going to be a Buccaneer. It's Chris Godwin's Super Bowl autograph. Tampa Bay, the X-Line with the Buccaneers. Here you go, Josh. Thanks, everyone, for getting in. We got another dual case break in the store. And remember, if you go hit list, you got a chance at those tops Trevor Lawrence boxes. T-Laws, so let's gather all the hit list, folks. My apologies, sad times here, but a little second chance opportunity, which is always nice. So let's get him into a new list. Let's clear out the blank lines here. And these will be shipped to you sealed. If you choose to open it, let me know if you get a T-Law autograph. You can sell it if you want to. Get a little money back. It's all good. It's yours. All right, top two after five. Two and a three, five times. Good luck. One, two, three, four. And fingers crossed, fifth and final time. After five, Josh had the odds on his side. The X-Line, top five or top two. After five, you're going to get those T-Law boxes. All right, thanks, everybody. We've got more of this in the store. Check it out, jaspyscasebreaks.com. I'm Joe, and I'll hit parade with you some more next time. Bye-bye.


And in the same vein, Britain has accused Sudan's military of having betrayed all the people of Sudan after it arrested the country's Prime Minister in seized power. Hundreds of people poured onto the streets of the capital, Khatum, and its twin city of Om Duman following the early morning arrest of Prime Minister Abdullah Hamdoq and other officials. The UK most strongly condemns the arrest of civilian members of Sudan's transitional government by the military today. We are also deeply concerned by reports of shooting of protesters. This must stop. Over the past two years, Sudan has been on the delicate pathway from oppressive autocratic rule towards freedom and democracy. The UK has been a consistent and firm advocate for the democratic transition since the 2019 revolution. The acts of the military today represent an unacceptable betrayal of the Sudanese people and their journey to democracy.