uld be with the microphone, I'm assuming I'm not gonna have the headset. Okay, perfect, perfect, okay. So, oh, I love a dim setting. Is it controlled, I mean, yeah. Looks like Ryan is coming to investigate. There is, yeah. Oh, that's great. Everyone can see all right, these slides? Great. All right, all right, I'll sort of put this, oh gosh, I don't like have a belt or anything, so. We'll just put this right here. All right, okay. I think I'm properly hooked up. Yep, okay, now I'll stop fussing with this one. Now I'll fuss a little bit, okay. One more fuss, okay. All right, how am I sounding? Everyone here, oh yeah, great. I also naturally project, so yeah. But I'll give some, I'll give some mic love. All right, why don't we get started, y'all? Why don't we get started? Hey, so, you're at this talk. So, let me ask you this, like who does this for a job? Yes, show of hands, that's what I want. So, like, y'all work with production systems in your job. May I have some job titles, please? Just shout out, anybody's job title, cooler or the better? DBA. DBA, I mean DBA. Assistant engineer. Yes, assistant engineer, assistant engineer. You got any performance, you know? Any network engineers in the house? No, I mean, mad respect for network engineers. Anyways, yeah, glad to have y'all. It sounds like we all have a pretty good idea of, oh, next slide, yeah. We all have a pretty good idea what production is, and so I'm gonna skip all that part.  So when you have an issue at home, what do you do? Well, you could do, yeah, you could restart brand, you know, restart the whole note if you wanted.  You know, it works sometimes. We all need to be cleaning our cases more often. You know? I don't know what's causing the problem, as you do, as you do, as you do. Yep, or maybe, maybe it is just time for an upgrade anyway, you know? That's, let's use that as the, production, our job, our data, our livelihood. No, no, no, can't, can't, can't. No, no, no, no, no, no, wrong. I'm gonna get you fired, mm-mm. So, it would definitely get me fired because this is not any of my stuff. This is other people's stuff. So I'm a systems engineer at LinBit. I work with high availability clients, so they have clusters of highly available storage that they need to fail over. And we all know it. This isn't a talk about DRBD or LinBit, by the way, this is just, yeah, too bad. That's just context for what I am doing, and it informs my need for knowledge on performance and how to troubleshoot, et cetera. Diagnose those issues so that I can help out the people that have these systems. They also have other clients that they will make available their infrastructure for, so by the time that I am getting a support request from them, I mean, there's some definite zeal for that issue to be resolved. All right, yeah. Oh, I guess I did put, yeah, so that you would need to, you'd be able to know what an AHA cluster is. But again, if this is a, see me in the booth tomorrow if you wanted to know about that.  I mean, is that all right? I'm breathing. All right, so it's gonna go way over here because I'm pretty good at projecting. Yeah, I mean, is five good? You know, mm-hmm. So it's unfortunately a little bit complicated of a question because it depends on your hardware, it depends on your network, it depends on your workload itself, your application, your SLAs. There isn't a good answer for it, so you can't just go Google, is it good number? It's just too hard of a question, but I'm hoping that I can establish today for you some direction towards is this gonna be good or how can I make it less bad at least? All right, what's the lot I have next? Ah, yes, so I'm going to give us a few rough concepts and methods for how we're going to be doing this. And then I'm going to dive into, I have six demos for us, live terminal demos. That's that they're going to use those sort of concepts and we're just gonna go through them. They're imaginary scenarios, but they are based on true stories, things that I experienced day-to-day as I support HA clients at Lynbit. So hopefully we can learn along the way. Oh, yeah, the slide seemed a little empty, so I just needed to put a little graphic there. All right, so overhead, I mean, it's not that. Overhead is what we have when you have maybe Rantop or something like that, and it's going to be costing you processing power as well. The performance impact to actually gathering the metric, which can be difficult because you can't stop on a production system. Yep, so there's ways that you can make it better, make it worse.  It's not going to be good. I just wanted some examples. So I mean, just here's top. One of the slides look good and I circled it. So one of the ways that we're going to rely the most on in diagnostics for production system, they fall under observability, which is observing the systems as they are, where you're not simulating anything, you're not applying any sort of test workloads. So all those metrics that are static to the system, they exist and you can just peel them, peel them out of there. This also can apply to static configuration. All right, but sometimes that isn't enough and we will actually have to do some benchmarking. Oh yeah, I guess this is a, yeah, if you're just catting like the pressure statistics for CPU, that would be an example. Yeah, pressures, all right. So microbenchmarking, that's going to be a lot better than macrobenchmarking. It is a taking a component of the system. So something within the data back, but not the entire application workload. And we'll simulate it. And, but it would be an example of like, you are running some packets across the network. It's going to have lower overhead than just doing like whatever your application might be, assimilating that entirely, and then saying, I don't know what's doing it, but I'm just going to apply that application again and again. That's not going to get you any closer to the diagnostic of the specific thing you need to improve to get rid of this bottleneck. Macrobenchmarking definitely has its place and sometimes you need to do a mix of the two, but I would say you're going to want to, if you can get some observability metrics, then we want to go for some microbenchmarking, some targeted microbenchmarking. But yeah, you might need to do some macrobenchmarking. This one example, I mean, paying is technically an example of a microbenchmark. It's a little one, lightweight. Oh yeah. So yeah, it's a simulation. You aren't, this isn't, macrobenchmarking wouldn't be where it's just running in prod and you are observing it. You are applying it in this case. You are applying a workload yourself and then you are checking on how the system performs based on that. All right. Yeah, settle that. All right, well, I mean, I guess I tried, I wanted to demonstrate a little bit for each one and then I was just like, okay, yeah. You based on that, you, all right. Well, here's how it performed. In order to do that sometimes, you're going to have to characterize the workload that you have because not all IOPS are created equal. They are, you need to know how your application is impacting the system, the qualities that it has. This is especially nuanced when it comes to storage. There are so many aspects to storage that I'm so excited about that we'll get into later, but they will have immensely different results in the system. They might look similar, but there's a lot of different qualities of them that you can just admire the nature of. Once you can break that down, you can do more effective microbenchmarks that are actually going to be reflective of the workload you intend to put on the system. All right. I didn't, yeah, I just, I needed this application to be a little more explicit so that I could, yeah, decide that I needed that block size. And so this one just happens to tell me that it writes 512 sequential bytes to the disk and so I am doing a DD that will, has a block size of 512. And then I'm writing it to the disk. Performance tuning is what we would want to do kind of last with our production system. That is what you'd expect. It's changing things and seeing if it makes it better. That's something that I would probably do right away at my home lab. I am going to really narrow in on what I'm trying to do because you don't, you also could just, you're not only you're going to be impacting, your day-to-day money machine, you are probably going to be getting different results and you're going to be confused with the issue. If you do not target it first and you're not, you don't have a diagnosis first. You don't have a specific component that you're zeroing in on. You just change your random things, you're going to confuse everybody. Yeah, so I guess I read nice this as an example. So anyone familiar with the use method? No, well I'm about to tell you what it is. So there's a lot of methodologies you can use to diagnose performance bottlenecks. The one that is used most commonly in LinBit and that I find is my favorite that I'm going to zero in on today, that is the use method. And this was developed by Brendan Gregg. He is a, well he was a performance engineer for Netflix and he has written some really interesting things. Oh yeah, I just, honestly, the book is great. He also has another book that he recently came out with which is on EBPF that is really exciting. I don't have time to get into EBPF today but really nuanced stuff. If you want a deeper dive on this, I would definitely recommend. He also has some really good utilities on the website. If you just need good like checklists, they're just super handy to have. And so it'll save you a lot of work. In case you wanted to make a checklist, now you just take what he's already done. It's on his website. But yeah, it's use is utilization, saturation errors. I am going to go into what those all mean right now. All right, well I guess I'm going to go into what a functional block diagram is first. So for every resource on the system, you want to know what you iterate on for each thing. Oh yeah, this is the one we would use. This is from the DRBD user guide. That's a piece of software that Linbit supports. And this would be, it's an example of a functional block diagram. These are, it's not a picture of the system but rather a picture of the relationships and the components inherent to it. So we have, these have drivers on them but I'm looking at the physical hard disk here. You have the file system, the caches also and then the service layer. So you can tell where everything is interacting. That'll help you, if you don't have this for a system, I recommend you draw it just to get started or just ask your client if they have relationships like that, relationship diagrams so that you can have somewhere to start. But this is how you determine what your resources are that you're going to be applying the use method two. Continue. All right, so utilization as it is defined within the use method is the time, the percentage of time that a resource, a component like we saw in the functional block diagram is performing work. So let's say we have disks and they're totally idle. They're 0% utilized. We have some disks. They are churning, spinning 100% of the time. There's not a time that they're not active. They're not moving around. That is 100% utilized. But I wanna be clear with utilization in this context because with definitions it could be a little bit nuanced. A fully 100% utilized resource can still potentially accept more work. I don't know if I say that, yeah. Oh, well, saturate some context. It's related to utilization, but let's say you have a CPU and it's waiting on this guy or something like that. It potentially could get a higher priority process that comes in and it's able to move things around. And so because it's able to re-queue things, it could be 100% it's doing something all the time, it might not necessarily be saturated, just something you keep in mind. Saturation is related, but that is the time that a resource is, it has more work than it's able to process and it's basically how much work it can't process. Something to note about saturation and utilization in your interaction between the two. So let's say that you have a restaurant that is slow most of the time, but then you have a real banging lunch hour and people are waiting outside. Yeah, you could have overall low utilization. That's just something to consider when you see some scenarios later. Low utilization, overall, you can still get peaks. Those peaks are important because they cascade through the system and cause saturation, but then everything else is waiting for it and yet just keep in mind those bursts can make a big impact. So even if you have overall low utilization. The E of it is a little bit easier. Airbors don't really explain errors. I think we've all encountered one or two in our day. One of the most, with the quickest ways often to diagnose what's wrong with system is say, what is it telling us? So ask it, look in the logs. There is a, Brendan quantifies these as like the number of errors. I don't necessarily do that because I don't, yeah, I can have a bunch of errors, but what are those errors can often be pretty important, not necessarily how chatty the logs are because I feel like you could just sort of toggle that into a sort of thing. All right. Ah, yes. All right. Now we're getting to some fun stuff. Let's get to demo number one. All right. Oh, it's getting all, it's all dark now cause my, my terminal is all dark. Great. Okay. So I'm going to set up a hypothetical for you. So I have a, yes, probably. Condro plus. Oh yeah. Is that, that's not true. I've got to find the plus sign. I have never done that before. Excellent. Thank you. Yeah, I would not have noticed that because I can see that just fine. Thank you so much. And if you can't hear me or anything like that, yeah, feel free to shout it out cause I'm not going to notice. So just, yeah, hey. All right. So this is a hypothetical scenario of, let's say that there is a client that I have and they are running a DRBD cluster. So they have a, like just a few notes in a cluster. They have said to me, they have configured the instance and they've said that they've recently seen some performance issues and they believe that DRBD is the cause they would like me to investigate. So first, first. Oh yes, yes, yes, yes, yes. Oh yes, yes, yes. Mm-hmm. Oh yeah. Ha, ha, ha. That's, don't look at this. Don't look at this one. I didn't want to run it early cause I was afraid I was going to tank everything. All right. So I have my CPU one up. Now, whoops. I did not have the other CPU one up. Control plus, right? How's that? Oh, a couple more time. If y'all don't know about Vagran, by the way, it is super nice for if you need to just test some things. I, we use it all the time at Linbit so you can just get Vagran stacks that you sort of, you set them up in advance. You, then you could use Ansible Playbooks on them so you can quickly configure some VMs to whatever use and abuse you have. And let me see, Vagran CPU zero. And I get in there. All right. So let's take a look at what could be happening. All right, well, like, what I want to point out here, don't look at the obvious, but what I want to point out here is some things about load average. So, load average, that number, like, is it a good number? It really depends on what your system is. How many cores does your system have? How do you find out how many cores your system has? You can, let me see what I got in here. I've been in the history. I saw this in the Ceph demo, so I kind of like it where they sort of just have all of the command history and they just sort of paste it in for themselves. You know the demos never go right. Yeah, I'm getting to that. I'm getting to it. Yeah, but I just sort of put the command history in and then I could do all that just so that I don't forget anything. So if we wanted to use, like, Nproc, and we can come up with that. Let's say we're gonna use Nproc. Okay, eight. So the load average we saw was kind of high for eight because my stress NG is being particularly performance right now, which it wasn't earlier, but that's okay. Let's say that we wanted to use Listopo. Oh, okay, I didn't install it, but maybe I should install it now. I think I need Herberluck GUI. Oh, I want to use Herberluck. Yeah, sure. This is nice because if you have systems with Numa nodes, Listopo will be able to show you a diagram of them in some cases. Otherwise, it'll look like this. It's good to look at the relationships between things and sort of make sense of the system. If you aren't in this system all the time and you look at a lot of systems, it's just really nice to get your head on straight and what the number means and what you're seeing when you dive into a system. Yeah, how many cores does it have? What is it? What are we doing with it? So just keep that in mind. You could do that. What else we got? Yeah, what did I say that we should do? Oh, yeah, you can cat proxy view info if you so desire. What we got for that? All bunch, but these are examples of how you can take a look at things. But yeah, let we go back to top. Let's see if this has been distributed evenly. Maybe we can't really tell from this. Oh, man, it's really tanking this guy. Anyways, let's go ahead and we wanna see how this is being broken down by processor. So I wanna take a look at the MP stat. You need the capital P flag for PubM, you know. And then we can take a look at, so what we should have seen, my test might have just completed. So what we should see is let's do a PID stat. Don't worry, we've got five more of these. So notice all of these that are pinned to, well, one might consider the fourth CPU that is entitled three here. So let's imagine this isn't StressNG and let's imagine it's somebody clever and they have their application on here. And then DRBD, usually it's pretty smart and if you don't do anything to it, it's going to pick what CPU, oh, it's going to pick what CPU it thinks is going to have the least impact to the system. It doesn't want to mess anything up for you. However, you can decide you do know better than it and you said to hit, I want a CPU mask of one processor or a certain subset of the many processors you have, eight, you might be less confused, but let's say that you have like 64 of them. That person might have said, they might have set the mask wrong, they said a set of CPUs that are going to be associated with particular NUMA nodes. Did I tell you what NUMA nodes is? If anyone doesn't know what NUMA is, that's non-uniform memory access and just really simply oversimplified is the memory module is closer, just physically closer to the processor, a particular processor, so it's more efficient because it's just physically closer together. So you would want to pin it to particular parts. So you might think, okay, well, I want to do this because I want to make the system more efficient based on what I know about it. And this person made a mistake in this hypothetical as they made a mistake and they decide, you can have an overall load average that is low, but you want to look at that particular processor breakdown to see if there is a particular one that has odd use. That's usually not going to happen, but keep those tools in mind to see what you've got. PID stat, PID stat's a great one, and MP stat is one I like to use for looking at individual processors, what application is churning on each one. And if you intentionally do things with them, just keep that in mind. All right, all right, I'm done with you CPU. I'm done with you. I don't want you doing this anymore. Koi, all right. Do I have my history in here? Oh, I need to control plus all of these. And, and, number two. Just so we know, we're moving on. We've had enough of that. Let's see, cool. So here's our second hypothetical scenario. We have a client. This isn't our machine. This is someone else. They might be in a different time zone. They might be halfway across the world. This is inconvenient for us because they write us emails saying, the system is slow. The system is not working. And then you come and you shell in when you should be awake and everything is fine for you. What do you do? Well, so they said the system, in this case, they said the system is being intermittently slow. We've got, you know, it's, we got replication speeds for DRBD slowing down at particular points. But what do we do if we take a look at, I don't know, top and we're seeing it looks fine. Well, let's set up SAR. So SAR is the system activity reporter. And yeah, it's make sure I have it in my history here. I actually think I uninstalled it from this system just so that I can install it. All right, so it's part of the sysstat package. And at least it is for, you know, Debian based systems in this case. You want to make sure that you enable it as well. You can't like you installed it, but we do need to enable it. So, let's see where we go to enable it. I think I put it in here as well. Ah, yes, I am going to. Okay, well, it's not there. Oh, I sysstat, whoops, sysstat. Okay, right now it's false. Right, let's get on down here. I think you all have an idea of what's gonna happen with this that troop. You need to do that, at least for Ubuntu. I believe you also need to enable it, I'm not sure. But we're gonna do it to make sure. See if we want historical data. Okay, now we need to restart it. This is the initial setup. If you already have this running that this is just for your first time, and then you have it. Okay, now all we need to do is like wait a day, right?  Nah, okay, I have just like a cooking show. I've already done it on this one. Oh yeah, I forgot on the other one. In order to configure it, because I didn't even say that, this is how we configure it. It's just like Kron. If anyone set up a Kron job, it works identically, at least in Debian-based distros. So we are going to go into, I'm trying to use all of my things I prepared for myself, but I'm just gonna go into Etsy. And then Kronty. See, we got Sysstat hidden here. It has already dropped something in. We just installed this. It already has that file in there. Oh, that's why I didn't put it in the command, because it already has it the way I want it. But I want you to look at it anyway, because just like you would expect, it looks like you would have in a Kron table. For anyone that needs a refresher on that, we have for five minutes, every 10 minutes, and you have, I believe it is, what is it, month? It's a day, hour, day, month, 10 minutes, yeah. Just all the time, week, day of the week. Yeah, and that's what it says. But that's what it means. And then it also rotates nicely for you, so you don't have to remember. But we have that, since I don't know more about the particular behavior, I just came into the system, somebody said it's intermittently slow, they didn't give me a capture of when to focus on. I'll just take five minutes, every 10 minutes, sounds good to me. Maybe we can see something from that. So let's do that, let's just keep that at that. And let's see. All right, where did I put that? All right. Yeah, so it is, I actually would love you to see as much as I already had that prepared. Let me show you what we look, what a Varlog sys stat goes, because that's where everything is kept. You can change how many are kept. Yeah, so I have, since I set it up, how many days I have, and you can reference it by the day of the month, in this case.  In this case, I have, this is a network one, and I have, I'm passing it to a file here. Again, refer to the man page for more detail, but overall, just the general sense, this is the 12th day of the month, because this is when I set up this VM. But it has all of that information in here. And this is the return I get from that. It's real nice, cause you can.  All right, so let's see if we see anything going on here. Da, da, da, da, da, da. This PM looks fine, looks fine, looks fine. Was there a question? Is there a huge jump? Yeah, are we still going up? Down, the other direction.  Yeah, 420, and then 420 over here. I'm still, oh yes, oh, the huge jump. I thought you meant there was a skip in the logs, cause that would have been obvious I shut off the VM because I, yeah. So there was a, I was doing a lot of things that, this is not, so in this case, it does look like there was some pretty heavy utilization during this time. You would, this wasn't part of what I was doing. I was rerunning some tests at this point, that's why that showed up. Okay, but this is what you would look for because that is, like this is a huge amount of utilization here for having nothing on it before. If someone's saying that I have an intermittently slow re-sync, why do I have all of this traffic here, just at 4, 10 p.m., in this case it was, because I was running tests on the system, and yeah, I had a low generator that I was working with, so this is what I wanted. So, why is it starting at 2 a.m. exactly is the point that I was trying to make. You're seeing a lot of traffic happen at a certain time. You can be looking for patterns of behavior because you get this output. You're saying, what is the pattern behavior that I have for the particular things that I'm searching for, so network activity on particular interfaces? I don't think there's a way to break it down by interface, or at least if anyone knows, let me know, because I don't know how to refine, sorry, finer than these flags that I have, but this gives me each of the interfaces and how much throughput was going through them at a particular point in time, so excellent. So, yes, hello. Oh, absolute, that's a great question because I have a whole, you said memory, right? Oh, yes, great, because I have a whole thing on that. Yes, yes, SAR is super useful. It's incredibly versatile, and you are able to filter down, also notice when I set up SAR that I wasn't like, okay, I wanna do a network test. I didn't need to do that, I just let it get everything and then I filtered for what I wanted at a particular day. So, I know that we all are sort of inundated by information, sometimes it's an information wall. You need to figure out ways to break it down, make it smaller for your human brain to pack it in, and this is one way you could see it. You can say the time, hello. There is, it also is SYSTAT. I do not believe that it is based on Cron. I do believe you can figure it slightly differently. I didn't have time to get into both scenarios in this case. I also wanted to show Cron for that setup because I do believe in, like, ALBA, you do it, I think you do it through system D, but it is slightly different, but it is through SYSTAT. SYSTAT package is one you would use. All right, so why do we have Activity Spike at 2 a.m.? I don't know, maybe it's a backup.shell at 2 a.m. Could be, it's something to look for. I would have, in a normal situation, in an actual day to day in my job, I would ask, like, I would probably look in Cron myself before I, you know, set it and forget it. However, let's say that this wasn't just directly on the system, maybe it is just traffic that's just hitting that system. This isn't a cluster anyway, so not everything's going to be scheduled directly on it. It's just something to keep in mind. If you're, especially if you look at multiple days over time and you say the same thing happening at the same time every day, something to keep in mind, just something that you can say, is this a scheduled activity? Why are we seeing this now? And then you can look for processes or other activities that are happening within the data path that might coincide with that. Just one way you could do it. Yeah, in this case, this was just, yeah, this is an IPER-3, just congest the entire thing, throw it at it for a while, just to make a little demo. So that's SAR for you, but we're going to continue to talk about SAR throughout the rest of this lecture because it is really useful, right? We're going to go back to demo number three.  Okay, so now we're going to the next scenario. Ah, yes, here we are. This one, control plan, we good? Oh, cool, wonderful, because this one actually, I will want you to see the details on here. So I appreciate you letting me know about this. So this is a scenario that we have sort of building on the other scenarios. You go into the system, I'm not just gonna go through top again, like you're gonna go through a system, now it looks like the activity is fine right now. I'm not seeing any huge amount of utilization anywhere, not a particular process that seems like it's hammering on one single processor. And I also looked through SAR earlier and I didn't see anything historical that seemed of note. So, but why would that be? This is just a particular case that you might want to check for. So, let's not forget about the E-section of, yeah, the E-section of the use method. We wanna look at what we got over here. So I didn't see any heavy CPU utilization, but like it's running like hot. So that's an interesting thing to note. Why is it so hot? And don't like, it would say de-message, not de-message, oh, I actually didn't know what would happen if I just replaced de-message, so I just put it at, I just made a different file. Let's say I wanted to look at just the, okay, well not, oh, extract. Okay, well not that one either. I'm gonna go into our log. Oh yeah. Oh, okay. Okay, so this is one that isn't, it's specific to the application that I work with a lot, and that is a Corosync, that's a, it's got a heartbeat between nodes. The nodes need to know that the other one is alive, so that it doesn't, so they could do what it needs to do, avoiding split brain, knowing that it needs to kill the other one in case it stops fawning, things like that. Really important to be stay connected. So I happen to know that this process is a real-time process, that's basically the highest priority you can have. It'll like everything at the expense of all of this process, it needs to happen. But it's telling me that this has not been happening, like this second, full seconds at a time, we're not seeing this being scheduled. But again, why are we not seeing some heavy CPU utilization? Why are we not seeing in SAR? We're not seeing all of this traffic having the CPU, why is it not being hammered? Well, it's because in this case, this is a VM. This is a VM that is on a hypervisor. What can happen is there's a few things that could happen when you've got a guest and you have noisy neighbors. You have a hug, a different tenant, perhaps, that uses all the CPU on the hypervisor, and then this VM didn't get any. He can't schedule anything, but it also can't really tell you anything about that. No, I'm dying. It can't do anything, because it can't be. It doesn't have that visibility in most cases. In most cases, it doesn't have that visibility. This could be also a case where the VM was paused on the hypervisor, it's not gonna be able to tell you. I don't think it was in this case because the heat, the temperature, but that's something just to consider that it might be. Let's see, what did I say else about that? Yeah, one thing that you can look at that is an indicator is VM stat has, if you do a VM stat, you can see. Let's see if I had the VM stat in here. Do I have a VM stat? Well, let's see what I got. VM stat, no, I just had, because it's just VM stat. VM stat's one you can use. I wanted to show this, there's a lot of ways to see the ST here. I think I wanna use the V stat on there, but. We see that in top as well, that ST. Do we see it in top? Yeah, we should see it in top somewhere, but the ST, that's CPU steel time. That steel time is when a VM is waiting for a CPU from the hypervisor, and that's the amount of time it's waiting for that. So if you see high steel time, that might be an indication. It's not necessarily going to always show that way, because of the situation I'm describing, especially if someone paused a VM to back it up or something else happened. So that's not going to be a guarantee. That is one metric, though, that you definitely wanna keep a look at, especially if you know that you're working with a VM. A lot of times in my work, nobody's gonna give me the hypervisor logs. They are just saying, well, this cluster is slow. They're not gonna say to me, well, because I didn't give it resources to run. That's also another thing is that people could, on the hypervisor level that someone could get, could intentionally under provision something, and to keep it and reign it in to a certain amount of CPU or any sort of resource usage. Arbitrarily. So it might be kept at a bottom up because it's been told to, but it doesn't know that. So just something to keep in mind. These are factors to consider, questions to ask, and defining a problem statement. One question to ask is this a VM? Could you possibly give me some information on what's happening with this hypervisor during this time? That would be helpful. If they can, I would love that. You would love that. Everyone would love that. Everyone liked that. Okay, so there's not gonna, surprise, it's different. Okay, so this is demo number four, but I wanted to show you something about it before we dove into it was I wanted to show you, recall that functional block diagram that we were talking about earlier because this is to give you a primer on how this is working here. Is that DRBD, it does a synchronous write in most cases where you get, you have a primary where they get the rights, something's written to it. Now it needs to make that synchronous copy and it needs to do that over the LAN. So the network here is oftentimes the weakest point. That's often one that I'm gonna look at if I'm looking for a bottleneck. It's also one that's fairly easy to check. Something that's easy to check is, something I wanna check first because our production systems are fussy and I'd rather do the easy things first. So that's just giving you some context of why did I check the network first in this particular scenario where someone is saying that they're not getting the eye-ups that they were promised on a newly deployed system. They said that they did test before. They thought that they should be getting a particular, that the network latency that they're saying is higher. There could be a bunch of things in this case. I'm not sure what I wrote about that. But as yet, they're not getting the eye-ups that they originally benched Mark IV. But I wanna check the network. I could do it if I was not on a production system. I could do some A to B testing. I could shut the network, like the functionality for DRBD. I could say, just do the reads. Or just write to the primary. Don't even do the replication to the secondary. Because then it's not waiting for the acknowledgement across the network. It's not going across the network at that point. I could do that, but they're using this system. I'm not gonna do that. So what do I do? Well, well. Okay. We're gonna go back to Boink. I'm gonna use Control plus all the time. All right, what do we got over here? Oh yeah, all of my stuff. Great, wonderful. Oh yes. So I wanna see how they've configured this. I wanna see, in this case, this is a DRBD specific command. But that's just so that I can see what IP, what is their traffic running on what, how are they configured? So we see some IP addresses here. We usually say, for a DRBD setup, if you could put it on a, like a replication network specifically dedicated for replication specifically. So it's own, it's own little lane of traffic. Because then you don't have to deal with other traffic on management or other applications that are happening within your network that are potentially going to congest everything and delay replication. We just want something nice and clean and dedicated. One thing that is helpful for that and this client did do this. It's helpful is that I could also, it's better for diagnosing particular issues with if we're seeing some flowness there. Oh, close again. Okay, all right. So what did I say that we could do now? All right, so I do want to see at that point, what do we got? How's the, what's this configuration look like? What interface is that? What interfaces do we have on this, on this network here? And we've got these ones here and we can through the, we can take a look here at IPS. I'm sure people use if config. There's some deprecated ones. IP I think is like the one that we should be using lately. IP, IP link show should give you some information on the interfaces and that's specific to the traffic that you're seeing on them. You can see some things that could be indicative of issues with the network could be a large amount of dropped packets. But what we're looking for right here in this case, we just wanted to see what interfaces we have and the overall setup of this interface. Let's see, did I do the link show? Yes, I did. And that was the S-length, yes. And then I do the ETH tool. Ah, yes, cat, cognitive. That's another way we can look at it. A lot of these are, they're looking at the same systems counters. Here's just a different way to get it. Use what feels good to you. Because a lot of times look in the documentation on what system counter it is referencing. That's another good thing you might wanna do when you are doing a diagnostics especially if you are using some manner of observability tool is to use these different tools and use tools that do the same thing and see if you get the same number from it. And if you get different numbers for running what you ostensibly thought did the same thing, well now you've got reason to look at it. Is it maybe not quite what you thought? All right, so we've got, that's just another way that we can look at this interface and we can see the number of packets that are transferred over it and if there was potentially anything associated. Do I have my ETH tool command? Because I have a ETH tool, yes. I wanted to look at particularly that interface that was configured for DRBD in this case. So that's ETH tool. ETH tool is great because you can use it to find what is the, the static configuration. How it is, what is it? It's currently negotiated speed. See it's full duplex here. It is, yeah. This per, like so in, I wasn't very creative with this particular demo because like I ran out of inspiration. So what I thought in this case was that this person said I did some nick-trunking and I should be getting double this. They said that, oh well, but no, I'm just checking behind them and I said no, no, no. This is a misconfiguration. So the reason they're not getting what they expected is that it's not configured correctly but this is just one way that you can check. What are our assumptions versus what is the system telling us? It's quite easy to misconfigure when you're trying to do some nick-trunking. And I have seen people that are trying to bond the interfaces together and they believe they have but they made a mistake somewhere just because it's difficult to do sometimes. So just something to keep in mind. Check behind yourself. ETHTools, great for doing that. Let's see, what else did I say about that? Yeah, yeah, SAR. We were talking about SAR earlier and it's other uses. I love what SAR does for what you can look for the network statistic. We looked at these earlier, I think for the, yeah. Now, so what I wanna say about this one is that if you wanted to be able to measure throughput, notice I'm doing SAR every second. So how many things are, how many packets are transmitted? How many are received per second? Well, you might need that metric but now you're doing it per second. Now you haven't. Just keeping that in mind. Like that's a way you could do it. You can average these out and do what you need to if you're not necessarily getting that metric where somebody is doing the math for you. You could do the math also just by running SAR every second. I don't know if I also just showed SAR before. This is just, you can say how often to repeat it. This is a, if you put one here, it's once every second. You also can enable live mode with SAR without having to enable historical data. Just a cool thing. If you just have it ready to go in the chamber, you don't need to set anything up in advance for it. My use SAR all the time. Couple other things that aren't, well they don't fit into the demo but I'm gonna do them anyway just so that I can say some things about them. There are a couple ways that you can check for latency on the network but let's say that we want to do to Ping, you could do, yeah, Ping is a valid way that you can measure round trip time of an ICMP echo packet. I just wanna say something about Ping. A lot of times firewalls, if it doesn't outright block them it will put them at a lower priority these packets that we're sending so you're not necessarily gonna get an accurate measure of what your TCP traffic, we're gonna shut that off. Trace route, same way, same way for trace route but trace route, so it will also get de-prioritized by the firewall. Sometimes shows each particular hop might be useful for you. You can, I'm just, I think I did eight, I was just doing it through the Google DNS, just eight, eight, eight, eight, eight, eight. Might as well, easy remember. But you can use, I don't know if I did this here but you can use the T flag on trace route and you can switch it to TCP traffic so you wanna get around where it's gonna be de-prioritized just switch it to TCP, you could do that with trace route. Just something to keep in mind. All right, moving right along. Now we're gonna get into, well, it's gonna say what you expected to say. Demo five. All right, I'm doing on time. I just wanna make sure I think we're, we started a little bit late but I'm going to make sure that we can, I'm gonna kind of speed through these so that we can make sure to be, get y'all to dinner and whatever you need to on time. Let's see. So this person said for DRBD, they extended the disk size. You can do that. Let's say that you've extended the backing volume through a variety of methods. Extend the file system because it's, you're able to do that, LVM. They made the backing volume bigger and then they resized the DRBD volume. You don't need to know necessarily how that works but that they did work for them. They said that worked, it got resized, I could use a space but now all my system is crashing intermittently. What do? I do. So what do we look at when I have a hunch? So let me show you. Oh, got him. How are we doing? Yeah. Thank you. Thank you. All right, so SAR with the B flag, that is going to be showing us particular, you see what we're looking for is a large amount of page outs. This one is not necessarily showing a lot right now because I think I did not do what I was expecting to do with this, but neither here nor there. Okay, so this is one that we want to look at. Also, if you have swap, not every, swap is not necessarily used anymore on a lot of dice. You might be like, well, it's not hitting swap. That's, a lot of things don't have swap anymore, but if you're looking for page outs, that's a high number of page outs is something to keep in mind. Look out for when you're trying to diagnose a memory issue. Sidebar memory issue, if you do a free, free command, that's a common one you use for memory, that can over count your memory because it also says you have the available memory of what is also shared between like you have a library. It's shared between different processes. They're all using that, but it's saying that you have more than you do because it's counting those multiple times. Just keep that in mind. It might not be a necessarily accurate measurement. I like to use the pressure stall information. So we can do that by cutting over here. Okay, so we're not seeing anything necessarily here. This is an average of 10 seconds, 60 seconds and 300 seconds, and it is where you have IO stalled waiting for memory. So if it doesn't have it and it's waiting for it, this is what it would show. It doesn't show anything right now. It might not show anything for a number of reasons. It could be a really recent spike. I would suggest you use also they have the totals. So if you had a spike in the past, then you can say, well it wasn't necessarily reflected in these measures yet. Take a look at that total, see if that's been increased. So you can take a look at it, even if it hasn't been now averaged out. Let's see, yes. Okay, so what was the problem? Well, I didn't actually run the problem because it didn't work the way I wanted to, but in this case, why did I suspect memory because they resized the DRBD volume. DRBD requires one megabyte of memory per tebabyte of storage. They increased the storage. They did not remember. Well, maybe I didn't remember, but they didn't remember. They could get the size. Maybe they were right at capacity for memory, then they increased, but now they need all that more. And so maybe they had enough, but now they've increased their map that they need to set up for. So all right, all right. One last one. And it is my favorite one. So it's the last. All right, so a client says that they have newly deployed cluster and it's not performing the way that they expected it to. There's higher IO latency that they expected to have. And I said, how did you run the test? Really important because how you've decided to write your test for disk IO latency will make a huge difference in how it is represented. So let's say, oh, there's disk two over here. Puff puff puff puff. All right. So there's a few factors that go on with when you have this hypothetical. Let's talk about working set size. That's the biggest thing and that's what this demo is ultimately gonna show, but I'm sort of speeding through it. In this case, you have a working set size that is very small. A working set size is super small can fit all the way just in a cache entirely. Cache, as we can expect, a married cache super fast. This could be a difference of a great magnitude. So like this could be, you could get latency of less than 100 microseconds. If it hit the disk because that how it could not, it was too big to be fully in the cache, had to hit the disk, then that's way different. That's, could be up to like eight milliseconds more even. But that's, the, what I'm trying to say is that the working set size is a multimodal distribution where you have everything that's in the cache and if you can be done in the cache, it's gonna be super fast. You have to hit, you have to hit the disk way slower. Just keep that in mind for if someone has a super small working set size for their application where all the memory it requires can fit in there. If now they've tested with something different, a different workload that cannot do that, it's going to be a much different situation. All right. So I am kind of, I wanna make sure that I respect everybody's time. So I'm going to get back to the slide, but I have everything sort of summarized on here. So I can still talk about it. I'm just not gonna be fumbling around on the command line. Imagine that, okay. So don't worry, like I'm talking about the same things. Okay, so random versus sequential is super important. Random, well, let's sequential first. Sequential is where something has been written. Where it starts is where the last right ended. So the offset is exactly at the end of the last offset. As you might expect for hard disk drives that are magnetic rotational, you're gonna have a lot of IO, disk IO is seek time where it's moving around. So less of a concern for SSDs, but if you have, you have just a platter-based drive. If you have a sequential, if you're doing a test with a, a lot of people will test with DD, which is only going to do like a single-threaded sequential write. Well, what if your application is actually, well, it's not writing sequentially to your disk in a perfect order. It's going all over the place, as you do. You can't really test for that with DD. You can with FIO. So FIO is what you wanna use for random writes. Is it the flexible IO tester? I love FIO because you are able to, not only can you do sequential, but you can also do random. You can, you can toggle various settings. The amount of, you can fork it into different jobs that you repeat in the same iteration. Doing a test multiple times, oftentimes you can feel more confident in the results that you've done multiple times or just one time. You gotta think also though about, if you warmed up the cache, that may be, that might be a factor. Ideally, you would be able to clear buffers and caches between, like let's say that you're just writing to it, especially if it's the same type of write, but we can't, we're in a production system. So we can't clear our buffers and cache that be rude of us. So we're not gonna do that, but keep in mind, a warm cache might give you different results. Overall though, if you're benchmarking in general, repeating the benchmark numerous times in numerous different settings will give you more confidence as a result. FIO is great because you could say just like num jobs for, perfect. Set it and forget it. All right, so let's see. One way that you can tell, this is how I believe that you should in general can make sure that, I'm just gonna put this up there. Yeah, cause I have there. Nope, I'm gonna go back. All right, I'll just have this all up here. Cause, yeah, the size of the individual writes. How can you tell if it's hitting the cache real fast?  IOSTat is enabled in the kernel by default. It's super low overhead, so you don't need to worry about running it. Just keep it running in a different tab while you do your tests. That way, because IOSTat is a tool that you really should be called like disk IOSTat cause it's only doing disk IO. So if you see activity there, that's, and you expected it, great. If you don't expect it to hit the disk, then it is, well, now you know. If you, conversely, if you're doing a test and you see it never hits the disk, you might wanna do a different test if you expect that this stuff is gonna hit the disks. All right, so just be real fast to be respectful of folks' time. The ratio of reads to writes. So if you have a metric that combines the two, be suspicious of that because your application has a different ratio of reads to writes in a lot of cases. Figure out what that is and test accordingly. But if you lump them together in a metric, you can't see the different results for reads versus writes so you can get less of an accurate metric to what's happening. Obviously, SSDs, as we talked about with the seek time, they perform differently than magnetic rotational hard drives. Also, they are not, I don't know if anyone's seen the YouTube video of Brendan Gregg shouting at a disk array. I would say it's super funny if you ever wanna check that out. This was back before they sort of did more sound proofing and he had a J-Bod in a server room and he just, ah! And then he can say, oh, look, the Iolate and see one way up. It's great, love it. Because of the vibration, because it didn't like it. Yeah, the disks didn't like it. And the rate configuration act, the rate configuration can make a big deal as well for what's happening with how things are written for your application. If you have parity rates or a RAID five, so like that's a, you've got a parity disk, you will have to do a lot more calculations for parity that are gonna cause a lot more Iol operations to happen. It's going to be less performant than let's say a stripe. So across a couple disks. Stripe size is another concept of RAID that can be configured. But the size of the rights can be matched to the kind of stripes of how the data is composed across that disk when you have your different disks that you are adding together for your increased performance. Match that stripe size to what your workload actually is. Find out what the size of your rights are and then you can get better stripes. If your rights are bigger than the stripe size on the disk, you might have some problems because it can't, now it's gotta move around a lot for that. We have size of the individual rights, that's not only that. A larger right is going to generally, it's gonna have higher throughput, but the latency is gonna go down. So just, they behave differently.  All right, I sort of flew through this one. So I'm trying to see if there is anything else before I go to the last slide because I know that I'm kind of going over. Yeah, yeah, okay. You could use the direct options to try and to account for not hitting the file system when you're doing your disk IO tests. You can do that for DD as well. Just keep that in mind. It might not be appreciable. Most of the disk, the IO you're gonna find is this IO. One way that you can check to make sure that it is actually disks that are the problem, your bottleneck. Check your total application latency for, let's say that we're doing a macro benchmark, talking macro benchmark again. And if the latency for the whole thing matches what you're seeing in IO stat, that's probably the disk because you just found it. There it is. It matches. If it doesn't look the same, it might not be them. Might not be them. All right. Okay, we're just gonna move. Yeah, we already talked about IO stat. IO weight is interesting because it's a little bit deceptive. IO weight is the measure of the time the CPUs are waiting for disk IO to complete. But keep in mind that that keeps, that's how busy are your CPUs is a huge factor to that. So let's say that you're, they're performing some super inefficient processing. You're doing bad things to these processors and they're working hard. Well then they won't notice that they're waiting for anything because they won't have time to do anything to wait for the disks. When it could be that you upgrade a system or you make a program more efficient, now you're seeing some higher IO weight and you're like, but I upgraded the CPUs. Why am I seeing more of this IO weight? Because now it has time to wait. But conversely, IO weight can go down and that's only because your CPUs got bigger. Your problem might've gotten worse, but you won't be able to tell. So you might need to, that sort of relationship is just something that you wanna consider. Any IO weight, keep an eye on it because that is very indicative. There's a bottleneck in the relationship between those two. And it's very vendor specific, smart CTL or RAID controller tools. Those are from the vendor, but you can pull them accordingly. And if you need to check individual disks to scrutinize them, then that's how you do. We're gonna go over there. Well, CPU profile, I think you could do that with Perf. We didn't have time to get that today. We didn't have time to get to a lot of things, but we certainly didn't have time to get to Perf and we didn't have time to get to EPVF. I genuinely, it's so exciting if you read about it though. So I would say these slides will be available for scale, so if you have not looked at the extended Berkeley Peck filter and the extensions thereof, definitely take a look. Thanks. Okay, well, overall what I was to conclude, what I said is that I've said a lot of things about DRBD and clusters. All of the tools that I've focused on and mentioned, those can be ran in pretty much any Linux system and hopefully there's some things that I've shown today. They might not show you everything, but they hopefully are directionally correct and sometimes that's as much as you need and you're just trying to narrow it down. You don't have a lot of time and you don't have the overhead to do a stack trace.  All right, well, I'm gonna definitely, if anyone wants to get questions, I have my business cards and stuff like that, we can talk on over email, but overall now I'm gonna, I'll let y'all go so that you can go get dinner. Yeah, hey. The CP, like are you talking about the IO8? Well, we talked about steel time. I'm not sure. I didn't, I don't think I mentioned, I mentioned steel time. I'm not sure. Steel time. Yeah, steel time is the one of the VM side. Okay, how do you spell that? S-T-E-A-L. S-T-E-A-L.  And so, steel space time, but it also is when you look at it on top or VM stat, you're gonna see it as S-T. But in the man pages, you'll see it as steel time. But that's what it's, steel time. S-E-A-L. Steel time. S-E-E-A-L. Yes, like I've taken something from you and you did not want me to do that. And you said you have steel, yeah, I didn't, I didn't do that for happy. Okay, and where would you see that? You would see that when you're on the VM level? Yeah, you can see it on top and you can see it in VM stat as well. There's a couple of other metrics. Steel time is shown in like a collective. Some of these that have a bunch of different performance metrics they show. So SAR is one you could also see steel time as well. Certain flags for different, you're tools of choice. You're gonna often see steel time, but this is what it's going to refer to universally. So just when you- Yeah, yeah, it's in the top section. It's in the top section of that. Yes. There's a lot on the top of the top, so I won't say it's the tip top, but I will say it's in the top of the top, yes. Yeah, of course. Yeah. I know that doesn't like report like network statistics, but what about NFS, which is just kind of storage place? Oh man, so the NFS, NFS has its, I didn't want to get into NFS storage either because that is a whole different beast. Yeah, because NFS it, yeah, it is separate, but I would consider that a logical IO. So that is not physical IO. And so IO step would not be reporting on that. You would not see that. So you would only be seeing physical IO, IO of the discs of the movement, logical IO, which that would be covered under.


Hi everyone, Joe for jaspyscasebreaks.com coming at you with a good one 2020 Panini National Treasures football two-box half-case number two from jaspyscasebreaks.com Random team break everyone gets a team in the NFL big. Thanks everybody who got their spots straight up and Congrats again to the people who won their spots in those filler packs always look out for those on jaspyscasebreaks.com all 32 are in Let's roll it. Let's randomize names and teams two and a three five times one two three four and Fifth and final time we got PJ all the way down to Chris All right two and a three five times four the teams one two three four and Fifth and final time we got the buccaneers down to the Cleveland Browns All right, so good luck. Here's how it shakes out boys and girls appreciate Everybody getting into this one PJ with the Tampa Bay Buccaneers Eric with the Steelers Matthew with the Cowboys Ryan with the Ravens Matthew with the Titans Eric with the Falcons Craig with the Pats Ofer with the Lions the X-Line with the Dolphins Kevin with the Niners David with the Colts offer with the Jaguars Paul with the Packers David with the Texans Robert with the Rams PJ with the Cardinals Matt with the Eagles PJ with the Chargers and the Broncos Patrick with the Saints PJ with the Bears Chris with the Washington football team last bottom mojo Eric with the Panthers Craig with the Vikings PJ with my Raiders Shane with the Bengals Jonathan with the Giants Jets for Chris Raymond with the Bills PJ with the Seahawks David with the Chiefs and Chris Maxwell with the Cleveland Browns Let's sort this by column B Now I will pause the video and give people a little bit of a trade window, but Trade at your own risk. I'll be right back All right, welcome back everybody. There was a lot of trade chatter, but in the end. No deals were done on this Saturday evening Thanks everyone for joining us and making this part of your weekend Let's roll. Let's see what we got in here Some kind of treasures I've been hiding in here. So remember I marked those RT2 so we know that they're from the same case Dodgers at Angels on in the background Kershaw in the mouth facing Ohtani and maybe trout after this. There is the blockchain card and it's Blockchain the future Chase young so that'll go to Washington Chris last spot mojo. Let's actually put a Blank card right there and let's see what we got Good luck everybody Start off with Charles Woodson to 35 that'll be for PJs got my raiders Got DJ more 72 out of 99 Hey, no worries Chris. Good to see you back in the action Eric with Carolina. I got Carson Wentz 13 out of 99 century materials For the Fly Eagles fly Matt Buckler Maybe there was a tried to trade couldn't trade mojo situation there. We got colossal Relic Minka Fitzpatrick That is 10 out of 99 for the Steelers Eric Houston with Pittsburgh Hershaw versus trout one and two Tremendous treasures David Montgomery 23 I 25 for the Bears PJ with Chicago 11 17 out of 25. That is Justin Jefferson two-color dual relic for Craig Thomas John Hightower the fourth 14 out of 35 For the Fly Eagles fly I tried to trade couldn't trade mojo from that Kersh and we've got James Robinson rookie Jersey and autographed 26 out of 99 for the Jags Oh for Abramson with that one the new look Jaguars next year They'll be the they'll be the pricey team next year Now Michael Vic 14 out of 25 The dynamic Michael Vic Atlanta Falcons Eric Houston With the peerless on-card autograph One more and a redemption on the bottom nice cam acres RPA 87 out of 99 Three-color patch on card autograph. He's got some good penmanship there Robert Flores with the Rams Remember ladies Jim we are using larger top loaders just because of the The top loader shortage out there, so we're gonna put blue painters tape right there But just be careful and you're taking them out of the packaging case the tape has come off or you're switching toplers or whatever All right, let's see what we got here rookie patch autographs Hard number 182 Who can you be? It's See Chase He's often a redemption chase clay pool I'm gonna say young but chase clay pool That's Eric Houston with the Pittsburgh Steelers. All right. See you Thomas. Thanks, man Chase Claypool. All right next box Last box so just a quick little two box break another two box break from a fresh case in the store right now Jasky's case breaks Com packed with hits here. What's going on here? There we go. Oh another redemption on the bottom And he guesses on that redemption. Here's a blockchain redemption Joe Burrow Nice rookie treasures NFL shield Shane white and the bangles. I I think it's all aboard the big hit express whoop whoop NFL shields usually a one-of-one, right? It's got to be this this one Actually do pretty well in the secondary market. All right. There you go Shane white bangles All right last box Good luck. Joey Bosa to 25 Chargers that's gonna be for PJ Tyler Murray 13 out of 99 Cardinals ace earned 2019 rookie of the year award treasured moments for that is a treasured moment for him PJ with the Redbirds Jalen reyegor for the Eagles That is 61 out of 99 for the fly you go fly Matt Buckler Julius Peppers 4 out of 99 Jersey Carolina Eric Century materials 7 out of 25 Debo Samuel 49ers Kevin With San Francisco We got a quad relic 8 out of 10 a quartet of chargers Justin Herbert Keenan Allen Mike Williams Hunter Henry Hunter Henry the only guy that's Not there anymore for PJ and the Chargers Last bits here. We got Trayvon digs 10 out of 25 10 out of 50 that is Stefan's brother. That's gonna be for Matthew Shearer and the Cowboys Couple more and then the redemption Brian Dawkins 8 out of 15 Another one for the Eagles. They were out there up for trade and we've got Jersey and autographed Marshall Falk He's pretty good 9 out of 25 Jersey and autograph Going old St. Louis Ram going to the current Los Angeles Rams. That's Robert Flores and the redemption Good luck everybody. No one had any guesses It's a prodigy patch Autographs tag. Is that a is that a train whistle? It sounds like a laundry laundry tag. Let's go to cardboardconnection.com search 2020 national treasures football Checklists on cardboardconnection.com. It's a great resource for everybody if you want to get that Darren Porter is guessing Chinalt Prodigy patch, where are you? Checklist is pretty huge Prodigy patch autographs Gold to 25 brand logo one-of-one team logo one-of-one So I would assume that Prodigy patch autographs tag must be like some sort of brand logo We'll give it a train whistle TJ guessing Jim McMahon Chris Maxwell guessing Chase Young. It's gonna be card number 13 C Ly I think we know who this is. It's Clyde Edwards E. Lair Prodigy patch autographs tag. David Garcia, Kansas City with a spot that he got straight up David All aboard the Big Hit Express And there you go boys and girls that's a very nice break 2020 Panini National Treasures football two box half case break number two Break three is from a fresh case.  I'll see you next time for the next break. Bye. Bye


Hi everyone, Joe for jaspeaskasebrakes.com coming at you with a quick little pack of a 2022 Panini Donners Optic Football, random team break number one. This sells off the break because we're giving away 11 teams from a Donners Optic Football. Six spots, pick your team number one coming up in the next video. We'll do the pack break itself first and that'll be very quick. Then we'll re-randomize everybody's names. Top 11 will get teams in that order. Big thanks to this group for making it happen. Appreciate you. All NFL teams are in. Let's roll it, randomize it. One and a three, four times. One, two, three, and four. Four times for the teams. One, two, three, and four. All right, there's the first half of the list right there and the second half over here. No trade windows, not in this little quick pack break. All right, we've got a parallel. We've got six out of 15 Tariq woollen. Rated rookie, cracked ice, Carlos and Seattle. All right, now let's flip back to this list here. Let's see who's gonna get teams in the next break in the next video. Let's gather all these names here. There's everybody. We'll match up with the teams right there after two and a two for the hard way. One, two, three, and four. Now from 12 on down, sad times. 12 through 32, John, you got close just not quite this time. We'll get him next time. 12 through 32, thank you. Sad times though. Happy times for the top 11. You've got teams in the next break. David is happy. So is Chris, David, Martin, John. There you go, John. Carlos, Brian, Adam, Brian, Adam, and John. Congrats to the top 11 right here after for the hard way. Let's match you up with the team. All right, so John, you got the Steelers. Adam with the Falcons, Brian with the Bills. Adam with the Bengals. Brian with the Seahawks, Carlos with the Dolphins. John, you got the Panthers. Martin with the Jags. David with the Bears. Chris with the Eagles. And David with the Cardinals. We'll put Opt Pack One next to your name so you know that's where you want it. I'm Joe for jaskiescasebreaks.com. I'll see you in the next video for the break itself. Bye-bye.


Four Bemidji residents have been sentenced in federal court for their roles in a home invasion on the Red Lake Indian Reservation. United States Attorney Erika H. McDonald announced today the sentencing of 22-year-old Derek John Paddy, 32-year-old Kevin Robertson, 26-year-old Cheria Michelle Clark, and 29-year-old Shanice Francis Osland for their roles in a burglary on the Red Lake Reservation in the early morning hours of November 24th of 2017. Osland and Robertson were found guilty of burglary in the second degree and end of robbery following a jury trial. Osland was sentenced to three years probation. Robertson was sentenced to 51 months in prison. Paddy and Clark each pleaded guilty to one count of burglary in the second degree. Paddy was sentenced to 29 months in prison. Clark was sentenced to 21 months. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


We're very proud that this week the Royal Mail has issued as one of their new collection of stamps, a stamp of our founder, Bard Reward, who would have been a hundred this year had she still lived. It's enormously pleasing to see her recognised in this way. For us we know that she was a real star, but now people much more widely can find out about her and the ideas that drove the foundation of IIED. These are ideas that have finally become recognised as hugely important. That's to say that the only way in which we can live on this shared world is by addressing the environmental constraints of a single planet, but at the same time sharing the prosperity much more widely across all of the nations. I never met Bard Reward sadly because she died in 1981 and I didn't join IIED until six years later, but everybody I've met who did know her was struck by her incredible resolve, her moral fibre and most of all her ffarsightedness in terms of her capacity to shape the ideas that today are now being recognised. So I'm going to buy lots of stamps and sending lots of letters and I hope you will too so many more people will find out about Bard Reward.