The Tug Fork Tire Tug of War cleanup is back. Due to inclement weather, cleanup efforts were rescheduled earlier this month. Friday we spoke with Tug Fork friends member John Birchit about their recent success and their new cleanup plan. 819 tires in one day is a pretty good achievement for us. In 2019 we got 2321 tires with four days work. So 819 in a day is really good. We were happy with that. It was raining on us that day. The river was coming up on us that day. The water clarity was going down on us that day. But we managed to get a good day's work in and we were happy with the results that day. Even with over 800 tires out of the river, more is needing to be done. This September 20th Birchit and volunteers will continue their cleanup efforts and they need your help. Things are changing a little bit. We're going to set up mostly at the South Williamson boat ramp by Beside El Azul's restaurant. We'll have some different river access. We're not sure exactly where that's going to be just yet. But we're going to be working probably this part of the river and on down. So look for us. If you come out to volunteer, look for us at the boat ramp there beside El Azul's. And we might want to go somewhere else to actually get into water but we'll have instructions there as to where we're actually working. Here at Pikeville Medical Center's Hart and Vascular Institute, we have assembled a comprehensive team of cardiac specialists bringing expertise from all regions of the nation and the world. We have coupled that with cutting edge technology providing them the best equipment and operating rooms available. The result is comprehensive cardiac care for the people of our region that is second to none. The Hart and Vascular Institute at Pikeville Medical Center. With the cleanup efforts in place, Birchit says it'll help the area's tourism progress. We got approval of our West Virginia Flatwater Trail for the length of the Tug Fork River that stretches along Mingo County. It's a 60 mile stretch of river that is now a West Virginia Flatwater Trail. We named it the Bloody Mingo Flatwater Trail after all of the bloody history of Mingo County. It has nothing to do with the water itself. It's a flat water float trip so there's no white water in it. But Bloody Mingo, just to honor the history of Mingo County, all the bloodshed in the early days of our county. Again, the Tug Fork Tire Tug of War cleanup will be held outside the flood gate near the Many Park on September 20th and 22nd from 9 a.m. to 2 p.m. For Mountain Top News, I'm Joel Korgel.


We've got Jos van der Oever, who's going to talk about Rust with QT. Please give him a really warm welcome. Well, thank you very much. I'm happy to finally have made it into this room. It turned out you have to be a speaker and then it's easy to get in. So, I assume that most people here are firstly Rust developers, and maybe secondly also there's some cute guys here. So, who is a cute developer here? Just to, oh great, quite a few. Yeah. So, some of the stuff will look more familiar to you than most of the other stuff. So, yeah, for the people who don't know Qt so much, I've also added some code examples there. Since there are not so many cute developers, I think also this image here, who has ever seen this image before? Two people, yeah, I guess three even, right. So, this is like a logo which was on the top of the documentation of Qt 1 there about. So, it's like somewhere in the 90s. But considering that combining Rust and Qt is not so easy, I thought this sort of man looking a bit distraught would be a good opening image for this talk. But the good news is it's actually totally feasible to combine those two things and the Rust Qt binding generator will help you with that. To start us off, well, a takeaway. I think this talk would be good for Rust developers, for Qt developers, for people developing Qt because I think they can learn a lot from the Rust language. But also people who are interested in writing a native Rust UI, learning from what Qt has done over the past 20 years could also be very insightful. So, for them, I hope this talk is also one that has some nice messages. So, here are three applications. This is a tiny one that I blogged a while back and which was also on the Rust reddit. As a tutorial, this is basically a QML application. I'll explain later what QML is that is powered by Rust. And all that Rust does here is it has a thread and it gives some seconds every second there's an event and then this Rust logo rotates. The Rust Qt binding generator repository has a demo application which showcases all the different styles of widgets you have in Qt. So, this is just a different styling, but you can also use Qt Quick, which is a type of widget that is very much optimized for your graphics processor. And we also have Qt Quick controls too, which is more in line with what you see on mobile devices. And all of these things are bundled into one application here, but later on in this talk we'll mostly be talking about this demo application we're going to walk through how to write code like this to power a to-do application with Rust and Qt. So, back to the slides. The goal of this project was to find a way to write programs that combine Qt to Rust. But since I was doing it in the summer as a hobby, it should be a small effort to create this tool. But when it's finished, it should also be a small effort to actually use it. It should be easy to combine these two languages, even though C++ and Rust are so very different. It should be simple with minimal dependencies. And this is very important to me.  So, what is Qt and where is it used? These are logos from some of the projects that are using Qt. Here's one that Linda Storwald helps coding. If you'll see, you all know. The KDE community is basically built on Qt. The Plasma desktop uses Qt. You can write 3D applications with Qt. And there you have a sort of state machine to steer everything. So it's a very rich, very large set of libraries to write your user interface in. And therefore, it would be great if you could use it with Rust. Also in the embedded world, Qt is used a lot. Here's a picture from a promo video. And you see that they're pointing the Qt logo to lots of parts of the car. Not exactly sure what they mean by that, but I think the overall message is we have some low-power processors and display on it and we power it with Qt. So you can also put it on your fridge or on your thermostat, whatever. There are tons of bindings already for Qt, lots of different languages. And yeah, why do you normally have bindings? You want to use languages which is more familiar to you. Qt itself is written in C++. And maybe you're more familiar with one of the languages on top here. Or you have a large project. And maybe all the guys from the lots of logos before, those are existing projects, are interested in adding some Rust into their mix, into their projects. So you might want to combine existing code with it. And then you also need a library. The typical approach to do a binding is to take the Qt C++ base, build a binding on top, and then put the other language there. And that's not the approach I'm going to take here. And I'll explain later what we are going to do. Well, I guess you know these advantages by heart. Why is Rust so cool? It's very safe. One of my favorite things of Rust is the algebraic data types, also known as simply the enums. That's something that C++ simply doesn't have. I think it's great. The macros are very clever. Cargo is great. But all of these things are not really there in C++, at least not in the way that Qt uses them. It's a great library, but these innovations in Rust simply haven't made it into Qt yet. So if we're going to make an API of Rust on top of Qt, then we're leaving a lot of advantages on the table. So that's why I think that wrapping a Qt API in Rust would be like fitting a square peg in a round hole. That's not going to work. But there's a solution. What we could do is just shine a light differently on it. And well, here's the square. Here's round. Great. We just put some JSON in the middle. So yeah, how will that work? OK, so we're going to do the binding in two directions. First, you're going to write your data model in JSON. And I'll show you later how to do that. And then you generate bindings from that. And these bindings talk to your user interface and to Rust. And all the purple boxes here are stuff that you have to write. So if you came in here thinking, I'm going to write a user interface only in Rust, that's not what this talk is about. You still will have to either use QML, which is a very nice declarative language, or Qt, which is C++, on top of your Rust. But the advantage is, if you take this approach, then you have separated the logic of your application very well. And if you want, you could also put then a different UI on top like HTML. So this is the approach we're going to take. Now I need to explain a bit about the core principle of how Qt works. The basic thing in Qt is QObject. A QObject is a class. And you can give it signals and slots. And outside of Qt, people call these things events and listeners. And so you combine, build a loosely coupled system. And it's loosely coupled if one of the objects is destroyed, then the connection is broken. It's the same as what you have in JavaScript. Imagine. So it takes you to have memory manager because you have hierarchical ownership. You just say, when you create something, this is the parent. When you destroy it, the whole hierarchy is destroyed. And objects have properties. Now there's another important building block in Qt. And that's the Q abstract item model, long, complex name. But basically what it is, it's the way data is represented from your logic to the user interface. And it's used everywhere. So here is a table and a graph. And there's one data model underneath, which is an implementation of this model. And it's used in two places in a very different way. And here you see a table. I'm not sure if everybody in the back can see it here. But one of the cells in the table shows a percentage. And it's filled partially. And that's also a trick which is implemented on the UI level. But the data underneath is provided by this abstract item model. And you can also have trees. So this is used everywhere in Qt. But it's kind of hard to implement, even if you just do it in C++. So the tool that I'm presenting is focusing very strongly on making it easy to do this. So yeah, basically you can even forget about this. And just know that we're going to build a model which gives you all of that. OK, so how is it going to work? We write a model description in JSON. And then the tool would generate a binding.h and a binding.cpp file, which we use in our C++ code, and an interface RS, which will contain a trait, a Rust trait, which you have to implement.  And you're either imported with binding.h or if you're using QML with import Rust code. The Rust code is a name you can think of yourself. This is what I use in my examples. Briefly about QML, Qt modeling language. This is a very nice innovation, which was triggered when Nokia bought Qt. They wanted to make phones with very performant and easy to write user interfaces. So what they did is they came up with a declarative user interface language. And it can do 2D programs or 3D even. And it's used in lots of mobile applications. So it started off at Nokia, Mego phones. The Ubuntu phones are using it everywhere. Silfish, KDE Plasma desktop, KDE Plasma mobile. And you can also use it for iOS and Android apps, if you like that stuff, which apparently is the majority of the people. It's designed for efficiency on graphics cards. And that's actually, when we're talking about Rust, is not unlike WebRender, which has quite a lot of similarities in the way it's designed. OK, so now we're going to write a small application. So you understand a bit more of how all of this works together. This is a to-do application. And this application, there is a website called to-do-mvc.com, where the same specification is implemented in 20 or 30 different JavaScript frameworks. And I took that specification and implemented this with QML and Rust, just because I didn't really know what I should write. I thought, well, here's a specification already given. Let's just implement that. And I can look at the application here a bit. So what else should I need to do? Well, keep talking, I guess. And you can tick off stuff here. And you can just see the active parts, or only the completed ones. I can clear the stuff that's completed, go back to all. I can toggle here everything. So it has some basic functionality. And the UI part of it is about 150 lines of code. And the model beneath is fairly simple. So if you want to build this, how do we start? First, we need to write the binding, where we describe our model. Now, the object we call totals. And the type will be a list. We have a list of to-do items. What you can put here is either an object, which is only one item, or a list, or a tree. It will have properties. And the properties should be a count of a number of items and the active count, which is the number of items that still need to do. And every item in the list is either completed or not. So this is a Boolean field. And it has a text, a description. Probably some people here are thinking, why not just write a Rust macro? Well, maybe we'll do that at some point. But for now, it's just a JSON description. OK, that's checked off. Now we need to run the generator. So that's just one executable. You call it with the file. And this is set up like this so that you can integrate it with CMake, or with Cargo, and build RS. And this generates these three files that I mentioned. And then what I need to do is I need to write an implementation.rs, which is a Rust file. OK, now we're going to look inside the three generated files, and they're going to look a bit complicated, but you don't have to do that. It's just to show you a bit of how it goes, what goes on underneath. So this is a bit of C++. It's the class to do. And it implements abstract item model. It has a constructor, a destructor. It declares some properties. And it declares also some functions for setting and getting the values. So this is fairly simple. And there's an implementation along with it. So here is the foreign function interface that talks to Rust. So these functions are declared in the C++ part and should be implemented by Rust. You also won't have to do this, but this is the point where the two languages meet, where they connect up. And here is a bit of example code, which is typical for this data model. You have the class, and there's a function data. You give it an index and a role. And the role can be I would like to have a display, or a color, or in our case, the description, or if it's completed or not. That can also be your role. And then here's this big switch statement, which calls into the Rust code. So it calls to Rust and asks, get me the completed version for this index or the description for this index. And again, all of this is generated, so you don't have to worry about it. But how does it look on the Rust side? Well, on the Rust side, we have a trait called the to-do's trait. And it has a constructor, which you need to implement. And it has an emitter. The emitter is there so that you can emit signals when something has changed. So you can signal to the user interface that something has changed. For example, some item was added to the list. There's functions for inserting rows. And here are the functions to get the value if number 1, 2, 3, or 5 is completed or not. And you can set it to be completed, or you can get the description or set the description. So two items left. And this is the work that you will have to do when you write an application with Rust and Qt. So we need to implement the Rust part and the Qt part. So for the Rust part, we are implementing a list. The list has items. So we just have a struct with a Boolean and a string, so far so good. And then we have the object that contains the whole list of these items. And it also contains an account of how many items have not been finished yet. This is a cached value. And it has two objects to talk to the outside. Now we implement the trade. And this is also fairly simple. We need to get the value for the active count. And we just return the value also for the total count. We just return the length of the list. And when we want to know the description at a certain index, we just give back that value. So this is quite straightforward and simple. And this is the part which you will have to implement when you are writing a connection between Rust and Qt. So how do we do the last bit of it? And that's the QML part in our case. We have created this object. And this object, we want to expose to this Qt code. So we import the Rust code. We have called it version 1. Well, you can give it any number you like. The application window is basically something between brackets here. So this is QML. If in case you've never seen it, it's a fairly simple language. And here we have our object. So this object here is what we have implemented with Rust. And we give it an ID. So all of our to-do information is in this object. And we put it here. It's not a graphical thing, but we put it here so that we can access it from our graphical interface. And our main graphical interface is below here. That is a flickable list view. And it has a model. The model is the to-do model, which we have defined here. And we have a delegate. And the delegate is the UI item that we show for every item in the list. And that's defined here. So it's a component. It has a pane. And it has a checkbox. The checkbox is a checkbox checked or not. It's a declarative language. It is checked if the value of completed is true. So this is the information which is gotten from the Rust side. And this description is also gotten from the Rust side. But in here, it just looks very natural. You just write the name of the value you want. And it's automatically assigned to this text here, to the text of this label. Yeah, and that's it. So our application is done. So in conclusion, if you want to combine Rust and Qt, you should not, in my opinion, at least shoehorn the whole of Qt into a Rust API. But instead, you should wrap your Rust code in a Q object or a Q abstract item model, which you can then naturally use in your Qt or QML application.  That's it. Yes, yes. OK, so the question is, how do you deal with different threads because Qt must have a user interface thread? Well, it does. If you have a Qt application, the main thread, the thread that you start, is the Qt thread. And if you want additional threads in your Rust code, you will simply start when you construct your object. You start a separate thread. And that is then alive as long as your object is kept alive by your Qt code. So you can do any type of number of threads. You can keep them running. And the signals you send to the user interface, those are thread safe. But you don't have to worry about that. It's automatically taken care of. What you need to know is that in Rust, you can just start as many threads as you want in the object that is generated for you. So in your implementation.rs, you can go wild with the threads you want. Yes. The question is, how do you deal with build systems? Do you take a lot of advantage of cargo or not? Basically, you're free to do whatever you like with your build system. All you need to do is call this binary to generate the code for you. And we have some templates that have CMake which call into cargo. But if you don't like that, if you would like to put everything in cargo, you're free to do that. But this project doesn't currently do it. But it's a matter of just calling that in build.rs in cargo. So it's not in the box. But you can do it. Yes? Yes. The question is, how do you do asynchronous work? Like with Hyper, for example, or other Tokyo things. That works perfectly fine.  And that is running in your Rust thread, then, which you create yourself. And there, you set up everything for asynchronous IO. And Qt is not involved at all. Whenever you get asynchronous information, you can decide if it warrants a signal to the user interface. And then the user interface is updated. So it's thread safe. And it's kind of separate. Yes? What was the code that was instantiating the QMF scene and exposing all the C++ classes to QML? The question was, where is the code that exposes the Q object to the QML? Is that the question? And it's initiating the QML scene at work. The initiating the QML scene? Yes. Yes. OK, so the question is, how did you create the QML scene? And how did you tie the Rust stuff into QML? That was a 10 line C++ file, which you're right. It didn't show, but it's mainly always the same. You initiate your QML scene. You instantiate. Sorry, you register your Rust object into the QML scene, and then it's off to the races. That's not generated. You have to do it yourself. Yeah, but it's always the same. So you can copy the file from the template. Yeah, yeah. Yes? Right. Can you repeat the first part of the question? Yes? Yes. Right, yeah. The question, again, is how do you register the Rust Qt objects to the QML? And can I do it live as well? The answer is the same. You do what you normally do on the C++ side is you have a Q object. You instantiate it, and you register it to the QML scene. And yes, I'm pretty sure you can do it live, although I don't have an example which does that. Yes? Have you considered doing this with derives or something? Yeah, the question is the implementation looks pretty straightforward. Have you considered doing it with derive? And I assume that you mean derive on the Rust side, so basically a macro. I'm not entirely sure how derive is implemented, to be honest. It's basically just Rust code generating Rust code. OK, yeah. So could you implement this in Rust as an annotation on a class? You probably could. Yes, you probably could. I simply haven't done so because initially I wanted to be very hands-on on the code that was generated. In fact, all the demo codes and the testing codes, which is in the project, which is generated, is also committed so that whenever I change the generator that I have a sort of testing that the code is still the same. And if I would have done it with a derive, then it would be only there during the compile step. I would need to do some different things to actually check, but it's perfectly possible. Yes. Yes? Oh. Just on the outside? Yes, that's fine. Yeah. OK.


Welcome to another edition of RCE again. This is Brock Palin and I have with me Jeff Squires from Cisco systems open MPI and also HW loke which happens to be our topic today Yeah, we're kind of cheating on this one. Aren't we? Yes, a little bit interviewing one of the other developers from HW loke even though I'm one of the developers too But if anybody looks at the subversion commit log You'll see that this guy has written a whole lot more of the HW loke code than I have as a matter of fact I I believe it was part of his thesis work and whatnot. I've been doing you know ancillary build system configure system And he's been working on the heart of it. So he can speak much much more intelligently about this stuff than I can So we're cheating but only sorda Before we introduce our guests we have a website RCE dash cast calm there you can nominate new shows pick up old ones subscribe to itunes There's an RSS feed. So please stop by there and pick up any back shows Also, I gotta I gotta throw in the plug for my own blog the MPIB cast blog out there that It's on blogs. That's this go calm and there's a link off the RCE cast and I try to get about one entry a week out There and I try to say something at least Nominally interesting last week. I talked about traffic and my my act my favorite acronym that I'm trying to get people to use Nuna non-uniform network architecture Because it's all about the networks these days. There's networks inside the servers, right? So makes it more complicated It's good stuff Nuna start using it in casual conversations Well, actually the software package we're talking about today HW loke is for kind of understanding the topology inside of a server I believe so let's go ahead and introduce our guests and he can give us a better idea. We have Samuel T bolt Who's from France so Samuel how about you introduce yourself say what are your work and how you got started on HW look? Yeah, sure. So I'm a researcher in Bordeaux between the University and the Inria Institute I'm working mostly on runtime support for machine parallel machines and and So beat multiple accelerators based up So that's why I was interested in in topologies and locality in machines On the side note. I'm also a deviant developer focused on the accessibility topics in particular for blind people So I feel compelled at this point to say that the standard disclaimer that we're ugly Americans and we're horrible at pronouncing Other people other language names and so on so Samuel Can you pronounce your name as as you say it because I'm quite sure that we're not saying it properly? In French we say Samuela with a U which is difficult to say in English indeed Okay, we'll try but I but I can't problem So Samuel, can you go ahead and give us a rundown of what HW loke is we just mentioned Nuna and networks and just give us a 10,000 foot view of what HW loke is and what it does Sure, so It's actually a set of tools to detect how HPC components relate with each other. So how Processors share cash on memory Network boards, etc. Things like this and then tools to bind the HPC applications accordingly So it doesn't provide the intelligent part of the mapping, etc Just the portability of the tools to actually discover and pin things around Although we do provide some examples for people to get inspired from So actually let me extend that definition just a little bit. I I think this is actually more wildly applicable than just HPC I think there's quite a lot of applications and enterprise and elsewhere that you know as We're going to more and more cores in a system They actually care about this stuff. And so I think this is not just an HPC Kind of thing. I think this is where it's gonna we're gonna cut our chops and make our name But I think it's gonna be wildly wildly popular after that just my opinion. No Yeah, I mean modern machines now you're looking at 48 cores and like a regular to you server So it's getting kind of crazy. So Samuel, can you give us a? A Where the name for hw low came from I guess this is an interesting story Well, it's actually a very long story The thing is you at the start during my PhD was thinking about the leap topology simply The thing is there was another leap topology that existed afterwards So we had to change the name and we had a lot of discussions We can find on the list and on the wiki the various names we Invented some very difficult to remember some very difficult to Google and eventually we thought that just hardware locality abbreviated into hw log Does Google quite fine and why it's good enough for what we needed So why exactly I guess Jeff you could probably answer his question too because you've wrote a number of blog posts about this Why is hardware locality and process affinity important in modern systems? So mostly from our part we we believe them Important thing is and portability of performance They are for instance HPC libraries like to and buying threads to get them better performance But they need a portable way to do this because it's not standardized So hw log does provide it More than that You need to be aware of the architecture of the machine because if you consider it as a flat set of CPUs Then you would get congestions on the links between Processors so it's better to split and distribute the application according to the underlying machine. So be it for communication communication between Processors or spin locked strategies and talking with network boards and things like this So give another disclaimer here, I think I'm gonna go back and forth between being interviewer and interviewee So everything that Samuel just said let me add just a little bit more on that is that with these you know more and more cores in a machine what is easy to forget is that there's actually a network inside that machine and You know if you're not careful You know most people don't think too much You know when they're when they're writing in an enterprise class application They just write their application They don't care and it gets some baseline performance and if they need to they optimize it a bit to get to the performance to where it Needs but with these larger and larger machines. It might be Pretty easy to shoot yourself in the foot performance wise because you're not careful and you don't realize that there's actually communication going on Just in writing a normal serial or even a multi-threaded app and that communication could get pretty intensive and you could be really Hurting yourself or hurting the overall per machine performance if you're not aware of the locality of your processing and the locality of your data and so You know HW look is really all about making The topology of the machine available to the programmer so that they can make some intelligent decisions about you know How are they gonna run? And how are they gonna? You know organize their data locality. It's just like real estate. It's location location location Yes So does HW look just give you information about topology or does it actually give you the functionality to do affinity? Well, it does both the thing is it separates the the two notions So that applications can just sit between the two And decide according to what they need the problem is that you cannot really build a tool that Say distributes your application of a machine because it doesn't know how the application is structured So that's why hardware locality provides the structure information. Let's the application do whatever Interesting heuristic Before for the between the application structure and the machine and then use the tools provided by HW log to actually enforce some binding or some some strategies for communications So what kind of information about the topology does it give you is it just the number of cores and which cores Share a socket or does it give you more information than that? Well the on the basic information that I wanted to put into it when I first designed it was To provide the tree of objects in in the machine So basically the base structure for it is a tree of objects the human nodes sockets caches things like this and then there are some attributes for instance the cache size or memory size and Then from that structure you can for instance anywhere turn on the number of sockets or No, which cache is shared between two and processing units and things like that So the idea was to have a very simple structure that people can use the way they prefer So how is this information delivered? You know if I'm a I'm a programmer or I'm a system administrator or something, you know How do I how do I use the hw? What's available there you have several ways to access it actually either you can Just use the tree so you start from the root which is in the whole system And then you go down to find out there are some human oats and then inside human oats You have sockets and inside etc or you can Browse it by level Say you look at first the system then you see that there are there is a set of human oats You know there are four of them and then you see sockets. You see that there are 16 of them In total and you do not consider necessarily the inclusion between human oats and sockets Or you can directly access to some level for instance a lot of applications may just want to know Which processor is included in which? Human oats will be accessed directly to the new mode level and then see from there which processors are available All right, so you're talking about data structures. So this is a C API, right? Yes, so it's basically structures with pointers pointing at each other and an array of arrays to to get access to Particular levels of the same kinds of objects But if I'm a I'm not a programmer, I'm I'm say I'm a system administrator. I'm a scripting kind of guy There's there's probably some tools available for that too, right? Yes. Yes, the hardware quality was usually meant to be a library But it's also quite useful for us for instance when we buy a machine And we want to know the structure of the machine So we have a tool that provides the structure in a graphical way We also have tools to get a textual output so you can mix that into scripts and get the CPU masks and to then apply them to to some processes or use CPU sets Etc.  I was actually on a support call this morning with one of my customers I can't tell you who the customer was but they're a very large disk maker and their name rhymes with Schmidt app and I We were doing some things on some Cisco servers and they wanted to verify some processor settings And I said well, hey, you know, there's this great tool called HW low Can I said oh, what's HW Logan? So we downloaded it right there and built it took about 30 seconds And we ran it and they I have to tell you their their reaction was just almost comical to me They said holy criminy. This is great We need to get this installed on all of our machines and they were just going off They're like wow you can get it in XML to holy criminy we can script this up They were just thrilled with it and it was just fabulous to see that kind of reaction. I Can think of some applications of Vasp is the one that comes to mind It's a Fortran code, but normally it's easy to make a wrapper around C where when you compile it You need to tell it a preprocessor of how big your cache is for their FFT routine and then it's a static thing Could they just use HW look to kind of ask? Hey, how big is the cache on the thing? I'm running on Yes, so what you can do is just ask I do HW look The structure of the topology, but then you can browse it and see the various cache levels from the L1 L2 L3 and Whatever cache the levels there might be in the future and you get the size and how many processes are sharing it Which can be important where when you actually run some some computation there So a little bit of a clarification The affinity ability to use HW look to kind of pin Say executable things on a Linux system. Is it working with processes or with threads because they can be different Well on Linux we support both actually them There are some operating systems where we do not support all kinds of bindings, but on Linux we do either Bind threads individually or we can just bind all the threads of the process Be it the current process or another process So you said other operating systems. What operating systems is HW look support? Well, actually quite a lot there as many as possible probably there currently we support Linux Solaris Windows AI X and Darwin Free VST or SF that is true 64 HP UX and carry get so we are basically missing iric support Well, just because we don't have an account. Maybe some somebody listening to do as we'd have an account for us Thing is our code is quite independent from the architecture itself beat x86 or Itenium doesn't really matter Except in the x86 case we have some code for operating systems that don't provide enough information in that case and the x86 case and We can request the information from them processes itself So that's actually pretty impressive because the affinity stuff varies per operating system not every operating system uses it a standard way of specifying affinity Actually, the the idea of the library didn't came immediately the During my PhD, which was about mapping trees of threads on the tree of the machine Made me write on that topology discover a thing so that was between 2005 2006 I wrote the on portability code there Because I needed to to run that on other systems and Of course quickly I found out that it would be a good idea to make a library So I did it to my to-do file make a leap topology and that happened only three years later When people from my team Put some demand on this for and Priya MPI process mapping So we did it eventually during 2009 and then it became a hardware locality So what what kind of entities can HW loke? Detect so we've mentioned a couple already, you know pneumonodes and caches and things Well, what what is that? What is the full list of things that it can report in its tree? So the the full list would be Pneuma groups then Pneuma nodes sockets and caches Cores and the single threads within the course For now we only handle the objects that matter for For processing units that is objects that contain processing units In the future we want to add Network balls and accelerators and things like this which will be all the kinds of objects and as well as the BCI buses and Switches and etc So this is a phenomenal step forward There was there was a prior project that I was involved in called PLP a portable Linux processor affinity and an HW loke is is just way better than PLPA and in so many regards number one PLPA was Linux only and PLPA only understood sockets and cores and HW loke surpasses both of those in in so many ways and so we're very definitely deprecating PLPA in favor of HW loke simply because it's it's just better in every single respect and as a matter of fact I'm actually right in the middle of integrating HW loke into open MPI and That is actually the the current hold-up in in getting HW loke 1.0 out the door because those as I'm integrating it I run across a little thing and I go fix something in HW loke and so on and so I think the only thing that's between HW loke 1.0 RC 3 and Is is the fact that I've got one little thing that I need to fix that I discovered in Integrating with open MPI last night, so we're getting towards it. We're getting there So how well does HW loke understand brand new equipment when it hits the machine room floor? You guys haven't even seen it yet as HW loke developers Will it pretty well understand new equipment or do you does HW loke itself need to be updated? Well actually we had quite a few good surprises the thing is mostly the problem is the operating system support if The operating system layer that expresses things in the machine is already quite autogonal and things like this It works quite well for instance the MagniCore processor from AMD actually integrated the new manuals inside the socket So any interface that assumes that a socket is inside a new manual would be broken and Linux doesn't do this and actually somebody on the list reported that he's run a JBLD doc there and it just worked fine without any modifications The way that's for new kinds of arrangement of objects Of course if a new kind of object is introduced in the market We would need to add it to hardware locality However in some cases I'm thinking about AIX It provides an interface to express objects in very generic way so right at the operating system layer and hardware locality when it encounters these unknown kinds of objects just takes it and expresses it as Miscellaneous and so you actually have the object you don't know what it is But it at least provides you the structure information Yeah, there's there's no iPhone or Android app for a HW look yet, but you know maybe someday So Jeff a question for you as a I know you work on HW look But you also spend most of your time working on open MPI so I'm curious. How are you using and how? How as a user of open MPI will I see HW look in work? Ah, so that's the beauty of it. You won't as a user of MPI you're just gonna so we actually Already in an open MPI we sport a couple of command line parameters to MPI run like bind to core bind to socket and things like that and You won't notice the difference of our upgrade from PLPA to HW look But what you will be able to see is this will enable us to support hyper threads Because with the Nehalem series some of the higher-end Nehalem chips Hyper threading is now, you know somewhat interesting to HPC but before Everybody just turned off hyper threading because it was completely useless and now The picture is a little more gray. There are actually some HPC apps that benefit from hyper threading and so Open MPI's MPI run will probably grow, you know a bind to hardware thread kind of command line option and HW Local will allow us to to support that Internally We're gonna use it for for two purposes exactly what Samuel already talked about number one, you know the process binding and number two Also just looking at the topology of the machine So when we do our shared memory communications kinds of things will be able to be a little more Intelligent about it like oh this guy's local to that guy. They share a cache You know, we can do some stuff with that or this guy's just completely remote from that guy and you know We have to do we have to be a little more careful in in talking to him So wait a minute. Let me step back there You can tell the difference between a real core and a virtual hyper threaded core now with these with this library Absolutely Okay, that's actually really nice Yes, if you want to know if you want just calls then you request HW lock for the core level and then you know how many there are and they are CPU mask and When you have hyper threading enable you see in the CPU mask that there are two CPUs for that on the country If you want to address a threat hyper threads, then you will just request the processing unit level and Then you will really get the right side if they exist all course if there is no threats I have to say this is one of the things that actually was very exciting to the customer I was talking to this morning was the ability to distinguish between hyper threads and cores and It was it was a difficult thing before and now they have a tool that can just spit out some XML And they can parse it and put it in their scripts and they were they were thrilled with that Yeah, sometimes you're not really able to know whether you have Enabled hyper threading or not and just from the in bios settings of what the operating system reports It's quite useful for that Yeah, this this particular customer was looking at they they have hundreds and hundreds of machines and in some cases They want hyper threading on and in some cases they want it off and they were not Excited at all about going and manually tweaking all the bios is of all the machines to turn them off And so they're like wow we can use hwLocke and just take you know the second hardware thread offline In Linux and therefore I can script it up into you know an init.d script or however they they want to do it So the next thing I need is is I need I need to actually have this embedded in a resource manager I need to be able to tell PBS. I want one node with eight physical cores I want one node with With all the hyper threaded cores also and the resource manager somehow tell that to whatever MPI library or Open MP library I'm using and so it does the right thing and that would actually be a really killer integration All right, I'll take this one because this has actually been a lot of my contribution hwLocke is the ability to embed it in Other software and like I said, I'm working on embedding it into open MPI right now So we tried to make the library itself very friendly to you know dumping this into Other software projects so whether you just link to it, you know because hwLocke is installed on your system Or you actually have a small reference copy of hwLocke inside your software itself, which sometimes that is useful to do That is actually you know the the bulk of my contribution so far to hwLocke but to answer your question Yes, that is very definitely a target. We won't Want applications out there like resource managers to use hwLocke We want slurm and torque and all the rest to do it and I know that slurm is actually Fairly excited about using hwLocke number one because they use PLP a TA today And they want to be able to upgrade hwLocke and be able to see things like hardware threads and whatnot But they're also very much looking forward to being able to detect nicks and other PCI devices and accelerators and things like that They just pinged me the other day asking when that would be ready and so on so I think there are some real-world HPC applications that will be using hwLocke behind the scenes in the near future I know that the the mpitch Crew is also they've embedded a pre-release version of hwLocke in there in their code so far as well And to those lines too since I'm mr. Corporate suit here I have to mention that we very specifically made the license of hwLocke the BSD So it's friendly and you know we want people to to use this as as much as possible with no corporate lawyer fear or anything like that Yes, that's exactly the choice we had made for them the PC So let's talk about actual performance gains from using something like hwLocke Do you guys have any examples of a case where when we started using affinity with that application? We saw a significant performance improvement in an HPC application Yeah, sure. Well, you know it depends a lot on the application Of course the thing is merely binding threads on the on the processing units Sometimes can already give you 10 or 30 percent in performance the Thing is if you bind erroneously because you assume some process and numbering You might get really really wrong just because the numbering is quite often Interleaved and so you actually put your threads on separate sockets all the times And so sometimes if you just use our locality to know the proper numbering of CPUs You can get 50 or 10 times better that depends of course on the applications The arm what I could not is during my PhD I mean starting from a random binding to High record hierarchical binding according to the structure of both the application and the hardware I could get something like a 30 percent or 100 percent better performance Some cases it's actually two times or five times better than NPTEL and just because we do know the structure of the application the Breeze tried this morning a data broadcast inside the machine and using the topology information you could get 25 percent better On for new ma for core machine and We had some measurements for network bandwidth. We get twice bandwidth just thanks to getting the threads near to the Network Yeah, just an editorial note on this it's it's remember now it's it's Nuna, right? There are networks inside the machine and then our networks outside of the machine So you really do have a non-uniform Network architecture when you're dealing with say an HPC cluster and you know, we've seen Optimizations for years where people are saying well if I do a broadcast that's aware of the network topology I can you know cut down the traffic through the core and you know You get better performance that way the same exact issues are true now Inside the server and you need to be aware of the topology inside the server in order to be able to optimize that And I think that's exactly what Samuel was referring indeed actually that that's the Please work on what we call new you are which is in the non-uniform input output architecture so the Communications within the machine and between the external network and the insides of the machine as well So you go this is spawning a whole new class of N. U. Prefix to acronym So HW look on one dado Release candidate is out right now. What is significant about one dado versus the previous releases? Well, the thing is I don't think there is anything really fancy in it mostly it's a clean revamp of the the whole thing Be it the API that has changed quite a bit and the output of the tools a few details like not using the word The abbreviation proc because it would meet processor But what is a processor? Is it the socket or a processing unit? So we use just processing unit We revamp the documentation documentation a bit So there is also some new things like the free VSE and exit 6 port I also added the notion of online and allowed Processes so that you can show them in in management tools One new thing is getting the current CPU binding the issue is that Far from all operating system provided but well at least Linux does provide it. So it's useful and of course there is all the embedding work from From Jeff which doesn't really provide features, but which helps a huge lot for integrations in existing projects Yeah, and I do want to stress one point here is the documentation There is a nice glossy PDF that you can download and it tells you everything about HW look This is very I mean documentation can always be improved, right? but the fact that there's a nice PDF that that details, you know, the The rash or the strategy the overview of HW look and then shows you every single Public function and constant and things like that. That's there. It's a programmer guide. It's a man page guide It's everything all in one nice PDF. I think that's actually tremendously important and will help a lot of people start You know from ground zero with HW look and I think what what Samuel was saying. I have to completely agree with it's HW look provides some of the same features that that prior projects like PLPA have you know socket and core determination But it provides so many more features and in such a better Wrapped up package there. There's so many more Things that you can do in a nice way that we kind of expect people will want to use it to get that information That it's just a fundamentally better package than anything that has come before and you know I say that with absolutely no bias whatsoever But it's it provides all this information in convenient ways that we think will be genuinely useful to a a wide class of Applications and so that's what we think the value of 1.0 is it's it's nicely tied up There's great documentation and a really nice set of features surrounding that information that you want to need to get So what's coming after 1.0? Well there is one thing that we've been pushing apart for now is memory binding the thing is for CPU binding The interface is quite simple. You just set a CPU bind on a thread for memory binding You have to know whether you want to migrate some existing memory Allocate how should functions look like etc. So we just push that apart to get 1.0 out As soon as possible and then do this later. So we expect to do this for 1.1 We think we should provide as well the a new my distance between human oath since it will be useful for most heuristics There is also feature which is dynamic CPU sets for now the Number of process source Inside the CPU set is fixed to 4000 This worked on having that dynamic So it's already ready, but needs some testing. So we push that for 1.1 What is expected as well is the PCI topology to to know where the network unique is It is already quite ready. We have a branch that exists that does work. Maybe it will be in 1.1 I don't know yet the Race worked on a lot on this so he would be better at telling about it the I'd also like to push later maybe not for 1.1 the topology of the new manuals and the topology of the network around the machine that is the output of Hardware locality wouldn't only be limited to the local machine but also the other machines are on the for instance the cluster to express the the Connections between the machines and the switches How I key are also switches things like this, of course that would require External information from tools that we don't really maintain. So that probably is through some plugins And also there is a crazy ID that would be to detect the USB trees and all the devices Cool stuff, I'm sure we will have some fine Cisco based plugins for reading things out of Cisco switches and whatnot as well Just had to throw that in there But as One thing we didn't really touch on is the fact that HW loc is now an official sub project of the open MPI Project, it really has nothing to do with MPI But open MPI actually is kind of an umbrella for a couple of Projects open MPI itself the software package is probably the most well known But there are a couple of smaller things and HW loc we think is actually going to become just as well known as Open MPI and it really has nothing to do with MPI. It's a nice little standalone utility in itself But this is kind of a tongue-in-cheek question Brock is asking me here on IM saying well You normally ask all these open source projects what they use for for source code control And do you want to ask yourself these questions? So I'll say that we use subversion as our main tree But I do almost all of my work in mercurial and I know that brice does get samuel. Actually, what do you use? Well, actually for hw york and I just use svm actually But for all the projects I just use anything that the project uses Cool Okay, so is there a website mailing list? I mean, where do you get information for support and getting involved with hw look? one-tenth Yeah, well, it's it's the open mpi website So if you if you go to open mpi.org or com or net, you know, we have all those domains If you go to that on the left hand side, there's a little navigation for sub projects And the first project in the list is hardware locality And you go there it has our mailing lists links to the you know download the pdfs and the html versions of the documentation and latest our ball tar balls and everything and Just like anything else we love to see other people get involved particularly to support architectures and platforms that We don't natively have access to so, you know reports from the wild are are greatly appreciated Okay, well, thank you very much, uh samuel and jeff and we will announce this on the hw Localist when this show comes out. Thank you Thanks Thanks for your time samuel. Thanks


Hi everyone, Joe for jaspyscasebreaks.com coming at you with a two-box break of 2021 upper-deck SP game used golf Random box break number eight now big. Thanks to Danny because he got both boxes saves me the trouble of running a randomizer So this is technically break number eight for you Danny. Thank you very much making life easy for me now Let's hopefully let's hopefully we can pull something awesome out of this SP game use golf We know who we're looking for And perfect timing for this golf too and ladies and gentlemen if you're watching live We have two more breaks of this left. We have four boxes left. So breaks nine and ten. This is break eight Break nine and ten of the last ones. It's probably a great time to get into this golf because If any of these golfers start to go off Right in the tournament, and I'm sure the values of these cards will start to start to go up nicely All right, like what if Kevin Keisner wins at all?  There's Bryson D. Shambot. You see He's kind of a favorite. I think he's right up there. I think he's maybe the third or fourth favorite Him and Kepka will not be playing together We have Blair O'Neill 52 out of 149 a couple pieces of her shirt Tour gear and the autograph is going to be Nancy Lopez. Yeah, she's from she's from around here That's just a little bit to the to the south of us. Yeah, she's got these number of wins. All right Next box. We've got Tom layman Doug Guim and your hits we got a couple relics and then the auto got Justin Thomas and Bryson D. Shambot a couple pieces of their shirt Justin Thomas could be in the mix on Sunday in the US Open D Shambot as well. Though. I think he might still be that knee might still be bothering him We got a triple relic here 13 out of 49 Matthew Wolf Maverick McNeely and Justin Sue. Nice. They're all pieces of their shirt And the last one here is a piece of their shirt and Autograph is Justin Sue four out of 49 Inked rookie fabrics Justin Sue. All right, so No tiger, but the search continues. We have four more boxes left Breaks nine and break 10. This was break eight Danny big. Thanks again for grabbing those two boxes. We appreciate it I'm Joe. I'll see you next time for the next one. Bye. Bye


How do you do great pleasure to be here with you well pleased to have you and Frank Skinner president nice to be back here well good to see you Mickey McGuire see you well you're nice to let us come visit with you appreciate the invitation back to the White House and especially this week I know this is the time of family loss and we particularly appreciate you let me tell you I hope you'll pass see you to Mrs. Reagan out of sympathy well I will be joining her tomorrow in Phoenix the last time I was here was about a year ago as the president and I was here to present to you telephone which commemorated the 75th anniversary of the telephone pioneers and you were kind enough to give us a videotape to use at our General Assembly out in Minneapolis that General Assembly focused on the 75th anniversary and your videotape message was an absolute highlight of that meeting well pleased to hear that it was a highlight not just for the 1,400 people pioneers who were in that assembly but for more than 200,000 pioneers that have seen that tape all over the United States and Canada at local meetings and again it's been a big hit we're doing that you'll recall perhaps also that in that tape you and Mrs. Reagan issued a challenge to the pioneers to join the fight yes again struggle that you've been and we have indeed and I think you'd be proud of the way in which pioneers all across this continent have joined in that fight I wish there were time to tell you in great detail about what's been done but let me just mention two or three examples for example in the state of Oregon and the state of Washington pioneers there had joined with Debbie Armstrong who was a 1984 Olympic ski champion in taking the message just say no to the school children throughout those states another example in another part of the nation would be an attempt to take the message to the general public in the states of Florida and Arizona up in Vermont and New Hampshire the thrust has been toward educating entire families on the dangers of drugs and I think especially significant is the fact that our pioneers in Canada have joined in that fight also that's great and every single chapter in Canada has joined in a fundraising effort to refurbish drug rehabilitation centers so that they too are part of that response and so Mr. President I think it says pretty clearly that your message planted a seed that's still growing and it's going to continue to grow for some time and I think it's a great example of how people all across this nation have responded to your private sector initiative and I can promise you the telephone pioneers are going to keep on with that effort not just in the drug abuse area but in other things as well when I was here before I told you that pioneers had raised four point two million dollars in the Statue of Liberty rehabilitation and that's just another example I wrote a letter sometime back telling you that we would like for you to be an honorary member of the Association since that time my term of office has expired I'm not eligible for reelection you know how that works I'm turning it over I have turned over the reins to my success and Tom Thompson and Tom is not only now president of the telephone pioneers of America but he is also president of AT&T technology systems and I know that Tom joins me in saying that the telephone pioneers are going to continue to fight the fight against drugs and to continue to be an important part of all the things that they've been known for in the way of meeting human need in the local communities and I think Tom would echo that some more importantly than echoing it I can tell you that in the year ending 86 we had 37 active programs that were drug related the year just ending that's up to 108 that's almost a three four increase but your take made a more important statement our pioneers had 20 million hours of volunteer service to the community last year and your statement on voluntarism is at the foundation and that earned you the right to be an honorary member well on an association-wide basis there's only been 10 in 76 years of history the first was Alexander Graham Bell in 1912 and we have Watson in there we have Mabel Hubbard Bell the last one was in 1925 there's never been a United States president that is an honorary pioneer so you're breaking some new ground well I'm very proud and we have an official commemoration of that which is a statement that says that president Ronald W. Reagan is a honorary pioneer and more powerful than this is the fact that as a pioneer you get you earn the right to do two things one is to wear a pin that is the symbolic pin of officership if you notice both Frank and myself have one well yes I well thank you very much and even more powerful within the pioneer circles is the rights to wear a windbreaker with a pioneer triangle on it and your name that says that you're really a pioneer of America and we have one for you with your name on it and then one in the box for Mrs. Reagan as well with her name on it my goodness thank you very much thank you very proud to have these and to have to be a member well you're number 11 on the association wide membership and maybe it won't be another 50 years before we have another honorary member Mr. President you're number one in our hearts well thank you very much thank you our pleasure to have you I have just some little souvenirs here you see standing oh thank you so much we really appreciate that that's very thoughtful by me we used to do it and I'm greatly honored thank you thank you thank you for the time and your support we're deeply grateful Mr. President thank you for all that you're doing thanks for all you mean we're stopping here all right thank you Robert Crancer see you I also have a jacket for you that was made up for our team. Thank you very much. Team USA. You've got to recognize that that's kind of a switch for a more force category. Well, that's great. I thank you very much. This country has 50 nations. You've won it seven years. As a matter of fact, we all started in 81, didn't we? You'll still be performing after I'm done. I have some things here for you there. Wow. Key rings. Thank you very much. Thank you, sir. Thank you very much. Well, I'm pleased to have you all in here today. And I thank you very much for that. Very much a pleasure. Thank you. It's a privilege to be here. What? It's a privilege to be here. Well, I'm very proud to have you here. We're the trophy belongs now. We intend on keeping it for a while. Where will it be the next time? France. France. France. You have to do a lot of traveling there. Well, thank you all for coming. Thank you very much. Thank you. All right. Thank you. Thank you, sir. Thank you, sir. Especially that Gorbachev is going to screw me out of the Thanksgiving trip. Mr. President, this is Newton Mayo. Hello there. And this is his son, Dr. John Mayo. How do you do? See you. You work a lot. Well, you're great. Step up guys there. This is the auditorium. Well, let's give them a picture with them. What if you... Yeah. Yeah. And then it sweeps in a little bit better. That's great. Sir, did you look right here? Thank you. I assure you this hangs in my study upstairs. We just brought it down here for this picture. Oh, I love it. And the... That's no strings. That's the horse that Nancy... But now she can only ride him occasionally. He's getting the rule of the hill. This is the one I ride. Well, there's the house. Now you work... I work through the house though. There through a magnifying glass. The pictures are so small. We couldn't... So we're sort of... They've been out in all of... This just came out well. Tell me. They couldn't tell me. Did they... They gave you photographs? Yeah. But did they give you photographs? As I thought, I understood it. Separately of the horses and then one just into the view. And then you had to... Well, one of the photographs had both of them in it, didn't it? No. That's right. There were two separate ones. We had three photographs. One was of the background. And then we had this horse by himself and this one by itself. And I had another one in the back. But it though... But I had more time, right? I think we could have done a little bit of job, but it worked out all right. I don't know if there were any more pictures. All these fences didn't even get out of telephone for a while. And me and two other fellas out there, we did it. We built that fence. Well, it's hand digging the post holes. And on the job, you have... This is my pier. I've never owned anything in it before. He's ever owned a map. Yeah. But this is... I don't call it that. It appears on the map as part of all of them. You see that part of the hill behind this is one of his... The Romney station for the airlines. But I just love this and it keeps me home sick. I was there to see Dennis Ayers yesterday and checking on Kimmich. I understand that you were tired of him altogether. That's what I understand that he is. Yes, he is. He's a heartbreak to me. Yeah, I know. I'm the one that worked on him originally. Oh. I'm the one that... The veterinarian that they brought him to. I also, when you were at Middlebury before the election, I did... When Margie Love was taking care of your horse, I'm the one that took care of the horse. Well... Looked after him a little bit. And just a couple of souvenirs of your visit here. We thought a couple of Western buckles. All right. We appreciate you taking the time to see us. Well, listen. I appreciate this. This stands there. There's Hanks right in front of my desk. I can study there so that every day I can look up and get home sick. Make sure I'm sick. Yes. Thank you. I hope you like it. I really do. Give our best. I sure will. I'm sorry. I'm sorry about that. The Lord was very kind in whatever she was. Thank you, sir. Take it easy. She no longer knew us. All right. Just quietly went to sleep. That's the way it goes. Thank you very much. All right. Thank you.


of sexually assaulting more than a dozen teenage girls. And now lawyers for 19-year-old Tyler Cost are pushing to get social media records from his accusers. ABC 15's Lauren Vargas reports from Penal County Superior Court where the teen's attorneys say he was set up. Tyler Cost's attorneys claim this entire ordeal was just a shocking conspiracy to put the teenager behind bars. And they pleaded with Judge White here at the courthouse to access some other social media accounts for alleged victims. Tyler Cost was in court today, but never said a word. The defense says they already have access to 98,000 pages of Facebook content, thanks to investigations by the Penal County Sheriff's Office. Some of those messages by the accusers allegedly show the girls making plans to teach Tyler Cost a lesson just like the movie John Tucker Must Die, where ex-girlfriends take revenge on a former boyfriend. Prosecutors would not comment on the ongoing case, but the defense attorneys say this is all about finding the truth. There were a group of girls who conspired against Tyler to put him in jail. They succeeded at that, and Tyler has been in jail for over a year based on those lies. When it was all said and done, the judge said he would take the issue under advisement. Both sides also asked the judge to push back that September trial date so they'd have enough time to interview all the witnesses. They're scheduled to be back in court on May 4 with a decision on that social media issue. Back to you. Well, Penal County Attorney Lando Voiles released this statement about today's hearing. Quote, in all cases, while defendants are presumed innocent until or unless proven guilty, the state prepares to proceed with all evidence, whether exculpatory or inculpatory. The state's responsibility requires justice be served by helping a jury to find the truth.


Happy New Year, it's time to hit it with a pop pop. Ah, ah, ah, yeah buddy. All you need is some kebasa and cabbage, and we're gonna make a meal for under 15 bucks and get this year started out right. Start out by cutting your kebasa. Cut your cabbage into little squares like that.  Two 48 ounce containers of chicken stock. Some fresh ground black pepper. So I ran to Walmart and to Kroger. Why did I go to both of them? Well, because Kroger was sold out of the spicy and Walmart was sold out of the original because it's flying off the shelf. And we're gonna hide us a little bit of both. Last thing, we're gonna hide us a lucky penny to bring us wealth in 23. It's been about 45 minutes and it's time to get a bite. And that's how you do it, my friends. Easy, delicious, and under $15. Hope you have a happy 2023. Mm.