What's up everyone as you may know David and I have a deep passion for basketball and sneakers but time away from the NBA has made us think more about why we like the shoes that we do and even though we are brothers it's clear that we have different body types and different foot shapes basically David has a wider than average foot and I have a narrower than average foot so we wanted to share our top 30 basketball performance sneakers of all time for wide and narrow feet so why should you think about your foot shape when shopping for your next hoop shoe and where can you buy them at? We'll explain. There are so many different factors that take into consideration when picking your performance basketball sneakers and some of them are not obvious to the regular person for example the instep is the thickness of your foot do you have a wide midfoot a wide forefoot do you have fallen arches or high arches some uppers do stretch some do not due to being synthetic materials how wide of a rubber base do you need on the outsole do you need an outrigger to prevent your ankle from rolling over do you feel comfortable wearing two pairs of socks to make a shoe fit better do you feel comfortable adding an additional insole to give you more cushioning there are definitely a lot of things to think about thank everybody for clicking on that video but I'm really excited to tell you about our sponsor today shop tagger if you guys shop online as much as I do you guys already know that there is no way of knowing when an item you want is about to go on sale or it's about to get restocked but now with this new free browser plugin shop tagger they will actually send you a notification when you're sneaker jeans jacket or even your couch goes on sale and it just works with so many stores whether that's Ikea essence grail Nike Foot Locker you name it shop tagger is actually really fun to use it's super easy to download then you'll go to the page of the item that you want you save it onto the list you can add a category to it and that's it you can get notifications on your phone or web browser and there's this new feature that helps you find coupons for you shop tagger cashback is almost here as well so cashback will allow you to earn real money automatically when you shop from top retailers so which shop tagger you'll be saving money three ways it saves you time it'll tell you when something goes on sale it'll help you get the coupon and there's cashback shop tagger is the ultimate shopping plugin for your chrome browser if you guys are interested in getting shop tagger make sure you hit that link down below to download it for free for your browser and make sure to hit that like button thank you and enjoy this video today you guys we have the ultimate basketball sneaker video the top 30 performance basketball kicks of all time David you're going to be giving a list of 15 basketball shoes for wide footers and I'm going to be giving my list of 15 basketball shoes for narrow footers because guess what people have different types of feet and different types of feet are fit for different shoes let's go number 15 for the wide footers I have an entire under armor section the curry seven and the anatomic spawn low under armors are really interesting because they're usually despite not having a lot of cushioning set on a very wide base and they have good traction so usually considered pretty stable shoes and they're really really good for court field I'm not a huge fan of hooping in them because for me they just don't have enough cushioning but I like the fit and I like the traction and I like the wide base all right guys for number 15 on the narrow footers list is the kobe threes rp to kobe what I loved about these shoes is that the upper was very flexible and they went up very high on your ankle so that means even if you had a narrow ankle you were able to tie them very tight so that it just wrapped around your foot it's also a very flexible shoe guys if you guys can ever get your hands on one of these I recommend them number 14 for wide footers I've got an adidas section andro I got a tie between the t-max millennium the marquee boost and the don issue one huge super wide bases I think the t-max millennium has the most cushioning I think the don issue one probably has the least but uh men andro if you look at how wide the base is on the don issue one I don't know if I ever seen a shoe that was that low cut with that wide of a base it almost felt more like a crossfit shoe than a basketball shoe all three shoes that I just listed whether it's the t-max millennium the marquee boost or the don issue one from adidas you probably can get for like 90 to 100 bucks anywhere online number 14 on my narrow footers list is the 2016 adidas crazy explosives guys full-length boost they have a mesh and prime net version but the mesh version felt just as good as a prime net and it's $20 cheaper so check that one out oh no that one I could tell you can choke that around some skinny ankles yeah if you look at the upper well as you can see who was very skinny that warm andrew wiggins yeah so if you're if you have a narrow foot and you're looking for a wide base definitely check out the crazy explosive 2016 number 13 for wide footers I've got the cp 11 and cp 12 really soft uppers really wide base really good traction I like to more work out in them lift running them and maybe shoot around I wouldn't really choose it as like my full pooping shoe but I know that people with wide feet absolutely love this shoe for number 13 on my narrow footers list I would say the Air Jordan 31 lows in half a size down now I the reason why I say half a size down is because I really love the fit of this shoe but it's slightly wide but I don't think it being shorter is really going to affect the longer narrower foot I do think that Jordan 31 is probably the most comfortable basketball shoe ever made number 12 for wide footers I think the way of wage six or the way of wage seven this is a a leaning shoe not easy to find in person in the US but you can order it online from a number of sites I mean if you look at the way Dwayne Wade is built you can assume that his shoes are built on a wide base I think if you look at a lot of the guards that I'm gonna you're I'm gonna have on this all-time wide shoe list you're gonna see Westbrook Game Lillard D Wade you're gonna see power guards and if you look at their bodies they are just like really sturdy NFL type bodies I actually think the way of wage six if you are able to get it is great for wide footers because it has a knit upper knit upper is great for wide footers because it can kind of expand on how wide your foot is and then just look at the base the base is so wide coming up at number 12 I got the Kobe Mamba focus I think uh any basically any Kobe shoe is going to be designed for people with more narrow feet I think any shoe of the Kobe line that pretty much has the Kobe brand in it is going to be made for slimmer feet but basically any Kobe is going to be flexible enough where you can tie it tight around your foot it's going to fit kind of like a soccer cleat no nobody's ever going to say that a Kobe is too loose coming in at number 11 my favorite shoe at this spot is the zoom GP retro whether a shoe is good for wide footers or not to me is really based off the shape of the insole the shape of the sole to me I actually really prefer really clamped on shoes whether that's with a shroud with a zipper or in the case of the zoom GP a ratchet mechanism coming up at number 11 on narrow footers again I do have to go with the Kobe 4 protros the reason why I particularly like this shoe is not really because of the cushioning but because for a lot of narrow-footed people they also have narrow ankles and this one really has a great heel counter lock in the cushioning that goes around the ankle is really big so when you tie it tight it just locks your whole foot in and to me for narrow footers lockdown is like the number one issue at number 10 I have the LeBron 15 low again guys you see the theme here cushioning knit uppers wide bases the only thing here is I would say the midfoot is a little bit narrow but because it's a knit it really allows it to expand right there I think the key about the LeBron 15 low is that it has a really curved cup outsole heel to toe transition is crazy and it's almost built like a Skechers booty popper oh those uh what are those called the poppers booty poppers it's something else hello booty popper yeah the ones that are supposed to tone your butt it's almost built like that because it has such a rocking motion having LeBrons on your list make a lot of sense because if you look at LeBron I mean he's a big guy so probably his foot is a little bit wider than usual coming in at number 10 for narrow footers you got the Nike PG 2.5 with the strap and you can actually get these still in stores right now on sale for about a hundred bucks on a lot of different sites I would say that narrow footers like the PG series yes if you look at PG I mean that's a skinny guy if you have a narrow foot you're gonna want to look for shoes that have a midfoot strap those things are actually going to help a lot I know back in the day some people used to feel like all the old stress they didn't do anything but no on the PG's they definitely do coming in at number nine for wide footers I've got a shoe you can get right now for $60 that I think everybody's gonna like the Dame 5 is an amazing shoe with the bounce cushioning in terms of how responsive it is and the kickback it gives you uh you know you want to feel that responsiveness in the cushioning you don't want to feel the pillowiness I think the Dame 5 is such an underrated shoe at number nine I got the Kobe AD Lowe's now a lot of Kobe ADs have come out since then but I'm talking about the first ones guys this is a great shoe especially for narrow footers one thing about the Kobe AD Lowe's Andrew for a bigger guy because I've tried those on too no cushioning if you happen to be a heavier wide footer that cushioning might not be sufficient enough eight for heavy guys this is I believe the only Kobe you're gonna see on this list okay the Kobe one Pro Tro from Nike yes this has a full-length zoom especially get the ones with the leather upper so they can expand a little bit more versus the other materials why I think the Pro Tro ones work better for wide footers than narrow footers is because the toebox is really wide and also that ankle area you're not really able to tighten it up that's more for wider ankles you're not gonna really feel secure if you have a slim ankle but man that is an amazing shoe like I said Nike let's just think let's figure something out let's keep Kobe's legacy alive by keeping the sneaker line available coming in at number eight one of the only pairs of Lebron's that you're gonna see on my list is the Lebron zoom soldier sixes now obviously you guys know that the soldier line for Lebron's they were built on a more narrow base and often sometimes Lebron did switch into those shoes in the middle of a game again one of the things you'll see in the shoe is the big mid-foot strap this one does not come loose it's not one of those dinky ones that every time you bend your foot or take a step it's gonna pop off when you try to wrap it too tight nah that thing was long it was like straight wrapping across the foot coming in at number seven for the wide footers I've got the zoom penny two wow you're taking it back I'm taking it back but they did retro this I believe already two times also the penny twos they have a lot of cushioning on the side so maybe if you're a narrow footer it could work out because there was so much like pillowy space yeah you got a max airbag in the heel you got a gigantic zoom unit in the front and um just built on a really wide base and it was really interesting because Anthony Hardaway penny Hardaway was very skinny but his shoes generally are pretty wide number seven for narrow footers I got the under armor clutch fit drive ones I love this shoe it kind of reminds me of the Kobe threes as in that the material is completely flexible it wraps around your foot it goes very high up around your ankle you're able to tie it very tight and the micro G cushioning implementation was amazing in these the traction is great guys this is uh Steph Curry was wearing these on his rise I still think that this has got to be one of under armor's best shoes ever this is one of their top shoes and guess what guys you can still get the shoe online on various websites even I saw them on walmart.com for like 75 bucks number six I've got the zoom GP glove uh I love the shoes that wrap them out of my foot they've got the monkey paw structure on the right I do think it's really important for guys with wide feet to not roll their ankles because typically guys with wider feet seven out of ten times are heavier a rolled ankle could be horrible you know if I'm just playing indoors and I know that it's only gonna last me like 20 30 games that is an amazing shoe that is a stable solid shoe even as a narrow footer I put them on and maybe my foot might move around a little bit more on the inside than I want but like I cannot move my foot back and forth like it is not going to get twisted coming in at number six man it's another Kobe but it's the Kobe mid a lot of people in the NBA ward it's a very simple shoe very light has lunar lawn cushioning all throughout guys it is a really really good shoe and I had a lot of fun playing in them I would say I did have to choke it pretty tight but given the material it allowed me to tie it tight you guys we are entering our own personal top five top 30 shoes of all time I've got the LeBron 10 wow David so I was compiling clips of you playing in the LeBron 10 and let me tell you there are a lot of clips of you playing in the LeBron 10 all right you got hooked in these for years I want to say I've had 10 pairs but I really like the regular version of LeBron 10 for myself even though the cushion is a little bit more mushy I don't think there's a shoe that gives you more energy return maybe ever than the LeBron 10 I do think you got to be muscular to wear these because I try to wear your LeBron 10s before it and although I could tell why you liked them man I felt like I was wearing a boot I think for big guys stiff uppers it's not uh restricting to us it's supportive whereas for a skittier narrow footed guys the stiff uppers they just seem restrictive number five on my list for narrow footers this is a shoe that you can still get is the PG-1 wow my goodness this is considered a cult classic one of one of the best basketball shoes of all time I gotta say yeah guys you could feel the four foot cushioning you could feel the four foot zoom it had a midfoot strap it was low it was it was just stable you know how I know shoes are good for narrow footers is when I can wear one pair of like regular thick socks and it fits well but if I have to double up and I have to like you know do all these tricks to kind of make up where the super super thick elite socks yeah then then I already know it's not necessarily made for me coming in at number four Andrew I've got the Dame 2 from Adidas probably the greatest $110 Adidas shoe of all time somehow some implementation of the bounce cushioning made it feel like it was almost like full length zoom wow and just the fit on it and it was good for wide footers because they had you know different materials on the uppers I think the PG-1 you feel a certain magic when you wear it the Dame 2 you feel a certain magic when you wear it in number four on my narrow footers list is the Kobe 9 Lowe's my goodness I don't wow I do not know how many friends told me that they could jump higher in these I almost I was throwing down dunks with a little ball in these at my you know at my peak and particularly the engineering mesh version now this is a shoe that when you put it on it's super light and the lunar lawn I don't know why it's just implemented in a really bouncy way in these shoes it is I would say the traction on the Kobe 9's might be an 11 out of 10 too you can argue that the traction on the Kobe 9's Andrew is too good yeah and I would say a lot of dunkers actually like this shoe a lot of slim dunkers really like this shoe and I almost threw down a dunk in these almost like a dunk to the small ball coming in at number three I've got the Russell Westbrook why not zero one both the mid and the low the why not series Andrew is one of the best Jordan performance series ever like I don't know how far it's going to go it might stop at like six or something like that but I would not be surprised if those six go down in history as some of these strongest shoe per shoe line in terms of for performance for power garden well if you look at how Russell Westbrook plays he's going to need some very sturdy shoes with great cushioning and that's what the why not series really provides I don't know if there is a shoe that is more supportive that allows you to do really hard cuts without any risk of twisting your did it make you want to attack the rim like Westbrook that's messed that could and number three for narrow footers this is one of the best basketball shoes you can get right now is the Kyrie Six's the lockdown is crazy especially for a narrow footer like I said I don't have to wear any sort of special socks they're zoom turbo so that means the zoom is very dense but it's very low to the ground there's great traction there's great lockdown I feel like Kyrie and these I want to just do like triple behind the backs and then a spin off and then do like a fade away that's what I feel like I would say in a way one of really the best looking performing shoes of all time too dog my top three picks I'm saying they're all on sale right now and you got to get all these top three shoes that I'm about to name the Kyrie Six is definitely one of them you can get them for $95 on Nike.com right now and many other sites coming in at number two this is another shoe I've probably had at least 10 pairs of Andrew the Jordan 31 mid this is a must cop for people with wide feet I don't care about the traction a lot of people really complain that the traction on the Jordan 31 is very iffy but to me if you have really wide feet and you're really heavy you're not moving that quick like Kyrie where the you know the flaws in the traction are going to be exacerbated well David let me tell you this you set a league record with seven threes in one game in these everybody always says that the Jordan 31 mid if it was not for the traction would be a top five shoe in history guys trust me if you're a wide footer it's the best fit you'll ever get wow and that's only number two all right coming in at number two for me for narrow footers guys if this is a must cop it is the hardened three with full length boost David I had like six pairs of these I loved them all they have this midfoot band that some people argue whether it helps or not I'm just going to go ahead and say it helps it's great for staying low to the ground super comfortable I could play like six seven eight games in them without my legs getting tired it was like an ultra boost basketball it was like all the best things you like about the ultra boost running shoe but really put into a basketball shoes low to the ground you could feel the boost it was very stable it was great for making moves left to right if I had to wear just one shoe on my feet 24 seven it would be the hardened three like it's the best to just walk around it all right you guys I have arrived at my number one pick for wide footers of all time David number one what what does being number one mean if you are a wide footer right now and you want to spend the money there's no better shoe that you can get than the Jordan 34 it's the newest Jordan Zion's wearing it Luca's wearing it I don't think it's surprising to pick a 2020 shoe as the greatest shoe of all time to me shoe tech is very similar to car tech it's always getting better these companies pour millions and millions and millions of dollars of r&d research every year and the Jordan 34 is something unlike I've ever seen before it reminds me of a Ferrari it's I've heard it be compared to a Ferrari before because the engine is so strong but it's so but the frame is so lightweight so if you look at the Jordan 34 the whole forefoot is a zoom unit it's not a cage zoom unit it's just the entire forefoot is a gigantic zoom unit I've never felt a shoe that propelled me forward in my foot strike more than the Jordan 34 and it's light yeah lightweight it's like a fighter jet but it's got incredible cushioning it's got incredible traction it's got support it really has zero weaknesses would you say it's like a nine out of ten on everything yeah I just think the Jordan 34 to me is really the progression of high-end basketball shoes moving forward of course the cushion is important the traction is important these are all conventional things we are now stepping into a new tier Andrew where the shoes are going to be helping you move Jordan brand is making the best shoes and basketball apparel in the world right now bar none okay so for my number one shoe guys there is just nothing better right now for a narrow foot than of course Katie it's the kd-12 right now guys the kd-12 for the price point $130 with double stack zoom air in the heel full-length zoom air strobe right underneath your foot great traction great stability solid comfort I think in my opinion for narrow footers it ranks nine out of ten on all those aspects the reason why they segment the full length zoom for skinny guys is for increased flexibility so that the zoom pack can kind of bend at these parts in ways that maybe for the bigger zoom packs they don't I don't even think why footers honestly can fit the kd-12 Andrew just just show this part this midsole that rides up your foot sits beneath this for me my foot is busting out of this to the point where it's incredibly uncomfortable guys I've just ordered a pair of the kd-13's I'm waiting for them to come but I'm really excited about it but basically right now you cannot go wrong with the kd-12's and they are going to be on sale for like the entire next year and a half so definitely get a pair all right you guys thank you so much for watching that performance basketball sneaker video in the comment section below please let us know one shoe we left off the list it's got to be one shoe for the wide footers and one shoe for the narrow footers and the best answer is going to get a 25 gift card and also let us know in the comments below if you guys want to see another sneaker video and what we should make it about because you know what man just missing the mba right now missing all sports period it just really got us reminiscent and just really doing deep dives in the sneaker all right guys make sure you hit that like button click subscribe turn on your notifications and until next time we out peace


Brooks, oh, I that's right. Michael Brooks worked for pacman Did he that's right? Yes, uh, that's how that's how we discovered him Yeah, oh, wow. Yeah, I made a comment about that years ago as it like as a funny joke just like a A little humor on the episode where we were remembering Michael after it happened And apparently I hurt pacman's feelings and he has not interacted with me since really. Um, yes It was just uh, you know, uh ribbing. It's okay. Yeah, that's all I thought it was but I guess some people just can't take it I mean, I I apologize I reached out via email and I said, you know, I I heard from your audience that you were like on your stream You mentioned how you you it hurt your feelings or whatever And I just wanted to reach out and say it was all in good fun and no no nothing personal I got back to me. Uh, not really very accepting of what I had to say He was like really really hurt my feelings. It was very nasty and you know, I'm not gonna say exactly what he said because it's a private email but He I mean, I've never was on a show before that but But it certainly weren't gonna be out there Talking about 2y2 we got a hash it out with pacman that he unfollowed me a couple years ago too I think it was because I was supposed to be a guest on a show But I kind of like I didn't realize it was happening like very soon. I so I kind of like Fucked him over by not responding right away. Um, we're not being able to do it right away Honestly, I I don't I don't I don't get how like you could do this and and this is in general I'm not talking about anyone in specific I I don't get how you can do this and and not and and care so much about stuff like that Like I literally went on Tim Poole's show and the dude called me like a groomer or or like pedo adjacent or something These words don't bother me Nothing about this but you cannot hurt me by calling me names. You can call me You could say i'm a look like a girl a woman call me whatever you'd like You could send me this you could call literally call me anything and it doesn't like you could make fun of me Nothing bothers me. I don't get how you could do this job Uh and be a streamer and and and and get your feelings hurt.   I did not see this If I get my $20,000 canceled then um, that is a very big win. I'll have to look into this I I heard the oral arguments and it didn't look good though. So if we got good news, then okay, okay Uh, Arthur says story that jimmy door was actually sam cedars burner account Nelson says can is coming for your your youtube jobs Uh, there's a bill going through it's not I don't think it's a series as people are making Anyways, um celeste says hi kings. Hi celeste. We love celeste Uh, mariposa says the media is not ready for them for the reality. That's straight men are the are the men in trans women's dms Uh gay men don't want trans Don't want trans clitty That's true. Yeah They're gonna find out though because they're getting a little bit too bold. Thanks marie Upper flyer says, uh, I love the format. We need more outlets. Yes, maybe include more indie hosts I'm every we like we like 20 people on the stream at once just talking over each other Stream yard has limits here and also the ability to hear each other's also a limit We need to read over each other so much like just adding one extra person makes it so much more difficult But we try right all right. We got leftist mafia But we need a new show called leftist royal rumble where it's just every 30 men enter every man for himself We vote each other out at the end of every 10 minute segment and then go to the next thing That's actually pretty funny That might actually work. Uh, just because it says i'm a boomer who really appreciates what you do. Oh, there you go Uh, um, mariposa says it's literally left to right small beard to large Which beer are you on the beard? That's pretty funny Robbie says your beard is that great gives me gender enemy. That's I think that's to you mic. I'm guessing Oh, is it? I think I don't see it. I think it's the mat. It's from uh, Robbie. Well, maybe it's from at that I don't see your beard is that great. I don't want to hear it. Is that amazing? It is for me, but listen, I I I I'm listen I'm all about facts over feelings and it is undoubtedly factual that mic has the best beard on the stream right now Well, listen matt, I appreciate it, but like if you're a bigger always better though, I don't know Yeah, maybe nice and and clean like this. Yeah, but he's also got he's also got a nice full beard I I can't I my my beard is just like I just have it because otherwise I mean, maybe who knows now I'm much older than I was back when I grew my beard. I've not shaved my beard in many many years decades But I originally just shaved like scare people just to be like my kids would my kids I've mentioned to my kids and they start crying. Don't do it daddy. Don't do it um But I don't I mean I had to grow a beard as soon as I could because I am someone who looks or who knows now But I looked a lot younger than I was. I was getting carded for all rated movies into my 20s I mean, it was I just needed to show people I was of a certain age by just being able to say look facial hair facial hair I've been I've been carded in my 30s at like the liquor store with a beard So Yeah, yeah, no, let me just say this matt like um It's unfair to compare your beard to like the best beard ever, which is mine, of course So, I mean your beard is good in its own right. Okay. Like if you're a singer you don't have to compare yourself to Beyonce, right? Like That's you got you got that like jim the anvil night heart status beard going on You know, it's uh, yeah, I've been cultivating it for quite some time. So it's it's evolving It's upping my power level and um Yeah, I would look weird without it for sure Here's firefly saying third chat and money for map a message for all your actions and compassion have made you worthy of the platforms And positions you hold rock on and much love very kind. Thank you. That's really sweet Kat says yes growing is good. But again, it's 2023 and we're in the middle of a genocide seven years being so ignorant While reporting on trans issues was inexcusable Uh, but this late in the game. It's egregious. I hear you. Uh, mr. P says left is best Yes, uh Taria taria. Oh Tara Yeah, this is one of my mods chatters were curious if you could add closed captions Yeah, so it auto generates But it takes like a little bit of time after the stream is over So if they come back in like a couple of hours, then it should be ready. Yeah, it'll it'll yeah It'll appear on Um flat job or flies the last one Says more hosts could also include alternating More hosts could also include all I mean if one of us miss for sure We'd get like oh you'd like miss a show. Yeah, someone will come in. Yeah, but apart from that. No, we're here for life I mean, I think you're rid of any of us leaving All right, there we go. Oh my god. It's like this is like an hour of the show I'm so sorry David. I feel so horrible Listen, I can't complain about money man. I mean, yeah, I would have done it for another hour That goes on latest. Fuck.  Oh, that's good. No. Yeah, let's do it. Yeah I'm not about missing it because we always have fun. No, definitely. Let's do it. Yeah, we do it. We do it every week But let's do it more. Yeah. Oh, yeah All right. Have a great night everyone. Have a good night. See y'all next week on 4 20 20th episode on 4 20 That's gonna be a good one. Mm-hmm. I may or may not appear Hi, we will see what happens. He will. All right Goodbye


This is an international security alert, please pay close attention. A dangerous anomaly has been disseminated around the world, and you must follow the advice in this alert to stay safe. SCP-714 is a jade ring. When worn, this ring will cause the wearer to randomly sleep and to become generally sluggish. Copies of this ring which cannot be removed have somehow been spread around the world through various postal services. If you encounter a jade ring, do not put it on. Store it and wait for a member of the police to collect it for destruction. Many individuals have worn an SCP-714 ring and died due to sleeping in dangerous situations, most often while driving. Do not succumb to this fate. Do not wear SCP-714. Various organizations are investigating the origins of the disseminated rings and will contain whoever is responsible. Thank you for your attention and cooperation. Secure. Contain. Protect.


Those guns are way, way better, so they don't mean to be better than those guys. Hey, Daddy. Okay, Ry-Ry. Hey! What's the object of bowling? We got a bowling. Here it goes, here it goes. What are you gonna get right now? What are you gonna do? I hope I'm gonna get a start. You can't go on the lane. Yeah, you can't go on the lane. What's gonna hook again? Ry-Ry, you're gonna set it on the ground. Push it too, you don't have to actually throw it. Last one, Dave is shooting a 67. I'm shooting a 129, let's go on Ry-Ry's shooting a 33. Oh, I gotta get it straight. Let's get up here, let's push it this time. Okay? Okay, I'll push it. You go up here, set it down. Oh geez, Ry-Ry, be careful. Now just push it, hold hands. Straight, push it straight. This is what you say? That was worse. That was a bad push, Ry-Ry. It's going really slow. It may not make it there. This is officially one of my worst ever attempts at bowling. And my last frame, I'm left with a split. You can get that split. You're right. Taller than you, and they might be really good. They might be professional. I want you to be the only official one. Thanks, Ry. 137, 67 to 34. We won, Ry-Ry. Just by surviving, we won. Oh, thank you.  Wait, what do people do?


No county shall in future be held more often than once a month, and where a greater interval has been customary, let it be greater. Nor shall any sheriff or bailiff make his torn through the hundred save twice a year, and then only in the due and accustomed place, that is to say, once after Easter and again after Mickelmus. And view of Frank Pledge shall be held then at the Mickelmus term without interference, that is to say, so that each has his liberties which he had and was accustomed to have in the time of King Henry our grandfather, or which he has since acquired. View of Frank Pledge shall be held in this manner, namely, that our peace be kept, that a tithing be kept full as it used to be, and that the sheriff shall not look for opportunities for exactions, but be satisfied with what a sheriff used to get from holding his view in the time of King Henry our grandfather.


The first bill of Minnesota's 2017 legislative session is in the books. Governor Mark Dayton assigned a tax bill into law Friday that will provide cuts for more than 200,000 Minnesota residents. It squares state law with federal taxes to provide help for teachers, college students, and homeowners. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.


Hello, my name is John Ognes and I'm here to do a talk about everything you need to know about writing an application, a real-time application under Linux. And there's a lot of small details you really need to pay attention to when you're writing an application on a general-purpose operating system like Linux. And I hope that we can go through the critical points here so that basically at the end you'll have a checklist and you can just, just as a reminder of everything that you need to do. So let's begin by talking about what real-time actually is, because that's also an important thing to understand. What is a real-time application? So of course when we're writing applications we want them to be bug-free. That's an important aspect to any application. But when you're talking about a real-time application you're talking about not only that it's running without bugs, but correctness means running at the correct time. So when let's say a certain task needs to wake up then it's really important for a real-time application that that task wakes up when it wanted to wake up. And if it wakes up later than that then that's a certain latency that we can measure. And if the latency is larger than that which we have in our requirements then we've failed to meet our requirements and the real-time system has failed. Now to really consider your system a real-time system it's not just enough that you would like your application to meet certain timing requirements or that it's important for you, it's actually a requirement which means if when it fails to meet these requirements you actually have an error in your system then you can call your system a real-time system. So it's really important that you define which tasks on my system, which applications or which threads are actually time critical. This is actually a quite important step because a lot of people make the mistake of saying okay we have a real-time system so let's just make everything on the system real-time. And this is generally false. You're going to have a lot of problems because real-time applications have totally different requirements than applications that are not real-time. So it's really important that you take a moment to really analyze your system and say what actually has the timing requirements that we have to meet on our system. Now in order to have a real-time system, an operating system that supports real-time, there's really three main tasks that have to be satisfied. So the first is that you have to have some sort of deterministic runtime or scheduling behavior otherwise you have no way of controlling the system at all. If I call a function and it could run days or it could run microseconds this does not help me. I have no chance of doing a deterministic low latency real-time system. It's also important that the system is interruptible because the CPU is always doing something, right? So it's really important in the system that it's able to interrupt whatever is currently running and to do something else. My important task, my real-time task when it needs to wake up then we need to kick somebody off the CPU for that. And third we need a way to avoid priority inversion. Now priority inversion is a situation where a high-priority task is waiting for a low-priority task. So in this particular picture that we see here we have task three which is low priority and in this scenario we can say that task three is holding a lock. Now task one comes along, has a higher priority and task one wants that lock. That situation is okay. You're allowed to grab locks as low priority and it could possibly be a situation that the high-priority task wants that lock. Anyway in that case the scheduler does something intelligent. It puts task three back on the CPU because task one wants that, the high-priority task wants that lock. And so the only one who can free that lock is task three. So it's put on back on the CPU. But now let's assume that task two comes along. Task two has nothing to do with task one, nothing to do with task three, nothing to do with this lock and this contention that we have there. But task two is higher priority than task three. So when task two comes along is actually a correct decision for the CPU to be assigned to task two when task three was previously there. So now we have a problem that a totally unrelated task is now holding up task three and actually indirectly task one. Now this is the priority inversion situation. The priority inversion is not between task one and task three. The priority inversion is between task one and task two. Here we have a high-priority task, task one, waiting indirectly on the lower priority task, task two, and task two might run forever. And this is actually quite serious problem and it can happen very easily in a complex system like Linux if you're not very careful. Now one way that Linux can handle this is that in the moment when task one wants that lock that task three has, then Linux will do something called priority boosting. And it'll actually boost the priority, actually called priority inheritance, but we often talk about priority boosting. The priority of task three will actually get boosted to the level of task one. And now when task three is running to free that lock, there is no chance that task two can become involved because task two has a lower priority than task one. And task three currently is running at the priority of task one. Now in the moment when task three gives up that lock in that exact moment, then task three will be de-boosted. It'll go back to its old priority and now task one can run with the lock. So we can actually grab the lock and keep running. So this is how priority inversion is avoided. Now in general when you're writing real-time applications under Linux, and this is actually one of the nice features of writing real-time applications under Linux, is that everything is done with the POSIX API. So there actually is a real-time extension to POSIX that's defined in the POSIX standard. And this is what Linux is using to implement the real-time functionality for Linux. So basically if you're used to working with file descriptors and opening, close, and reads, and writes, and creating sockets and all these things that are POSIX, then you'll also be very comfortable writing things for Linux with real-time. And in fact sked.h, time.h, and pthread.h are the only headers you really need to have full access to all of the real-time functions in Linux. So this is something that's really nice for writing applications, real-time applications under Linux. Now the scheduling policies in Linux vary depending on real-time or non-real-time. So there are currently three different scheduling policies for non-real-time tasks. And really the important thing to understand between non-real-time policies and real-time policies is that non-real-time policies have a fixed time slice. So it means at some point it doesn't matter if it's in a busy waiting loop, an infinite loop just going and going and going. If it's a non-real-time priority, it will eventually get scheduled out. And this is actually what's nice. So we don't have to worry about applications going out of control or taking over the system because they have certain time slices. And actually on your real-time system, I expect that most of the tasks running on that system will be with the non-real-time priority. So we have logging daemons or maybe web servers or any other kinds of middleware tasks and databases and things like this, things that are accessing the file system. These probably should not be running as real-time tasks. And so these are all things that are very well suited for non-real-time. Now some people could ask, well, why am I even talking about non-real-time? This is a real-time talk. And the reason is because even these non-real-time tasks, you have a lot of chances to still configure how they take over the CPU or how they get their piece of CPU. So there's things like nice values. There are things like control groups where you can actually limit for different tasks how much CPU they can have. Now this has nothing to do with real-time, but it gives you the opportunity to say, for example, my web server is more important than my logging daemon or vice versa. Or to say I want to make sure if I have a web browser running on my real-time system, I want to make sure the web browser never takes more than 20% of the CPU and things like this. These are all things that I can package my non-real-time applications into cages. And I really strongly encourage you to take the time to not only evaluate the real-time tasks, but to say how can I optimally, the non-real-time tasks within each other, for them, this whole group of non-real-time tasks, how can I optimally make sure that they are also sharing the CPU in a way that makes sense for my system? Now for the real-time tasks, these are all tasks that run as long as they want to run. So the only chance, for example, if I have a real-time task with priority 30, and the only way I'm going to be, that is going to end is if a priority, you know, something higher than 30 actually comes in and says I want to do something, becomes runnable. So with a real-time task that you have to be really careful with your code, you're not allowed to have some bugs that possible infinite loops because it could really render the system dead, or appear to be dead, just because this high-priority task has run away with the CPU. Now the SCEDFIFO is typically what people use. This means that I get the CPU as long as I want, and then at some point I'll go to sleep because I want to go to sleep because I'm waiting for an event or maybe I'm blocking some sort of IO, there's some reason for me to block, I'm blocking a resource, and at that point I will actually go into maybe the D state or the S state and freely give up the CPU. But as long as I actually have something to do, as long as I'm in the runnable state, I can run as long as I want, unless someone with higher priority comes. And with SCEDFIFO you have priorities from 1 to 99. 1 is the lowest, 99 is the highest, although please don't ever use 99 for your applications. There are some kernel threads that are very important that run at priority 99. They are more important than your application. So in my opinion you should never run between more than 98 for your highest priority. Now there's another policy called SCED Round Robin, SCEDRR, and this is exactly like SCEDFIFO actually. The only difference is if I have two tasks with the same priority, then there actually will be time slices between these two tasks of the same priority. So if I choose to make myself SCED Round Robin, if I'm a task running with SCED Round Robin, then I have set a time slice for myself if there's another task with the same priority that's waiting to get on the CPU. That's the only time, otherwise it's exactly SCEDFIFO. SCED deadline, I'm not going to talk about it all. This actually doesn't fit into this picture. It's another type of priority management, in which case the scheduler decides on the priorities based on timing constraints. There's some great talks from Steven Rostead if you go on YouTube or something like this, if you're interested about SCED deadline. The only thing that I want to mention here is if you do do anything with SCED deadline, SCED deadline wins over the highest priority SCEDFIFO SCOUT Round Robin. So SCED deadline, if you decide to do something with SCED deadline, it will always win and the SCEDFIFO SCED Round Robin, they take a backseat to SCED deadline. Mixing them is a little bit strange, almost a little bit of guaranteed priority inversion, but you might be interested in only SCED deadline. You won't learn about that in this talk, but you should know about it. Now, one thing I mentioned here at the bottom, which is really important, is that by default the Linux kernel limits the amount of time that all real-time tasks are allowed to use the CPU. So if the combination of CPU time for all of the real-time tasks adds up to more than 95% of a second, 950 milliseconds within a second, then for those last 50 milliseconds, no real-time task is allowed to run. Now, this is the worst kind of priority version you could have and you need to make sure that that doesn't happen in your system. So this PROXIS kernel SCED RT runtime US, this is an option where you can configure that maximum time. Setting that to a minus 1 will disable this feature because this is really a dangerous feature in a real-time system because if you do hit that limit and you'll see a message to show up in the kernel log, so dMessage, you'll see that it hit that. It's called runtime throttling, and if you hit this runtime throttling, then basically your system is broken because you went 50 milliseconds without any real-time tasks at all. So make sure to disable that. It's not a compiler option. It's not persistent.  Don't forget this. Really important. This slide is just basically showing you how you can set the priority of tasks. So every single thread, the task is a thread or a process, and every single thread, I'm going to say the word task, but it means threads or processes, every task can have its own real-time priority or SCED other with nice values, things like this. With the CHRT tool, here is how I can set the real-time priority and policy for certain tasks. So there's a minus P option that I can give, and this will specify, for example, the thread ID of a running task, so I can actually modify the real-time priority and policy of a running task, or I can actually start an application running. So I do CHRT minus F for FIFO, priority of 10, and then I can actually type the name of the application and it will actually start with that priority. Here you also see, in SCED.h, you can use SCED, set scheduler, you can also, through this function, programmatically set your own priorities. So if you prefer that instead of your boot scripts or external scripts properly setting the priorities, you can actually hard-code this in your applications that when they start, they automatically choose the priorities that you feel fit. Scheduling is one topic, but CPU affinity is also something that's very important because sometimes if I'm running my real-time applications, it may be critical that I actually have them isolated on certain CPUs. So let's say I have a system with eight CPUs, which is pretty common these days, then maybe I have six CPUs that I'm dedicating for the general system, the web server, logging daemon, system D maybe, and then I have the top two CPUs I'm just using for the real-time application. So with CPU affinity, CPU affinity is basically a bit mask of the CPUs that a task is allowed to run on, and this is also per task. So this is per thread or per process. You can actually say that this one task is only allowed to run on this CPU or these three CPUs or however you want. So you can set the CPU affinity for tasks, but you can also set CPU affinities for hardware interrupts, which we'll look at in a second. So when a hardware interrupt comes, an actual interrupt in the interrupt line, one of the CPUs is going to have to service that interrupt, and actually everyone is allowed to service it. So it doesn't matter who's going to interrupt, but whoever services that hardware interrupt is going to be interrupting whatever task was on the CPU before that. So if I have a real-time application on a certain CPU and I want to prevent it from being interrupted, for example, from hardware interrupts, you can also set CPU affinities for the hardware interrupt handlers. We'll take a look at that in a second. And lastly, it is also, that's actually the third point. The second point here is you can also boot the kernel and tell the kernel not to put any of its own kernel threads on the certain CPUs. So this is a way that you can do it so that when your computer boots or system boots, you already have some CPUs that are free for being assigned to your real-time application, and the kernel will not be using that for itself. Now I will want to mention here in this last point here, in this last little paragraph there, it's really important that you understand that in a lot of systems the multiple CPUs will share the same caches. So make sure you're aware of how your system is architected that you know that, for example, two of the CPUs are sharing a certain level two cache. Those are the two CPUs that probably should be hosting the real-time application, because otherwise, if you have non-real-time tasks and real-time tasks, they may be on different CPUs, but if they're sharing caches, then that non-real-time task can have an adverse effect on the real-time application. So you should just be aware of that of caching. It's a really important topic. So in this example, we're seeing how we can control the CPU affinity for applications. So with the task set tool, I can either start a program with a certain CPU affinity mask or I can modify any task, the CPU affinity mask for that task. So if I say task set minus P, then it says PID, but you're actually the task, I do the TID actually, the task ID that you provide there, because you can do it per thread. So I can actually set the mask for a thread. Now if I don't provide a mask, I just say minus P and give the PID, it'll actually tell me what the mask is for that thread at the moment. So I can look at what the current mask is and also set new ones. By the way, that also works for CHRT, the tool we just looked at for changing priorities. You can also see what is the current priority. Although in PS, you can see that anyway. So just like with setting the priorities in code, we can also set the CPU affinity in code. So if you want applications to set their own CPU affinity, you can do that as well. The reason why you need the GNU source to find here is because the SCED set affinity is actually a function that is not part of POSIX. It's implemented in the glibc, but with this defined GNU source, basically we're saying we know we're using a glibc, so this function is available and you actually need that defined for it to be available in the header file. I talked about the boot parameters, so you can actually have the kernel boot up and only use certain CPUs. So you have two different options for this. One is called maxCPUs. This actually limits the number of CPUs that the kernel can see. So I can actually, for example, if I have an eight-core system, I can actually, if I set maxCores to four, the kernel will only allow four CPUs to be used. So the other four CPUs will be left completely alone, and some people will use this, for example, to run certain bare metal applications, perhaps for real hardcore, real-time where they need basically busy waiting, pulling applications similar to microprocessor applications that are running directly on bare metal on the hardware, and those can then communicate through shared memory with Linux or something like this. Anyway, using maxCPUs, you can do that. So we can restrict Linux to a certain number of cores and then we can run our bare metal applications on the other cores. ISO CPUs works differently. ISO CPUs specifies certain CPUs where I want Linux to not put any of its own kernels, and when it starts any of its own kernel threads, and when it starts applications, it will also start those applications with a default CPU affinity mask so that nothing appears on those CPUs. So basically, we've isolated some CPUs. However, Linux is aware of those CPUs and is allowed to use them, right? So basically, when the system boots, there will be some CPUs that are free, and then I can choose to start my real-time applications on those CPUs. The next two sections of the slide, they're showing how you can set the default SMP affinity for, for example, when a new interrupt handler is registered, what do I want the CPU affinity for that hardware interrupt handler to be? Yeah, so that'll be the default. And for interrupts that are already registered, you can actually go into PROC IRQ, and then you can look in the effective affinity. This is what's showing you the CPU affinity mask for that hardware interrupt handler, and you can write values, write mask values into the SMP affinity virtual file to actually change that. Just make sure you look in the effective affinity after you write into the SMP affinity because some hardware is not capable of arbitrarily assigning CPUs to hardware interrupt handlers, right? So you can put whatever value you want into SMP affinity, then check the effective affinity to see what was actually set. Memory management with real-time applications is actually probably the most important topic when you're talking about real-time developing and possible latency, and it's because the kernel memory management functions as I'll explain right now. Basically, any time you allocate memory, you don't actually get the physical memory assigned to you. So you do a malloc, you're going to receive a pointer back, and it's going to be marked in your MMU table, so it's going to be marked that, okay, this amount of memory has been assigned in my virtual address space, but there are actual no physical memory mappings behind that. There are no pages that have been assigned to that yet. And this is only when you actually access that memory. The first time when I try to write to it, for example, that generated MMU exception and actually created a page fault, the kernel has a handler for this page fault, and in that moment it's going to find a free page. So if you're generally four kilobyte block of memory, physical RAM, it'll search for a free one, and then it'll do the assignment, and then the application can continue. So my malloc is actually very, very fast. If you do time, you can see that malloc is really, really fast. But as soon as you start touching that data, then all of a sudden it's really expensive. The first time, once that page fault has happened and the memory is there, then we're okay. And this is not just for the heap with mallocs. This is for all components of the processor virtual address space. So even the text segment, the initialized data segments, the stack and the heap, right? So for example, we're talking about the stack, when the stack grows, just moving into the next page, because the stack has grown into the next page, that's going to initially cause a page fault, right? So page faults are quite expensive, which means this is something that you want to avoid, and I'll show you how to avoid that. There's basically three steps that you need to do when you're doing avoiding. The first one is you need to tune glibc's malloc to make sure it's actually using the heap, because there are two ways that glibc can allocate memory. It can actually use its heap, really calling the system break system call to actually increase the heap, as most people learn in universities. The other method that glibc can use to allocate memory is it can actually do an mmap, and in this case, it's actually requesting the memory directly from the kernel. And that kernel is going to provide a block of memory that's not within this heap. It's something totally separate, right? And we don't want to use this separate memory mapping version of allocation, because when I get something from the kernel within malloc in this way, and then I do a free, then it goes back to the kernel. And that means that the next time I do a malloc, I'm going to get some fresh unpaged memory again. I need to page all of that first axis is very expensive. And when I do a free, it's gone again, right? So you're guaranteed, if when you do the mmap, is you're going to keep getting lots and lots of page faults, and we don't want that. What we want is we want basically to use this heap area with our mallocs, and we want that to be paged, to be paged in. So we're going to cause these page faults, and we want to hold on to that. So that's what we need to tune glibc to make sure it's always using the heap, never going with the memory mappings. We also need to lock down the allocated pages because we want to make sure that pages that we've allocated, memory of allocate, never goes back into the kernel or is never recycled to use for something else. Now, this is typically, we know this as swapping. And some of you out there might say, we have an embedded system. We don't do swapping. You're wrong. There are paces of your process address base that are available on disk. And that's, for example, the tech segments. So if the kernel really runs low on physical RAM, it's going to start recycling all of the pages it can where it knows it can get a copy again. So this includes, for example, the tech segments. So the function, the code that's actually getting executed could actually get paged out. And this would be horrible because it means the next time your application gets the CPU and tries to execute an instruction, that's going to cause a page fault and it has to pull that back from disk, which could be quite expensive. So don't think just because you don't have a swap file, you're immune from swapping, that is not true. So we need to make sure we lock down our pages for our real-time applications so that they cannot be paged out. And the third step on here is pre-faulting. And this means we want to create a heap and we want to cause all the page faults in that heap. And then we have memory available to us that's already ready to go. And this also goes for the stack. We don't want our stack to be growing and then suddenly we move into a new page and it causes a page fault. We don't want that. We want, it doesn't matter how deep we go in our stack, we want to make sure that's already memory that has already generated page faults so that we don't get surprised if our stack gets a little bit bigger. So here we can see for the first two steps of how do we can tune glibc. So here in this first one, we see the malopt function. Here, first of all, we're saying the maximum size for doing an mmap is zero, which basically is disabling it, which means glibc will never do an mmap to get that memory for this application. It will always go to the heap, which is what we want. The second thing we're doing here is we're disabling the trimming of the heap. So this is also implemented in glibc that in our heap, which is growing, if there's a large area that happens to be a contiguous area that happens to be free, glibc will actually trim that. It's called trimming and reduce that heap so that that physical memory is available to another application on the system. We also don't want to do that. We're going to go through the effort of paging, of pre-faulting, causing page faults for all of this heap. We don't want to be giving it back. We don't want to give back any pre-faulted memory that's for us to keep. So that's why we turn off the trimming threshold there. Then the mlock all. This is where we can lock our memory into actually lock it into RAM that I mentioned. And here's the flags current and future. So any memory we've allocated now or we will be allocating, that we want that all to stay. Most of it, for example, if I don't have a swap file, there's a lot of things that won't be paged out anyway. But just to make sure with mlock all, we know the entire virtual address space for this application cannot be paged out for any reason under any circumstance. Here's an example of how you can do heap pre-faulting. So we see we have a simple function, pre-fault heap that gets some sort of size. It's going to allocate a very large chunk of memory. Ideally, you should see the worst case from my application. What's the worst case size that I'm going to need? Maybe I need 10 megabytes, 12 megabytes, 100 megabytes. And we want to allocate this in the heap. And then we're just going to go through each page and we're going to touch all of that memory. And I'm really writing a non-zero value in there just to make sure we're definitely writing to each page. And that's going to be quite expensive. This is a quite expensive for loop. And at the end, we're going to do a free, right? So we're allocating a big chunk of memory. That's going to, since we have already configured glib seed to go to the heap, it's going to do the system break. It's going to grab all of that really into the heap. We're going to touch all of that, which is going to cause page faults. And then we're going to give it free. And now we have this huge heap area that's already been pre-faulted. And since we turned trimming off, we know it's not going to go back down, right? Something similar for the stack pre-faulting, but instead of doing a malloc that we get it from the heap, we're just going to create a very large stack frame, something that we learned we're never allowed to do. In this case, we want to do this, right? So here, this is an example from a 512 kilobyte stack that I want. I'm just going to have a function that I go into that has a 512 kilobyte character race sitting in the stack frame. And I'm just going to touch all of that. So basically, we've created a huge stack frame. We're just going to touch all of that, or one page from each of that. And then when the function ends, the stack frame disappears from the stack. And now we have this huge stack area that's ready and it's already been pre-faulted. And that also is not going to be given back because that is also something, because we've done mlock all, the kernel cannot take back that stack memory. It's ours to keep. Now, both of these functions, pre-fault heat and pre-fault stack, we only have to do once in our application, right? Right at the beginning, just a startup. So the startup is maybe a little bit slower, but now we have all of our memory available ready to go. So that was memory management, right? Avoid the page faults. Now we're going to talk about locking the POSIX provides the mutex for locking and the mutex is a very important object for synchronization because it has the concept of owners. So for example, if a certain task on a system takes a lock, that's actually the only task on the whole system that can release this lock. And this is an important attribute because if one thread takes a lock and another thread comes along and wants that lock, the kernel needs to know who do I need to put back on the CPU to free that lock? And the only way it can know that is if you're using mutex and if you're using something like semaphores or some other locking mechanism where there is no owners involved, the kernel cannot help you. So you're guaranteed to have a priority inversion there. Also really important is there's this priority inheritance that I already showed you. Unfortunately, this feature is not the default for P thread mutexes.  I don't know why you would not turn it on. And then there are a couple attributes like robustness and shared that you might have to turn on if your mutex is sitting in shared memory and you're doing so a multi-process application rather than multi-thread. So this is just a really simple example that shows you that you have your mutex here as an attribute object and I can initialize this attribute object, for example, to say I want to use priority inheritance. And then I can initialize my mutex using this attribute object from the mutex. Then I can do locking and unlocking. Pretty common. Most people are probably aware of this. The P thread, the shared and robust attributes are something of interest to you when you're doing multi-process. So I have multiple processes, not threads but processes and maybe a shared memory and there's actually a mutex sitting in that shared memory. It is critical that I activate the shared attribute for this. Also there's a feature called robust. I'm not going to go into, but there's a man page there that you can look at of how robust mutexes work. This is also very interesting for multi-process architecture. So you have multiple programs that are using shared memory to communicate and there's a mutex in that shared memory. This robust feature may be very interesting. Just be aware that if you turn on robust, there's also a lot of code you have to write. You can't just turn on robust and everything's fine. There's actually more things your application has to do at that point. Just be aware of that. So that was locking. Another thing is signaling. If I have two different threads that want to communicate through each other with signals, then there's something called conditional variables. And conditional variables are very nice because they connect a weight object, the conditional variables a weight object with a locking object. So if you have the pattern in your code where I'm waiting for something and I wake up and the next thing I do is I grab a lock. If you have that pattern, I'm waiting for something, I wake up and I grab a lock. That's exactly what the conditional variables were made for because it will actually connect that conditional variable with the lock object. If you're doing anything with real time, you probably want to avoid using signals, for example, just because with signals you have a really hard time. Actually, it's not even really possible to control the environment that when the signal handler comes, you're in an environment you're not really in control of. When you're using signals, basically you register a callback. And when that callback gets called, what's my priority? Am I allowed to take locks? What kinds of things can I do there? What's my scheduling policy and things like this? You actually don't know. And it actually depends. And it depends also on the G-Lib C implementation or the Lib C implementation you're using. Just try to avoid signals. The only thing signals are really good for is determining your application. And also really important with conditional variables, and we're going to see this in a minute, that the person who's sending the signal, it's really important that they send the signal while they're still holding the mutex. So here's an example, first of all, of initializing the conditional variable. There's also conditional attribute. Conditional variable attribute, but this convert attribute is only used if it's sitting in shared memory. So if it's not sitting in shared memory for multiprocess, it's just sitting in memory that's shared between different threads of the same process. Then actually you don't need a attribute and you can just use null here for this second option. So here's an example of what the code looks like. So we have the waiter. And we see that the waiter also has to take the lock. And then it calls this con-to-wait function and it goes into waiting. And in that moment where it begins to wait, actually the lock is released. This is an atomic operation that I go to sleep and I release the lock atomically. And when I wake up again, I will receive that lock again. So from the perspective of the receiver, I never lost the lock. I grab the lock, I'm waiting, and then when the wait function's done, I still have the lock. But I got it back and then I have to unlock it again. If I'm the sender, if I'm the person who's doing the notification, I'm also going to grab that lock, do the special, work on that special area that's synchronized, and then I can do a broadcast to wake up whoever's waiting on this wait object and then I can do the unlock. And here's what I'm talking about where it's important. I do the broadcast before the unlock. The reason why this is important is because in the moment when I'm holding a p-thread mutex, I might be extremely critical for the system and I don't know, I'm not aware of it, right? So we saw this example of priority inheritance as priority boosting. This will only happen when I'm holding a mutex. So it might be that all of a sudden I'm extremely important for the system, maybe my priority was boosted. This is the moment I should send the broadcast. If I wait until after I release that mutex, I might never get to the broadcast, right? So maybe in the moment when I release that broadcast, now I'm not important anymore. And there could be some higher priority tasks waiting on that conditional variable. So it's really important that I do the broadcast before the unlock. When you're dealing with clocks, you're also important that you're using the monotonic clock. So the monotonic clock is a clock that always moves forward. It always respects the human definition for seconds. And for example, it doesn't get changed twice a year for daylight savings. It's not adjusted by NTP to, you know, push the time forward and back or because of time zones or anything like this. It always just moves forward, but it moves with a tick that the humans define as seconds. So it's still using our correct units of seconds, microseconds, nanoseconds, but it is always moving in one direction, which is a much better clock to work with. And also it's important that you're using absolute times, right? Don't try to calculate, for example, if I'm in a cyclic task, which we'll see in a second, I shouldn't try to calculate how long I want to sleep for the next wake up. I should just increment an absolute time and just wake up at that time, right? I shouldn't be calculating. And here's an example of that. If we look down here at the cycle task main function, we're just sitting in a loop. Before we go into the loop, we get the clock once. We go into this loop and we're just in this loop, we're just going to do our critical real-time work. Then we're going to increment the absolute time to sleep until, and then we go to sleep. And then we wake up, we do our work. We increment that absolute time. We're not even checking the time in this case. We're just incrementing the absolute time and going back to sleep. So there's no way, even if like in 10 years down the road, we look at what this program is doing, it's still exactly waking up when it's supposed to, maybe exactly on the second, every second of forever, because we're using absolute times and we're not calculating how long we should sleep because you lose, you get a lot of jitter and variation when you try to work that way. The only thing to mention here is we have this norm ts function. This is just up here. It's implemented just because the time spec definition, the nanosecond field is not allowed to be more than the number of nanoseconds in a second. So it's just a normalization function to make sure that that stays true. When you want to evaluate real-time system, cyclic test is probably the best tool for that. The cyclic test is simply a tool that measures and tracks latencies at a certain priority level. So I can call it, for example, typically with cyclic test minus capital S minus M minus P, with the priority level that I'm interested in measuring the latency for. Also the second line task argument is also quite important. And generally you'll start this task and it's basically all it's doing is going to sleep and it knows when it's supposed to wake up and when it actually does wake up, it just checks the clock and see how what was the latency there. And it just does it again and again and again. It actually creates statistics. You can generate histograms for what is the latency at this level. And I say at this level because at priority 20, there might be different types of latencies than at priority 90 because there are real-time applications running. So I don't just want to know what's my real-time priority at 99. This is not very useful. I want to know, okay, my application is running it, for example, real-time priority 30. Then probably running a cyclic test on 31 is a good place just to see what kind of latencies priority 30 can expect while my application is running. The reason I say 31 instead of 30 is because we don't want that priority 30 task to be affecting the latencies of the test. Now while you're doing the cyclic tasks, basically you want to generate a lot of loads just for the system and try to basically cause latencies. So there's tools like Hackbench, which do scheduling loads. You can have an external ping flood, not internal, but externally a ping flood coming, generates a lot of interrupts. I can do things like running top minus D0 on serial or SSH, which just generates a lot of packets and a lot of interrupts, particularly with serial that's pretty painful. I can run OOM killers. So basically I'm taking every page of free memory and the system is starting to panic a little bit and it's starting to kill applications in order to free up memory. Even in this situation, my real-time application should run fine if I've configured everything correctly and I've done the tips that I showed you before. There's the stressNG tool. Obviously you want to stress all of the components of my system. So if I have Bluetooth, I should be doing Bluetooth stressing. If I have an SD card, I should do SD card stress testing. All of these things, the components are testing because I really want to shake out. Is there any possible code paths where I really do have a high latency situation? I need to shake those out and find them. Whatever the worst case I find, that's the worst case of the system. If you see a value you don't like, that's what you got. It's maybe a bug in the kernel. Maybe your kernel is not configured correctly. Maybe your applications are not configured correctly, but that's the reality of the situation. Lastly, don't forget to test an idle system. That's sometimes the biggest killer of real-time applications is the idle system. If I set cyclic test to wake up once a second, and the system keeps going into low-power CPU modes and things like this, that actually might be my worst latency. We're running out of time here, so I just want to quickly mention a couple of things. Perf is a great tool just to track, count certain hardware events, for example, page faults, cache misses, things like this I can track with Perf. There's also some examples here of how you can call that. For example, if my system is running in a pretty critical situation, it's nice to just start Perf and have it track. Are we getting page faults in this situation? Are we getting a lot of CPU cycles? We can actually look at things like which task or which symbol is causing the most CPU cycles or things like this. Perf is really a powerful tool, a profiling tool to identify places in your system or in your application where you have optimization available. If I see that my application is causing a lot of page faults, this is obviously something I can fix. I've shown you how to fix your page faults. There's also a tracing infrastructure available in Linux. This doesn't just profile what happens, but it actually takes timestamps of the events and you can see when it happens. Here's an example with trace command where I can, for example, trace, wake up in scheduling events and you can even look at these graphically. This can actually show me in this particular picture I can see that the Matte terminal lost the CPU, probably because of its time slice, but strangely enough, CPU zero is free here. I see there's actually a moment where Matte terminal is in the runnable state and I actually have an idle CPU. It looks like that could be interesting. Why is our application not on a CPU when we have a CPU free? With a trace is you can actually see these things and it can go really deep in depth. We can actually see the events for when, for example, the wake up happened and we're coming out of maybe a p3 mutex lock where schedule switching is happening. Actually, this would be the broadcast there. A switch is happening and this one's coming then out of the weight would be probably this condition. Actually, with Kernel Shark you can very easily just measure these distances and so you can get really good microsecond accurate values for how long these things are taking and you can actually see what's happening and it's nice to view things graphically because you can see what's going on there.  This is basically just a list of options that don't necessarily mean that your kernel is not configured correctly but there are really things to look at. If I look at a kernel from somebody and I want to see is it optimal for real-time these are the configurations I'm looking at just to see how they're configured and if we say, okay, you're using CPU idle is this something that, for example, you need? Do you need the low power CPU states or not? Is power saving important or is real-time more important? Things like this. These are just some things to be aware of when you're looking at the kernel. Lastly, here is the checklist that you've all been waiting for where you just need to go through consider your real-time priorities and also your non-real-time priorities. I can't emphasize that enough. Use nice and control groups to your advantage so that the things that are not real-time are still running the way you want them. The CPU affinity. Avoid page faults. They are deadly for real-time. Make sure you're using the monotonic clock with absolute time. Make sure your kernel is configured optimally Don't use signal leaves for anything except for killing your application. Priority inversion. It's the killer for all real-time applications and make sure you're verifying your results. Don't just assume you've programmed it correctly. Trace it, analyze it, profile it and look at those situations that you think are happening. If you think there's a situation where priority inheritance could really happen, trace it, see it happening and make sure you're aware, yes, we did implement it correctly. I can see the priority boosting happening. Now you know we did it well because maybe you forgot to activate priority inheritance for the mutex and it won't happen, right? You need to see it happening. So really trace and verify those results. Lastly, NMI's non-masculine interrupts are the absolute killer on any real-time system because Linux can't do anything about it. They're just going to come and jump into some biosvector or something. Make sure you know about them. Make sure you know how to avoid them. Talk to your hardware vendors and get that cleared up. So thank you. I went a little bit over. I hope that's okay. I really appreciate you taking some time to look at this. Virtual conference is a little bit difficult this year and the real-time Wiki site is the address I've got there. Go there. There's information about the configuring a real-time kernel, also examples of writing applications and if you see there's an article missing, write one. It's a great place to go. So I thank you for your time and have fun with real-time Linux.


Centric jaw relation records are an essential component of the materials submitted to the regional dental activity for the fabrication of various prosthodontic appliances. The proper preparation of these records is crucial to the final success of the appliances. When preparing centric jaw relation records, it is necessary to recognize that specific dentulous and edentulous situations dictate the use of specific record bases. These bases, in turn, dictate the appropriate methods to be employed as well as the types of materials to be used. In view of the special problems associated with receiving materials by mail, the RDA has designed specific packaging procedures. The RDA also highly recommends certain materials and methods for the preparation of centric jaw relation records. We will discuss the materials and procedures for obtaining centric jaw relation records associated with the fabrication of complete dentures and removable partial dentures. The records associated with fixed partial dentures are discussed in the second videotape in this series. When complete dentures are required, there are, of course, three basically different dentulous and edentulous situations that are possible. When both the maxillary and mandibular arches require complete dentures, it is suggested that auto-polymer resin be used as the material for the record base and that either modeling plastic, plaster, or zinc oxide eugenol paste be used as a recording material. The base plates and wax rims are tried in the mouth to check for stability on the edentulous ridges. The maxillary rim is adjusted to meet the aesthetic requirements regarding anterior tooth position and both rims are adjusted as needed according to the determined occluding vertical dimension. A vertical line should be cut into the maxillary wax rim to denote the midline and to act as a visual reference when the patient closes to the previously determined centric position. The patient is instructed to open and close several times to ensure that centric closure can be repeated. The maxillary wax rim is left intact except to cut several V-shaped notches in each side in the bicuspid first molar region. These notches will serve to index the recording material as the patient closes into it. The mandibular wax rim is cut away to a depth of 2 to 3 millimeters except for a small section anteriorly which will be the centric and vertical stop. This material is removed to provide adequate room for the recording material. The mandibular rim is removed from the mouth and the recording material is placed on the rim. It is then replaced in the mouth and the patient is instructed to close and to remain in that position until the recording material has become set. Excess material is trimmed away with a sharp knife so that there will be nothing to falsely guide the patient back to that position when the rim is replaced in the mouth. If the record is not trimmed, the patient will easily be guided back to the previously recorded position even though that position may not be correct. The trimmed record is put back into the mouth for verification. If the patient closes easily into the record without any deviation, it indicates that the patient can repeat the record and implies an accurate centric jaw relation record. The centric relation at the occlusal vertical dimension is recorded with the use of positive keys. When the maxillary arch requires a complete denture and the natural teeth of the mandibular arch are present, the same materials are recommended for use as when both arches require complete dentures. However, the methods for these situations are different. When natural mandibular teeth are present, centric relation is recorded at the occluding vertical dimension with recording material in the first molar and second bicuspid region only. When removable partial dentures are required, there are several basic situations possible. The partial dentures may be opposed by natural teeth or by another partial denture. Furthermore, in each situation the partial denture may be tooth-borne or free-end. In the case of tooth-borne partial dentures, materials are not necessary to make the centric jaw relation records because centric occlusion can be determined on the cast by means of tooth contact of the remaining teeth with the opposing teeth. The centric occlusion should be recorded on the cast with three widely spaced vertical witness lines which are placed on the labial and buckle surfaces of the opposing teeth. For free-end partial dentures, either unilateral or bilateral, auto-polymerizing resin should serve as the material for the record base. A removable partial denture framework can also serve as the recording base after it has been properly fitted to the mouth. Plaster or zinc oxide eugenol, which are both truly plastic in their liquid state, are recommended for the recording material. After the framework has been fitted to the mouth, acrylic bases with wax rims are attached. When the opposing arch is to be restored with a complete denture, centric relation and the occluding vertical dimension are determined according to the doctor's preferred method. We have used a small interior stop made of modeling plastic to record this position. It is highly visible, the material sets hard and thus acts as a positive check when doing the subsequent steps. Several v-shaped wedges are cut into the maxillary wax rim in the bicuspid first molar areas. These wedges will make positive indices in the recording material. It is also advisable to place some mechanical retention on the lower rim to hold the recording material since neither plaster nor zinc oxide eugenol will adhere well to wax. The maxillary base plate is placed back into the mouth. The recording material is prepared, placed on the wax rims, and the framework is also placed in the mouth. Here we are using impression plaster. The patient is instructed to close until the lower anterior teeth have seated into the anterior centric stop. The doctor must observe carefully to ensure that this occurs smoothly without any deviation or deflection which might indicate that the patient is not closing correctly. The recording material is allowed to set completely. The appliance frame and record is removed from the mouth and carefully trimmed with a sharp knife. It is then returned to the mouth to verify accuracy. All of the procedures that we have discussed as well as those which we will discuss in the second videotape in this series have been designed to aid the RDA in fabricating the prosthodontic appliances that will meet our patient's needs.


Good morning. Thanks very much, Jim, for that introduction. It is really, really good to be here. And I mean, it's really, really good to be here. As I look at the devastation that has happened across the globe with regard to the pandemic over the last two, two and a half years, from a work perspective, I think this community, open source community, has really been affected the least from a work perspective than any organization in the sense that, you know, yeah, lots of knowledge workers were able to work from home, but they had to make that transition. They had to figure out how to do that, how to communicate, what processes they need to put in place, what tools they needed. So it was different, but for us, one day to the next was actually the same. We were used to working from home. We're used to working from disparate parts of the globe, but we come together every now and then to re-sync. And so events, conferences, are actually very, very important. And I've been doing this for about 22 years now, working in open source, and I'm always building bridges between companies and the developers that are part of the community with those companies or with other companies. And one of the things that I've noticed is it's really difficult to explain to management that hasn't been involved in open source, how important these conferences are. But then just recently, I've been seeing all of these articles and various leaders of various companies talking about how important it is to come back to the office. And so I took one just recently from a technology leader and I replaced the word office with the word open source event, and I replaced the word company with community and project. And this is what I ended up with. Open source events are a place where we meet other people, create social contacts, establish new friendships, and for some, we may find the love of our life. Well, maybe that doesn't happen that much here, but I do know of cases. These things are unlikely to happen in a two-dimensional team school. Maybe even more importantly, the conferences provide for a way to integrate new talent into the project. It is very difficult for people new to the community to establish a new network without meeting physically. Conferences facilitate the transfer of skill between people. It is a way to develop as a professional and a way for inexperienced colleagues to learn from more experienced. I also believe that we need to meet in person to ensure that we have a uniform and strong culture in our community. That was just swapping out those two words, but I think that's an extremely good description of why coming to events is important. And I expect for many of us, as we ramp back up, getting permission to come to conferences may be difficult. Managers often think of conferences as the boondoggle where an employee sluffs off for a week and drinks a lot of beer. And while maybe there might be beer involved, there's a lot of good things that happen. And that's the type of interworking that we do when we come to conferences. So use this type of, you know, find out what your leaders are saying about coming back to the office and actually use those arguments for why it is you come here. The other thing I want to talk about briefly is just where we are from a perception standpoint and a pervasiveness standpoint with open source. 1991, Linus initiates the Linux project and they had about 10 years where everybody left them alone and they actually formed their own culture. They built the processes that for me, I know today as open source best practices. I started in 2001 when HP really got involved in Linux and that's when the rest of the big tech folk got involved as well. And I learned what it meant to be an open source project by what I saw in the Linux community. And I still to this day consider that the gold standard and the way open source projects should be run. So when I engage in another open source project, it's that model that I look at. Between 2001 and 2012, there was a lot of fear, uncertainty and doubt about open source. We had the scotch lawsuits against Linux. Everything was really unsure and a lot of organizations didn't know they were using open source. When I was at HP, I was working with a consulting group and we would go to, we would go to the financial services organizations and we'd go to their C-suite and we'd be talking to the CIO and as well as a bunch of their lieutenants and such and we'd say, so how are you managing open source coming into your organization? And invariably, their answer would be, well, we don't have to worry about that because we don't have any open source coming in. And also invariably, there would be some junior manager shyly raised his hand and said, well, actually, yeah, we got that DNS server as well as, you know, the email server and I got that lamp stack running. And so they realized that they now had open source in their environment and they had to do something about it. They needed to manage it. So the to do group open source program offices started to take off and that's how that occurred. A lot of that fear uncertainty and doubt went away around that time and in the teens, we've had an explosion of open source, right? So we went from tens of thousands of projects to tens of millions of projects. And that's what's happened over that decade coming up into 21 and 22. All good stuff. Again, you can't really start a platform. You can't really start an application without picking your open source components. But it's also the case that now in 2022, the world has woken up to open source and its ubiquity. Back in that 2012 period, it was organizations realizing they had open source components running in their data center. Now everybody has just woken up to the fact that any commercial application you buy has a lot of open source in it. And so it now matters to a lot more people. And I just want us as a community, as we're looking at what we're doing, as we're saying, we need to be more secure. We need to be more secure. That's true. We've always got room to improve. But the point being is that we're actually in a new era in the sense that everybody now recognizes the ubiquity and the value and the importance of open source. And I'm surprised that I can make this statement to pretty much any technology leader now. And I get zero pushback. Modern software development is based on open source. So just to say that it's 80% open source or whatever, that doesn't quite do it.  What's my storage going to be? What's my message bus going to be? What's my fundamental platform, be it containers, be it AI, whatever, you pick those components and then you build your logic to actually fit those pieces together. So we start with open source. And recognizing that as a company and as anybody who's using software, which as we know is pretty much everybody at this point, that's an important statement to make. I know that Ericsson understands it. That's why we actually are platinum members of the Linux Foundation, the Open Infrastructure Foundation, as well as the Open Software Security Foundation. We're here primarily today because we're continuing to expand. So we're hiring embedded Linux developers, Yachto developers, CICD engineers, Kubernetes, particularly in the networking space. If any of those sound interesting and Ericsson sounds like an interesting place to work, come talk to us at the booth. And with that, thank you for your time.