Proudly we hail from New York City where the American stage begins here is another program with a cast of outstanding players. Public service time has been made available by the station for your Army and your Air Force to bring you this story as proudly we hail the United States Air Force is entitled Mediterranean Rescue the story of Captain Kendrick Reeves United States Air Force the story of an amphibian pilot with the 58th Air Rescue Squadron seventh Air Rescue Group stationed at Willis Field in Tripoli, Libya the names of the other characters have been fictionalized in this story of a rescue mission that won for Captain Reeves a recommendation for the distinguished flying cross America looks up to her men in the sky yes our country looks up to the young men thundering their way to new glory in the skies theirs is a task held in high esteem by the entire nation you are needed to swell their ranks and can do so by enrolling now in the aviation cadet training program of the United States Air Force if you are between the ages of 19 and 26 and a half have had two or more years of college and are otherwise qualified visit your nearest United States Army and United States Air Force recruiting station and ask about the aviation cadet training program do it today and now your army and your Air Force present the proudly we hail production Mediterranean Rescue what you hear is a commercial airliner an American made DC 3 the British nicknamed it Dakota it took off from Malta less than an hour ago now somewhere over the Mediterranean it's heading toward Benghazi on the Libyan bulge a scheduled flight a routine flight let's meet some of the passengers first Mr. and Mrs. Bentley Margaret yes Joe tired oh no no dear just relaxing and thinking what a wonderful holiday this is being yes I'll regret when it's over to you so will I I hope we don't find Tewkesbury rather dull after this excuse me I'm sorry I didn't see you standing there my watch has stopped I would like the correct time I think so five past one thank you that is correct what is the time I have no reason to question it sir I set my watch at the airport in Malta by the clock of the waiting thank you does not matter well that's a pompous bore who is he I think his names may take I must have slipped into an air pocket all right now quite all right and they were quite all right but only for the moment in the meantime at Wheeler's Field in Tripoli Captain Kendrick Reeves and his co-pilot Captain Douglas Clayton were whiling away the last few minutes of their duty to her on alert 1335 keep moving hands oh stop watching that clock will you Doug you're making me sleep who's gonna watch it if I don't 25 minutes to go and then freedom sleep for chance to dream hey and that reminds me I never told you about that dream I met the other night what another one this was it the most gorgeous the most Ken I've never seen anything like her what happened what happened what happens every time I meet the woman of my dreams she has a husband or a boyfriend I'm the only guy in the world who seems to miss the boat every time oh don't voice yourself Doug I'll get it thanks for Reeves talking yeah okay Tom right away round up the boys and oh yeah make sure it's the SA 16 albatross because we may have to make a water landing thinly wants to see me there goes my sleep and mine head moving Doug this trouble on the water that's all Tom that's all I came through Ken could have lost its left engine tried to make it a Benghazi on the other one but couldn't one fan just wouldn't hold it they're losing altitude and preparing to ditch how long ago 15 minutes they were dumping all the luggage but they still couldn't maintain altitude was 1340 now didn't you have anything more from them not a peepkin nobody else did either I checked with the RAF base at Castel Benito and with the other okay Tom what was the position 34 degrees 10 minutes north 1710 East how many people aboard 32 including the crew 32 Dakota yeah I know what you're thinking with every break in the world they'll be lucky to stay afloat five minutes okay I better get going it might be one outside chance they get out of that plane at 1400 on the nose two o'clock Reeves Clayton and a crew of four took off in their Grumman albatross and headed toward the position reported by the distressed Dakota then an hour and 35 minutes later at 1535 34 degrees 10 minutes north 1710 East this about it can we're in the area the Dakota sent the distress call from see anything Doug no not on this side there's nothing on this side either check with radar see if they can see anything on the screens look at those swells kicking up must be at least 10 feet high what a time to get rough even if those people managed to get out what chance would they have against that even if they managed to get out if they had the time what do you mean Ken well Dakota might stay afloat at the most about 15 minutes those waves down there Doug get in touch with other search and rescue craft get information we'll make a route search along the Dakota track our route search along the Dakota track the track from Malta to Benghazi but no one knew whether the Dakota tried to return to Malta or flew toward Benghazi Captain Reeves had to use his judgment he followed the track toward Benghazi nothing yet can nobody has any information nobody sees any sign of a single survivor yeah 1600 Ken if there were any survivors we'd have seen them by now somebody would have seen maybe look at that water now it's getting rougher by the minute wouldn't you know it it's beginning to rain we're not quitting Doug who wants to quit that's my boy there's one possibility is anyone covering the track from position back to Malta yeah a couple of Lancasters okay we stick on the track to Benghazi two more minutes then we'll swing left that's probably the direction the Dakota took on account of the dead engine before they hit before they could let anyone know but he's even looking for us that is your capitalistic competence your bourgeois humanitarian life means nothing we don't care if your lectures Mr. Lady you will get my lectures Mrs. Bentley fly to Benghazi to get lost in the middle of the Mediterranean die like a wet pig and I do not appreciate your British courage or your British calm oh shut up will you you at least are in the rot my husband is that is my folder because there is no room for him you made sure I'm getting your face didn't you of course why not my life is worth more than all of you put together you precious life doll I'm all right are you sure yes how about you don't worry about me I've been hanging on to your arms alone don't worry about me told I could perhaps I could hold on to the side of the round no let me try only for a moment don't you dare let go of my arm again you almost went down I guess I'm more tired to help me lift him on to the rock Bentley you will not bring him into the rap you won't help me I'll do it myself I put it you're the crowd of ten people when there should be only seven the rest is losing air we will sink we will drown hold up don't look out you're capsizing the rap I can't do it myself no one Mrs. Bentley they're all sick they can do nothing for you essay 16 AF 907 to searching for Dakota survivors have you seen any over got somebody Doug yeah I can't hold it okay Baker and then keep it contact transmit period tones for VHF homing purposes we'll be listening to give you a steer over well how's the commercial airline what's the word nothing everybody's helping out but what good will it do I don't know 1740 pouring markets I'm just afraid we're four hours too late maybe we're not what do you mean look down there maybe we're not too late at all holy smoke one two three rats loaded send out a position Doug let them all know right all search and rescue craft all search and rescue craft essay 16 AF 907 to calling have located survivors have located Dakota survivors here's position surface craft proceeded once to this position 33 degrees 28 minutes north 1921 East repeating position of Dakota survivors 33 degrees 28 minutes north 1921 East will lead assistance here's hoping we see you soon and it's hoping we don't get ourselves banged up going into that green I better talk to the crew pilot to crew we've located survivors I'm gonna be tough getting down to them and staying with them but we've got to do passenger safety belts and prepare for water landing Doug yeah me a reading on the wind coming from the East 20 nights okay I'm gonna make a full flat full stall landing into the wind downwind from the raft good luck puppy holy smoke that rat knocked over by a wave we're listening to the proudly we hail production Mediterranean rescue our story will continue in just a moment after this important message America is depending on her leaders in the air if you are a young man between the ages of 19 and 26 and a half have had two or more years of college and are otherwise qualified you are eligible to join the ranks of America's leaders in the air you can become an aviation cadet the defense of our nation hangs heavily on our air strength the finest in the world but we cannot relax our efforts the Air Force still needs pilots and aircraft observers if you have the primary qualifications visit your nearest United States Army and United States Air Force recruiting station today ask about the aviation cadet training program do it now you are listening to proudly we hail and now we present the second act of Mediterranean rescue just imagine yourself in the cockpit of Captain Reeves playing just imagine yourself seeing what Captain Reeves saw as he brought his plane down on the heavy seas of the Mediterranean the people in the life rafts some shrieking with joy not seeming to mind the rain anymore some praying and some prostrate and numb with sickness unmindful of anything and the people in the water trying to swim for their lives clutching at the waves as if they were solid agents of salvation Captain Reeves made his full flap full stall landing into the wind several hundred feet from the survivors but as his plane made contact with the water it skipped off the water bounced like a rubber ball then when it made contact again duck place propellers in reverse to play power the plane squatted remained on the water it was undamaged in the landing rock of my baby what now can we'll tax it within about 50 feet okay should get closer but I'm afraid to take a chance might be swimmers we can't see I hope they're all there can yeah okay Doug we do power all right take over Doug I'm going to get the boys started right keep jockeying the throttles between forward and reverse thrust so we can hold on to this position right probably for you or anything hey wait a minute Ken matter some guy swimming toward the plane it looks like he's got a rope yeah yeah it is a rope and it's tied to the rats good at least they won't drift okay okay we're ready Jim Joe open the port main hatch so be ready all the blankets hey we got enough do you know captain 32 people we didn't figure on the rain and that raft getting spilled okay those don't get blankets will need dry clothes just have to use our own okay with me captain so long as you don't take my pants I bet we won't have to 32 people are we gonna put them well let's not worry about that now Jim Joe and look up the guy down there below hats from a line don't hit him on the head with it I ain't that good captain I used to pitch for the joints just wrap yourself in this blanket thank you sir James and co-pilot Reeves come on fellas put your muscle behind that tow line bring in those rafts sit down Jamison Jim give the inflator raft and pick up everybody you find the wire now Jamison when you ditched your plane was anyone hurt none of the passengers that I could check with Reeves of course most of them are sick and getting sicker how about the crew the pilot captain Harvey broke his arm he was thrown against the panel where is he he was in the second round the one that capsized no okay when when you loaded the rafts was everybody accounted for yes I made the check myself since we lost that raft I don't know Reeves there was 10 people in it okay I get some rest all right get that raft to float on the double Doug we need surface craft to get these people back to port we love it to ourselves I've got the plane off the water with a load we're gonna have I know I know back won't you get through to anybody I've been talking every ship that's in the area and there's one on the way here now which one the SS is serious great how soon she be here not before it's almost three hours I know Ken but she's getting here as fast as she can we got to do better those people out there have been kicked around for six hours already some of them may be dying the skipper the plane is a broken arm we've got to get help and we've got to have it before 2100 captain Reeves with the help of his two men hoisted the people out of the rafts and into the plane 20 exhausted sick and hysterical men and women 20 items of dead weight who were too numb from the ordeal to help themselves the operation took almost an hour then okay okay that doesn't for now rest of the people in the water let's hope that we can find them let's hope happen getting dark it's tough break Joe getting his so late Joe take care of those people fix them up with blankets Jake you better have some hot coffee too sir that's too bad so be captain it's not with you feeling sick captain rocking of the plane Pete said so be I need you can you hold on for a while I don't know captain I like okay to the best you can keep a watch on the hatch if the raft comes below sing out yeah Jamison what's the situation any real sick ones I don't know Reeves they seem to be suffering mostly from shocker exposure to her unconscious mrs. Bentley and miscontract looks more like exhaustion and everything else how you're standing up okay let's take a look at your pilot Captain Reeves yeah I'm Anton Ladeck diplomatic Korean a very important mission yeah I must get to my legation Benghazi at once how soon can we leave here how soon yes yes how soon I'm expected in Benghazi already I'm six hours behind schedule Mr. Ladeck there are 12 people out there in the water what do you want me to do leave them at this most embarrassing to me this delay sorry British destroyers around we might get here in about an hour duck an hour is no good I know that Ken but what can I do there's nothing closer how about the Asturias where's she now still proceeding under full steam but no dice before 2100 that's still an hour they're sick they need medical attention a couple of delirious the pilot Captain Harvey his arm is broke and have we got the whole Dakota crowd on board I don't know Jameson's County noses now okay let's say we've got him we can't take off with that load but maybe we can taxi maybe those swells might stop as cold okay Doug we'll take the chance if Jameson says we've got everybody excuse me mr. Ladeck you're in my way I must know Captain Reeves why are you wasting valuable time I look mr. Ladeck already it is dark already I'm more than seven hours late reporting to my legation in Benghazi I'm a diplomatic courier what are you waiting for we can't take off mr. Ladeck we've got 32 additional people on board besides the crew is six that makes 38 people I can add Captain Reeves okay then figure it out for yourself I've got sick people wait a minute you will concern yourself with me too Captain Reeves and you will stop giving me excuses I have diplomatic privilege and I demand that you take me to Benghazi I'm just a minute Ladeck don't you pull diplomatic privilege on me I'm in command here and I'll do what I think is right we're not behind the iron curtain in case you don't know you get out of my way I've got work to do you will not insult me in this way I shall report this incident to my government I shall complain about your conduct I share somatic I sick very sick inside yeah I know the symptoms the port main hatches still open stick your head through and let yourself go Jameson missy spently Reeves she came to a few minutes ago her husband I finished by counting he's missing oh Joe soupy yeah what's cooking chief man out there how do you feel so okay now it must have been something I ate out that even more raft you guys and start fishing for that man on the double we were doing all we can your husband's anywhere out there we'll find you'll never find him I wanted to bring him into the rock I tried man refuel he wouldn't help he wouldn't raise his hand miss Bentley I was holding on to Joe with all my strength and then I don't know what happened something happened to me I don't remember they tell me you lost consciousness yes that must have been it but that man Joe would be here he'd be a lot all day we waited so long as Bentley I don't know what I can say except we won't leave here until we're sure absolutely sure help should be arriving soon the ship may be two that'll get the others to court but we'll stay thank you Captain Reeve thank you so much but I feel it's quite hopeless maybe not let's wait and see please excuse me now of course thank you Jamison you think there's a chance your men will find him who knows those conditions out there it'll be a miracle how's your pilot I don't really know but I'm terribly afraid for it his arms badly swollen and discolored that's too bad try to make him as comfortable as possible I gotta check the radio Reeve my compliments you've been superb thanks a code stuck kept my chin up so long I've got a stiff neck any word from the rest of the world looks like it's gonna be the asterisk she's really bearing down between you and me I'd be glad to see a robot right now as a comrade coming along late egg groaning his guts out in the corner that's where he belongs what about Joe and soupy you seen them lately I've got all the lights on wings nose and tail getting worried about those guys can't so much it's not half an hour Doug if that guy was a float that a fauna by now yeah only hope they didn't get kicked over by a way well there are a couple of good swimmers can they could have made it back to the plane maybe but it's building out there maybe I was crazy to send them out but I don't know Doug we just couldn't forget the guy leave him we couldn't turn up back on him you're a tired poppy can listen your grandfather will you don't blame yourself you did what you had to and you did it fine but Joe and so Joe and soupy it'll take more than a Mediterranean to keep those corn cobblers down look out there where right there in the beam of that light the raft yeah and there are three guys they picked up that what else captain or the essay 16 his ordeal was over the ordeal of clinging to an overturned life raft for over three hours and 15 minutes later the asterisks arrived on the scene the ordeal was over for all of them the crew and passengers were transferred from the plane the sick and injured were promptly taken care of rescue mission accomplished to captain Kendrick Reeves USAF for courage beyond the endurance of men for the exercise of sound judgment under the most difficult conditions for the tenacity with which you stuck to your assignment until it was successfully completed proudly we hail salute you you captain Reeves your co-pilot and the men of your crew before we say goodbye to captain Reeves let's go to Wheeless field again where he and Doug Clayton are early on their way home after filing their report boy am I gonna sleep for chance to dream boy what dreams I'm gonna have I listen to happy if I want to dream I'll dream I'm off duty now I wasn't laughing at that no I was I was thinking of Anton latex nightmares his diplomatic mission is in Benghazi maybe he doesn't know it yet but the asterisks is bound for Malta yeah he's going back where he came from yeah young men of America your country is building a mighty air force to maintain the security of our nation this means that there is a job for you the chance to do an important job with one of the finest organizations of its kind in the world today the United States Air Force if you are between the ages of 19 and 26 and a half have had two or more years of college and are otherwise qualified the Air Force needs you as part of the aviation cadet training program for complete details visit your nearest United States Army and United States Air Force recruiting station today this has been another program on proudly we hail presented transcribed in cooperation with the station proudly we hail is produced by the recruiting publicity center for the United States Army and United States Air Force recruiting service this is Kenneth Bangard speaking and inviting you to tune in the same station next week for another interesting story on proudly we


Thanks, I want to talk a little bit. I was originally going to talk about memory error handling, but a few topics have come up already in the memory management sessions, which kind of overlap with an issue that I'm currently having with huge TLB PMD sharing. And I put up there that yeah, this technology is almost 16 years young. So it's not really something that we would normally talk about in LSFMM. But it wasn't something that I was actually aware of until I started looking deeply into the huge TLB code. So just as a brief, very brief overview, processes can share PMD pages if they do huge TLB shared mappings. And as I said, we actually share an entire PMD range, which means that that's a pod size sharing, which is usually one gigabyte on x86. And one of the reasons for doing this, it wasn't really explicitly the reason that it was originally done is memory savings. And one of the examples that our database group likes to give is, you know, a one terabyte shared mapping, they have 10,000 processes sharing that mapping. If you do the math that comes out to about 39 gig worth of memory savings, if you do PMD sharing. So just at a very high level, this is only a four to level page table but this is kind of what it looks like you have process a process B. They both point to the same PMD page and that's where the actual data pages get addressed. So you set up sharing like this. There's also this routine called huge PMD on share. And what happens is is that whenever you change the attributes like doing and protect read only mapping or a range of a mapping, we have to on share that shared mapping. And so what happens is is that we actually clear the PMD pointer in the PUD. So we effectively on map a PUD size area.  And so when we call huge PMD on share, it has to hold the page table lock. That seems kind of obvious since it's messing with the page table. And it also has to hold the IM map SEMA. And what that is is just the sharing is keyed off of the ref count of the PMD page itself. That's kind of a very brief overview on huge PMD sharing. So a couple of years ago, I stumbled upon this kind of very ugly race in the code. And what happens is is that usually during a page fault or various other places, you will look up, do a page table walk to get a pointer to a PTE. And or maybe even just do in the page fault code, allocate a PTE, you'll get a pointer to that thing. But the problem is, is that another thread could theoretically at the same time be doing a huge PMD on share, which means that that PTE pointer that you just looked up or that you know PTE that you just allocated. That is no longer valid. It could more than likely point to the page table of a process that you were sharing with or worse yet I guess it could even theoretically be unallocated and point way off somewhere else. So using that PTE pointer, even to do a page table lock, because it is kind of what you key off of to get the page table lock. That's invalid certainly writing to that PTE at that location is invalid. So that looked really scary to me. And once I knew that that was actually possible. It wasn't too hard to write user space code to actually hit that race condition. So that scared the heck out of me a couple of years ago, and resulted in this commit that's upstream today. And what this commit does is actually uses the IMF SEMA for synchronization. So as I mentioned earlier, you have to hold that semaphore in right mode when you're doing an on share operation. So my thought was, and this is what is in the code today is is that we hold that same semaphore in read mode during fault processing, or any time that we have to look up, do a page table walk, and look up and get the address of a PTE pointer, and use that for some purpose, you know, writing to it or doing something with that. The good thing about that is that faults can run in parallel. The huge PMD on share is blocked during false because it does require right mode, and all is safe, but the problem is is that that that that IMF SEMA is held for the duration of truncate whole punch. But maybe even more importantly, unmapped operations. So that can cause quite large delays in fault processing, and even more significant, I think, is is that, you know, process X, which is unmapping the shared area can cause delays and processes why page fault handling. I actually pointed out by on the mail this not too long ago, and I thought, Well, how bad can this really get so I took a test system with 48 CPUs, 320 gig of memory, and just have a very simple process that does in an infinite loop and maps a 250 gig file as shared mapping faults and all the pages and unmapped it have 48 instances of that running all in parallel, and run that and see, Well, what would be my worst latencies for faults out there. And as you can see from this slide, running that for a couple of hours, I end up a worst case of over a two second latency during fault processing waiting for that IMF SEMA lock. So, not really pretty. I started talking about different ways to address this upstream. The only feedback that I got from was from David and I think David said, Well, let's just disable huge PMD sharing which some people may not really agree with. The approach that I've come up with is something that was actually discussed a little bit when we're talking when Liam and Matthew were talking about MAP SEMA. So more scalability there. And so what I'm doing and is actually have some patches prepared to do is actually revert that patch that uses IMF SEMA for the synchronization. And this may sound familiar to that talk is actually add a per VMA semaphore for this PMD synchronization. And this would be huge TLB specific, I would hang it off of the VM private data field so it's not nothing that would be added to VM struct itself, and only added to VMAs that are capable of sharing. And just like how IMF SEMA is used today in upstream code it's held and read mode for false and write mode when calling huge PMD on share. The good thing about this is it actually limits the scope of contention in some respects, still things like truncate hole punch that can be done by anyone and migrate as well. When you do an unmap is only, you can only block threads in the same process in the same MM struck the same address space. So, just some quick comparisons as far as that same benchmark test week. If you'll notice here, our maximum fault wait time goes down for him, you know, two seconds to, I don't know, a little bit over a tenth of a second here. Our overall wait time isn't significantly less but our worst case seems to be quite a bit better so I was just curious about thoughts on this approach. I know that during David's talk about page table reclamation. The whole issue of trying to keep page table pointers stable while you're doing something like a page fault has to be addressed with I haven't looked at that series. But it seems like they're using, you know, sequence counts or something like that to keep that data stable. That may be overkill for this here. I don't know just Monday, get some additional thoughts on this. No thoughts. There's a lot of eye rolling in the room like I think you've broken a lot of people's brains here. I think Liam's hyperventilating in fact, there's one more kind of interesting data point here is that, you know, the upstream complaint was about this maximum fault wait time or actually they weren't complaining about a specific number, but it was another database vendor who was using huge TLB for shared data. And they just noticed latencies went up when they went with a more modern kernel. I'm thinking it's one with that new change in locking. So I went back to see, well, how bad, or how good was the scalability originally. And I just put this together last night so I'm not 100% confident in the numbers but if I just revert that and go back to the we're not safe where, you know, we could potentially corrupt data. The interesting things is that the maximum latency during fault is even worse than it is today.  So I don't think, you know, I still think we have the potential for, you know, we had the potential for these very long fault delays in the original code. And I guess that we had less opportunity to actually wait. And that, and that can be explained just because of the area of the locking we only did that when actually doing a PMD Alec and noticing that there was not a PMD page there and then had to actually kick in the locking to to set that up. So that's all that I had like says hoping for a little bit of comment. When David said tear out the huge PMD sharing I said well okay we'll just use M share instead of that but after this morning session I don't think that's going in next week or anything. What type of lock did you put in. Excuse me, what type of lock did you put in, and where did it where did it live again. So what I did so this is, Liam this kind of the it sparked my interest during the talk about the MAP semis scaling that you had earlier, maybe on Monday. So I actually added a per VMA lock. What type of lock. It's just a read writer semiform. Yeah, those will spin though. We're, we were thinking of an invo, we're thinking of flag or something for in not invalid inactive. Whoa, what just. We're thinking of putting some sort of flag in the show that it was in what was in the middle of an operation. So if you did that it might actually, is he gone. Okay, I'm here. Yeah, sorry the screen went blank. Sorry. So we were thinking of putting in a flag that would show that there was some sort of activity on the VMA and you would restart you try for anything else. If you use an RW sem, I think those things will spin and it will spin if there's any activity. So if there's any activity in fact it will get worse. So it might be you might be hitting that that making affecting your results in a negative way. You might have better luck with a different type of lock locking strategy. I don't know. I mean, it's the right lock to use. If the RW sem implementation is bad then it needs to get fixed. Well, it's, it's not bad. It exists for, for a reason. It's just not for this. Well, I don't know. I think the way he's using it is fine. My, my point, the point I wanted to make was that I kind of cringed a little bit when I saw, oh, yeah, we do huge PMD unshare when you do MPRTEX. Because I got thinking a minute, the database people said to me that what they really want is for MPRTEX to affect every process that is accessing the database. Yeah, that's what they want. They don't have that with the huge PMD sharing. So if we had the magic flag that we talked about in the earlier session that said, no, really I want you to share the page tables. I want you to share the MPRTEX information. And I think that's a good thing, as if things were multi threaded, but they're not for reasons. How much of this problem goes away. Yeah, I think most of it would. That's of course without looking at all the details and I think the suggestion was is to actually store somehow reference those page tables based upon your sharing object like the I know it or something correct. The model that we discussed yesterday where we have a flag in each VMA and you just use that with that work. Well, because this is transparent, we still have to have user space opt into opt into it because we can't just say, oh, yeah, we transparently shared the page tables and now MPRTEX affects everyone who has it and that without everyone who had it and that opting into believing in that. So if we keep it the way it is, and Mike's solution gets changed to use the RCU way of reading things and inactive flagging this particular special VMA. I'm not sure I can't picture it off of my head I need to think about that for a bit longer. Is David in the room I mean one of the things that like I say I haven't looked at those patches yet but so is it sequence counting that is used to keep the page table in a stable state while somebody else could be reclaiming or removing part of it. It's not really for when I was talking about the sequence lock the sequence lock is only used to synchronize for example in our for code against our get user pages code and it's not for synchronizing kind of page table activity. It's just to tell somebody who operates on the on the page tables to try again like to undo whatever you did try again. There was something in between for page tables especially under a good fast code. We actually use I think there are like the two variants either via RCU or via via to be flush where you can synchronize against that. So what you would do is after you you perform your changes. For example if you would want to rip out a page table you would rip out a page table not free yet you would do a TV flush and then you can free defer it because then you know like for example fast is no longer walking over it. If you need something similar the question would be like can you also defer for example some kind of freeing after that. But I mean the sequence count could eventually work just to tell whatever was walking and doing stuff to try again because something changed. But like the problem of access after free is a different story the sequence count is really just to tell somebody to try again because something changed. Yeah. Yeah, I think there I may be able to do that with some kind of a sequence count as well. What happens actually like if the last user of a shared PMD goes away because like the last one called PMD unshared will the page tables then get freed or how exactly does it work. Well it's actually when the second to the last user goes away, then basically it looks like just one person with their normal page table setup. Okay, so my gut feeling is that maybe good fast code isn't probably synchronized with that and we might have some weird issues already. That's my gut feeling but we should look into that. But yeah, I'll have to look into the details but maybe reusing even the same sequence count or something like that could already work and fix some other unexpected cases. Okay, well, that's all I have but I'm going to probably send out this VMA locking as another RFC. Just to see what people think about it. So thank you.


What are the advantages of 8A for a new business? I would never have gotten into business without the 8A program. Okay. I mean, I was a solopreneur consultant, but in order to grow, it allowed me an avenue to get more revenue. Okay. All right. Now, you were a solopreneur when you got into 8A. How did you even know to go into 8A? Because that woman in my industry? Yeah, but yeah, okay. But how did you know it was something that like would help you that would grow like this? Did you read it somewhere that someone tell you like that it would help you with your business? Well, I mean, at the time I saw her go from zero employees to 150 in like three years. Okay. There you go. All right. Okay. That makes a lot of sense. And then you said, I've got to do 8A. Now, again, we talked about a new business not being able to do it, but how are you guys able to do as a new business? Well, I did have some revenue behind me. Okay. Now, one of the things they found with my 8A application was we had filed as a single member LLC. Right. And with the IRS, we were multi-member. So in those two weeks after I applied, I had to quickly bring on my daughter.


Here's a new word for today herbivory. Now we know that carnivores eat mead and herbivores eat plants Herbivory is a nickel-plated word for what herbivores do and that is to eat plants Now elk grazing and deer browsing is herbivory a sage grass pecking on sagebrushes herbivory Hey a road neat and grass is herbivory Wyoming for eons has always been home to herbivores Anytime we look across Wyoming's range land landscapes There are dozens of species out there inflicting herbivory upon the range land plant resources The components of today's range lands are result of the influences of the past The plants and animals of the range land have evolved together and herbivory makes that linkage of their coexistence So here's a question to ponder Without our range plants would we have our range animals which evolved to take advantage of the range plants? Well the answer is no and without the herbivory of range animals We wouldn't have the range plants we have which are a result of that co-dependent relationship from the University of Wyoming Cooperative Extension Service. I'm Eric Peterson


I am Sachin Rathod working as an assistant professor in mechanical engineering department from Walsh Institute of Technology, Sollapur. Today we are dealing with simple gear train. So the learning outcome of this session is student will be able to calculate speed ratio of simple gear train. So simple gear train, generally the gear train are used for reducing the gear ratio or the speed ratio. So in that simple gear train if on one shaft only one gear is mounted then we can transmit the speed or the power from one shaft to another shaft by using the different gears mounted on the shaft. So in this case the shaft are parallel to each other. The gear having the input shaft and this one is the output shaft. So on this input shaft and output shaft the input shaft is called as a driver and the output shaft is called as a driven or follower. So here we are giving the input speed and we are getting the output speed. So here we are occurring the speed ratio, we have to find out the speed ratio. So the speed ratio is nothing but the speed of driver divided by speed of driven. So in this case this is gear number one, gear number two. Gear number one having the tith t1, gear number two is having the tith t2, gear number one is having the speed of n1, gear number two is having the speed of n2. So we can find out the speed ratio is equal to speed of driving n1 by speed of driven n2. So in this case we can easily find out the speed ratio. So suppose we have taken two gears, gear number one and this is a gear number two. These are the PCD, pitch circle diameter. So from the gear number one to the gear number two we can transmit the power or the motion. Suppose the gear one is rotated in the clockwise direction, gear number two will rotate in the anticlockwise direction. So in this case this PCD, so if the two gears are mating with each other the module of the two gears should be the same. So the module m is given by the ratio of PCD divided by the number of the tith. So gear number one and gear number two having the same model. So we are getting d1 by t1 is equal to d2 by t2. So we are getting the relation d1 by d2 is equal to t1 by t. So we are having another relation the PCD of this gear that the two gears are mating with each other at that time the pitch line velocity should be the same. So we are getting pi d1 n1 is equal to pi d2 n2. Therefore we are getting d1 by d2 is equal to n2 by n1. So we are getting the relation from equation number one and two we are getting n1 by n2 is equal to t2 by t1 which is the equation number three. So we are getting the one relation between n and t that is n1 upon n2 is equal to t2 by t1 that is the ratio of the driving gear to the driven gear is equal to the number of tith on the driven gear to the number of tith on the driving gear. So the speed is inversely proportional to the number of the tith. So by using this relation we can easily find out the speed ratio. So in this first diagram A here the driver and driven so we can find out first one speed ratio is equal to speed of the driving gear that is n1 upon n2 is equal to now it is a inversely proportional to the number of the tith that is the t2 by t1. So if you observe the figure number b speed ratio is equal to speed of the driver this is a driven gear and this is called as the intermediate gear. Gear number two is called as the intermediate gear. Speed ratio is speed of the driver divided by speed of the driven n1 by n2 sorry n1 by n3 so we have to find out the n1 by n3. So in this case one is meshing with the gear number two that is n1 by n2 is equal to t2 by t1 and two gear mesh with the gear number three n2 by n3 is equal to t3 by t2. Therefore multiplying equation number one and equation number two we are getting n1 by n2 into n2 by n3 is equal to t2 by t1 into t3 by t2. So here t2 will get cancer and n2 will get cancer therefore we are getting n1 by n3 is equal to t3 by t1 therefore the speed ratio is equal to n1 by n3 is equal to t3 by t1. So here we are getting the speed ratio is equal to speed of the driver divided by speed of the driven is equal to number of the teeth on the driven gear divided by number of the teeth on the driving gear. So here we are not considering this number of the teeth on the intermediate gear. So similarly for figure number c we have to find out the speed ratio is equal to speed of the driver gear number one is a driver and gear number four is a driven therefore n1 by n4. So similarly we have to calculate the speed ratio of the n1 by n4. So for that purpose gear number one is matching with the gear number two so n1 by n2 is equal to t2 by t1 another n2 gear mesh with the gear number three n2 by n3 is equal to t3 by t2 another gear gear number three mesh with the gear number four n3 by n4 is equal to t4 by t3. So by multiplying equation over one two three we are getting n1 and 2 and 3 divided by n3 and n4. So here n2 will get answer n3 will get answer is equal to t2 t3 t4 divided by t1 t2 t3 t2 t3 will get answer is equal to t4 by t3. Therefore the speed ratio is equal to n1 by n4 is equal to t4 by t3. So here also we are getting the speed ratio is equal to the speed of the driver divided by the speed of the driven is equal to number of the teeth on the driven gear divided by number of teeth. So here we can think about this why intermediate gear is used. So here we can use this intermediate gear here the it is does not affect on the speed ratio. So we are using the intermediate gear for if the distance between the driver gear and the driven gear is more at that time we are connecting the driver gear and driven gear by using this intermediate gear and for giving the direction of the motion means if this gears two gears are there if the driver gear is rotated in the clockwise driven gear rotate in the anticlockwise. So if the driver gear is rotated clockwise and we require the driven gear rotated in the clockwise at that time we are using this intermediate gear. So there is a two use of this intermediate gear for changing the direction of the output gear and for increasing the distance between the two gear. So here one question is there the gearing machines as shown in the figure the motor motor shaft is connected to the gear one and rotates at 975 rpm the gear wheels 1 2 3 and 4 are fixed to the parallel shaft rotating together the final gear 4 is fixed on the output shaft what is the speed of the gear 4 the number of teeth on the gears are given below. So directly we are knowing speed ratio is equal to the speed ratio of this is a simple gear and because on the one shaft only one gear is mounted and it's called as a simple gear. So speed ratio is equal to n1 by n4 is equal to this is the speed of the driver the speed of the driven is equal to the number of the teeth on the driven gear developed by number of the teeth on the driving gear. So here in this question they are given us gear number one means the speed of the driver n1 is equal to 975 and they are asked n1 is equal to 975 975 divided by n4 is equal to t4 by t1. So number of the teeth on the 4 is 45 divided by number of teeth on the 40. So therefore we are getting the n4 is equal to 975 into 40 divided by 4866.66 rpm. So like this we are getting the speed of the outputs. So I have taken these two references. Thank you.


This is Silicon Angles, the Cube, our flagship program. We go out to the events and extract the ceiling from the noise. I'm enjoying my co-host Jeff Frick, and we're at the end of our three days wall-to-wall coverage, over 30 interviews, a lot of blog posts, great coverage, great event. But one of the things we're really passionate about at Silicon Angles is fostering innovation and sharing and providing open source content and doing all that great stuff. But one of the things we're really passionate about is getting the word out about getting more people programming. And our slogan is where computer science meets social science. And I'm pleased to have Anita Cano here from the Canome project. Yes. Thanks for joining us. And we want to talk about getting more ladies involved in coding, bringing women and open source together. Absolutely. And obviously women in tech is a big focus you hear a lot about. Sheryl Sandberg had just recently wrote a book called Lean In, and there's a variety of other conversations. There's always debates around, oh, not enough women to see you. But that's not about it. This is about coding. So let's talk about, and what do you want to share with the folks here? Let's talk about this. Well, my exposure to OpenStack was through the Canome outreach program for women. And I'll just talk a little bit about what that means. Canome is a project that does desktops GUI for Linux operating systems. And they noticed in about 2006 that there was gender disparity and they wanted to work towards gender parity. And they tried a number of different things, and the only behavior that they found actually changed results in how many women were contributing and how many women were going and attending the conferences was actually doing internships.  And that ended up in more contributions to the master branch, to the code repo from women and also more women showing up to conferences and also continuing to contribute. That was in about 2006. And this program that I was involved with just, this was the first cohort in January of opening it up for other organizations. OpenStack was one of 10 organizations that took on women interns. And the requirements for becoming a GNOME outreach for program intern was be female or identify as female, not have done the program before and be available for the time commitment. The time commitment was three months, standard work week of 30 hours a week. So that is the program that I came into OpenStack with and it's been really, really beneficial because that provided me the support that I needed both emotionally, technically and financially, sorry, emotionally, financially and technically to get me over the hump with open source. Things are kind of in a raw state. They're always a work in progress. So they're not necessarily user friendly when you have to use them. And when you're coming in and you're coding, you have to create a development environment in which you can actually test your code. So make changes and test your code. And so actually creating that for OpenStack did present some issues for me and it was only with the help of my generous mentor and the whole internship team that I was able to continue to have hope and keep going back and asking questions that I was able to get that development environment created and actually start making contributions. And who was the internship with? Was it with the OpenStack Foundation or was it with a participating company that had OpenStack work going on? That's a really good question and it's really important to understand how this was set up. There were three sponsors for the internship. There were three positions and our three sponsors were the OpenStack Foundation, RackSpace and Red Hat. So thank you to those organizations for making those positions available. They sponsored the positions but the money was distributed by the foundation. So the foundation acted as an administrative umbrella organization. The money then went to the GNOME Foundation who were the umbrella organization for all 25 interns in this round and then they were the ones to actually distribute the money. So in terms of what company or corporation did I intern with, I don't actually know. It might have been a third, the foundation a third red hat and a third RackSpace. The point was is that all of the money came from the foundation so my loyalty, my emotional attachment and my gratitude goes towards the foundation. That actually freed me up when I was doing my internship. When I came in, I came in through one door working on the Glantz project and the Nova project and then I moved around. I did a commit to docs, I did a commit to Cinder and then various other things happened as they do and I ended up in the infrastructure team and so working with them. And because I wasn't limited by any one company I could go ahead and follow my nose and find out what was really attractive to me and just show up and start diving into code. So great experience, how do folks get involved? I know, first of all thanks for coming on. We really appreciate it. Thank you. This mission, so for the folks out there that want to get involved, other women with open source, website links and what should they do? Absolutely. If they're on IRC, they on the FreeNode server they can come to OpenStack-OPW, that's the channel I'm in. My IRC nickname is Antea. Ping me, say hello, say you watch this broadcast and I'd be happy to have a conversation with you. If you would like to contact Ann Gentel as the point person for OpenStack Foundation she's the administrator of the program. She'd be another good person to contact. What's her email address? Her email address is she's anngentel at Rackspace. I'm not sure of the exact syntax but if you do a search in Google at anngentel at Rackspace you'd be able to come up with her blog page and you'll get her email from there. And another way in is on the irc.gimp.org server and go to the OPW channel and that's where all of the OPW mentors and interns idle and ask for Marina or Karen SQ, those are the administrators for that program and they're actually taking applications right now for the next cohort that takes place this summer and they're actually looking for more people to apply and more mentors to apply if there's OpenStack people and wanting to mentor and they didn't know about the program before. Please contact any of those three points and let us know your interest and we'd love to work with you. Nina, thank you so much. Obviously there's so much to get involved in as an avenue to come on board into the communities. A lot of communities are welcoming more and more women. Also the ability to learn. I mean, Stanford offers free mobile development. The tools to get the COMSI education is out there. So if you want to brush up on something, there's a ton of resources. You've got Khan Academy for the younger kids but in the computer science, you've got Code Academy, you've got a variety of other avenues and I recommend going to the Stanford courses on mobile. They're a really great online course with the professors, great CS courses there. Thanks for coming on theCUBE. We want to see more women in tech programming and building code. Again, all it takes is an app in the App Store, throw it on some cloud, maybe build some OpenStack in there. Absolutely. And again, the job market will explode. Obviously, it's demand. That's right. So contributing, be having fun and building the market. So thank you so much for coming on theCUBE. Be right back with our next guest after this short break here inside theCUBE. SiliconANGLE's exclusive coverage of OpenStack Summit 2013.


prede ḍᵃᵃᵒᵉ  Responsibly I, who have been at aständ ḍᵃᵃ, ḍᵃᵃᵃᵉ ᵈᵃ ᵕᶠᵒᶘᵃᵉᵈᵃᵕᵃ  ᵕᵃᵃᵉ ᵕᵃᵃᵉ ᵍᵃ ᵕᵃᵃ ᵕᶠᵉ ᵕᶠᵉ ᵗᵉᵉ  dul qrer waw with two with two ᗜ' Ἐᰏ�isto ᗤაᗜ ᗡაᗛაᗛ꒺�刀ᗱ� Cena x Corporate Video


Hello. Thank you, everyone, for coming along. I understand, of course, this is getting on towards the end of the summit now, so you're probably all a little tired, but I appreciate you coming along. My name is Jeremy Phillips, and my better-dressed colleague here is Luca Cervini. We're from Pausie Supercomputing to talk to you about GPU on OpenStack for World Science. Now, before we start, just please be aware that while we did mention in our presentation brief that we would be talking about our results using RDMA, however, unfortunately, the results for that are not available at this point, so on that caveat, as long as that's not a deal-breaker for people, hopefully you'll be able to get something out of this. First off, just a little bit about ourselves. So Pausie Supercomputing, we're based in Perth, Western Australia. We are a non-incorporated joint venture between CSIRO, which is the federal government research organization, and four of the West Australian universities, Curtin, Edith Cowan, Murdoch, and UWA. We are also funded partly federally through the INCRS, the National Research Infrastructure for Australia Fund, as well as through the state government of Western Australia. Originally established in June 2000, primarily started with HPC and data storage, and just in recent years have started moving into cloud computing. We provide free computing resources and training to students, industry personnel, researchers, academics, and scientists, primarily, obviously, based within Australia, but we do have some other people coming from outside of Australia as well using our resources. From the HPC side, we have two Cray Supercomputers, an XC-30 and an XC-40. We also have about 10 petabytes of live storage and 40 petabytes of tapes. Quite a bit of that, obviously, is being used by the ASCAP, the Australian Square Kilometre Array Project Pathfinder, sorry, and, yes, on top of that, of course, we've got our cloud computing cluster. So, hello, everybody. So, our cloud facility is called Nimbus. It's not mentioned here, but it's an open-stack installation based on open-stack pike, and with some notes already, some services already in open-stack queens. I think we have Keystone and Cinder and Queens already. And we have 46 compute nodes, 39 storage nodes, and 12 service nodes, and the new entry that are these six GPU servers with dual NVIDIA V100 GPU card. In total, we have around 3,000 cores and one petabyte of raw storage with CIF. And the old cluster uses Wuntu 16 for the moment, and use Puppet and Mass as bare metal provisioning.  So, just so we take you through, briefly, at least, you know, a couple of the projects that are currently using our GPU nodes, just to give it an idea of the kind of fields that are being covered. So, obviously, we've got agriculture processing of multi-spectral imagery from remote sensing. That project is investigating land systems for water repellents in serial production and revegetation in low-brainfall farming systems. We've got psychology using TensorFlow to speed up sampling of large and complex Bayesian models. They're testing cognitive psychological models using statistical software to extend mathematical models of self-regulation to complex tasks involving rapid decision-making, for example, air defense. In biology, using molecular dynamics simulations to assess the interaction of glycans with their receptor proteins. That's NMR, spectroscopy, and molecular modeling techniques are used to find more about the structure, function, and dynamic of carbohydrates and glycans that play a pivotal role in cell-to-cell communications, cancer, and human pathogen interactions. We also have one, another interesting use case, which we've just done their own presentation recently on classification of shallow water fish, which is this one just here. So, basically, they, it's a deep learning model for real-time object detection and classification from underwater images. They use these baited underwater stations to attract the fish in, and normally they have trained oceanographers who do the manual classification, so instead they've started training this system instead. At the moment, they've got about an accuracy of over 90% on the validation data set, so that was run off of the GPUs. Just to sort of give you a bit of an idea of that work, which is cool. Well, we think it's cool anyway. Anyway, so you probably want to actually hear a bit more about the actual GPU nodes themselves. Yeah, we can. So, the GPU nodes are HPE servers, particularly at DL380s, and there's a dual CPU in them, and there's a 6132 Xeon server with 14 cores, 384 GB of RAM, and the two Tesla cards, and we have also the dual 100 gigabit Ethernet link. So, just to mention, so these servers have different affinity with each CPU and GPU. So, the first GPU has an affinity with the first CPU and the second GPU to the second CPU. So, these two have, in theory, we are running two GPU VMs in these servers plus other VMs that are not that they don't have a GPU in it. So, one of the VMs is running from the first CPU and the second one, the second GPU. So, to achieve so, as the OpenStack documentation, we use the CPU isolations for the CPUs that we want to run with Nova and with the GPUs. So, essentially, we enable IOMMU in Grab, and we use this parameter called ISO CPUs to isolate the CPUs we want to use for Nova. So, in Nova.conf, we use the CPU pinset parameter with the CPUs that we just mentioned above, and we enable the filters with the Noma topology filter for having the scheduler scheduled in the correct flavor to the instance. So, we also have IPer threading disables because following the advice of some colleagues from other centers for our workloads, it is better to keep IPer threading disabled to maximize performance on certain kind of workloads. So, Jeremy. So, yeah, the other part of that obviously is the PCI pass through for the GPUs themselves. The documentation that OpenStack provides is actually pretty good. So, very basically, we just run an LSPCI so that we can identify the vendor and product IDs for the two GPU cards, and then in Nova.conf on the controller side, we create an alias, in this case, we've called it V100, where we identify those particular cards. Of course, we have to use the PCI pass through filter for the scheduler as well. And then, on the compute side, we set up a white list for that vendor and product ID as well, so that the physical compute node itself knows to pass that through. Then we create the flavor itself. We just keep it to seven cores for the flavor. As Luca mentioned, these GPU nodes, we still use some of the resources for just running regular instances on them as well. So, for the actual GPU specific flavors, that's usually sufficient. We give it 90 gig of RAM. Again, the RAM is allocated, basically split between the two CPUs, so we have to use the new node adjacent RAM for each of those flavors, and 40 gig of disk for the root file system under CEP. In the flavor configuration, we just have three particular properties that are set, which are the, as you can see there, for aggregate instance extra specs, which is actually another filter that I forgot to mention, which should be on there. But anyway, so that set pin equals true. CPU policy is set to dedicated, and the PCI pass through, we refer to that alias of V100 that we defined earlier in nova.conf with just the one allocated to the PCI pass through. And then for the... When we create the host aggregate, we just make sure the pin equals true is set as a property for that host aggregate, so it knows that the flavor can run within that host aggregate. As a slight aside, if you wanted to fire up an instance that used both GPUs at once, there's some additional documentation under for OpenStack for CPU topologies that sheds a little light on that. The main difference, as you can see with that OpenStack flavor create statement, is you've got that PCI pass through alias, we're still using the V100 alias, but now we're telling it to use two instead of one, and the other line is highlighted in red, which is the human nodes equals two. So note, of course, because we've given it double the cores and double the RAM, that's so that it can pull the same amount of cores and RAM that would have, for one instance, from each side of the compute node itself. So I don't know if you mentioned it, but so this would like to be an open discussion afterwards to see and if you can comment with us our results. So going a bit more in details, so as you can see, these are the NUMA nodes of our compute node. So we allocated the first instance to the NUMA node zero and the second instance to the NUMA node three. We used just as much as memory as we could have used in the same NUMA node to decrease latency and having the VM perform as better as possible. And the GPU affinity, so down there you can see the GPU affinity and you can retrieve that with NVIDIA SMI topology. So the first GPU is, so GPU zero is got an affinity with the first NUMA node only and the second GPU with the third NUMA node. So you can see here an instance running with a vCPU pinning and you can see that the CPU affinity is only on the cores from zero to six. If you have another instance running without vCPU pinning, you would see the little y on all the cores, that means that the instance virtual CPUs can use any of the cores available on the hypervisor, but not in this case. So in our test we tried to compare our virtual machine with the GPU passthrough to a bare metal node. So obviously the bare metal node has multiple CPU and GPUs available. So the performance of the bare metal node had to be tuned down to be comparable to our instance flavor and GPU instance flavor. So as already mentioned, the VM is configured to use a single NUMA node and so we tried to reproduce that on the bare metal node that was exactly the same hypervisor we run the VM from, but with a directly provision bare metal. So to do so we removed the GPUs from the PCI using that command there and we removed also all the cores available except the first six on the first NUMA node. So the only difference we couldn't remove was the fact that the local storage was an SSD in the bare metal machine and with the virtual machine we still had a 40 gig volume on self and we have quite an old cluster and so for sure the 40 gig volume of self doesn't have the same IOPS as the local storage on SSD. That's an old SSD anyway. So the first benchmark we run is the high performance LIMPAC. LIMPAC is a well-known software library used for a floating point benchmark and as you can see the bare metal is on an average 4% faster than the virtual machine with around 5500 gigaflops against 5200 gigaflops. So LIMPAC is very GPU intensive but it's not ultra CPU intensive so I'll show you later why I'm mentioning this but so the result for this slide seems good results but in the other slides we had mixed level results. So this is a TensorFlow benchmark for I think everyone knows about TensorFlow but TensorFlow is an open source software library for high performance numerical computations and this particular benchmark uses the ResNet 50 image classification model and we see that the bare metal machine is only 0.6% faster than the VM and it seems quite a good result so you can see on the bottom the number of runs and it's pretty the performance are very similar to the bare metal. So the last benchmark we run is NUMD so we run this one only six times because we actually finished to do these results like I think it was the day before we arrived in Berlin. So essentially the bare metal node in this case is way faster than the VM and the difference in this test is the CPU is much more utilized than the GPU and this NUMD test is essentially a parallel molecular dynamic code for high performance simulation of large molecular dynamics and we've seen after many runs that consistently the VM is slower than the bare metal node. We still don't understand and actually we want to open the open discussion video. We still think that probably there's some issues with our novel configuration or the CPU pinning in which the CPU is not used as well as we think because yeah the CPU as soon as the CPU is loaded so it's very loaded the performance decrease compared to bare metal. So for the people that wants to stay we would like to have a small open discussion for people that had that already have a GPU deployment or even for people that don't have a GPU deployment and would like to have some and these results for the moment are good for us but just because we have a small cluster but recently we had a new capital refresh so we have to expand our cluster and an 11% difference in performance is not acceptable and so we would like to tune our configuration to have better results. So if you have some comments already I'm able to take some some questions or discussion. So these are already the the known issues that so each of the nicks obviously for how the PCI razors inside the machines are configured have a different affinity with a different CPU and though they are coupled they are bonded with LACP and so the two configuration creates issues with the virtual CPU pinning that we did before and so this TV the the instances cannot access directly both of the nicks they have to pass through the other numeral than the other CPU and so this increase latency and slow down performance though we don't think is this the reasons for the numD results. There are possible solutions for this and this is already some work that other centers for other computing centers are already trying to address and some guys already tried to pin half of the CPUs for the CPU 1 and 0 and half of the CPUs for the CPU 2 and 03 to have both GPUs and network be able to be accessed by the the correct NUMA memory but also we could also think about removing LACP and just have another network configuration but as you know the network guys will not be happy very happy to change all the network configuration at the center because we want to perform a bit better. And as going ahead and wishlist we really would like to test Nvidia Creed and VGPUs for Ubuntu hopefully the packages and the binaries already are available for Redat and Libert but they're not still available for Ubuntu. I asked actually before the Ubuntu guys if there was any news about it and they didn't know anything they told me that the kernel already supports the VGPUs from 4.10 or 4.12 but with no binaries I don't know if this is going to be possible and once that is done we really would like to try the VGPU support on Queens and Rocky and after that we want to do testing on RDMA on GPU to GPU through the network so if you want do you want to add anything? No that's pretty much it. Cool so well that's pretty much it for our presentation but did anyone have any questions? Yes my question is when you look at purely the performance of GPU bare metal versus in the virtual machine I saw the benchmarks that you presented were always CPU and GPU combined is that correct? Yeah except for TensorFlow it's mostly GPU. Right so did you see a difference there as well? Actually when it's only GPU bound purely GPU bound there's a minimal difference this even less than zero around 0.0 percent. Okay so it's mainly the CPU? Yes it's mainly the CPU yeah but we're still not sure if it's because the CPU with the NUMA node configuration with the VGPU pinning configuration and the NUMA how the GPU and CPU are joined together with the noises for that the performance decrease because 11 percent is quite a substantial difference. We don't have this difference on our CPU running without the GPU. Thanks so I'm kind of looking around the room for I'm hoping that there's somebody here who knows this stuff way better than I do but I don't know whether you kind of glossed over it or whether it's actually absent but there were a few pinning configuration options that I noticed that a conspicuous absence of. What are you talking about? You config like and again I don't know them off the top of my head so well but there's like it is a way to specifically define the CPUs and memories in the same NUMA. So you see that one is the first NUMA node node zero and the NUMA node has the first six cores from zero to six or seven cores and the memory adjacent memory. Right I get it but yeah sorry here I think it's what you were talking about so yeah this is from zero so this is the instance that is pinned to that first NUMA node and the cores are pinned and they stay within the NUMA node so if you see where the Y is it's a bit of an unreadable graph but VERSHA gives it to you this way. You can see that from cores zero to six is pinned so that is the instance information and these are the cores where the core lies on the physical core. Yeah so that's why I assumed that I had just you know you just hadn't shown whatever the configuration option because this looks like it's working the way you want it to in which case my hypothesis here is that you're fighting against the overhead from the you're not getting exclusive access to these CPUs when you're running virtual. Yeah we thought so as well but so we have all passed through already. The hypervisor we did the test was completely empty. Right but you're still fighting against the compute process. That was actually my question is limiting the overhead to not be on the vCPUs. I tried to do it but it didn't actually work so I don't know if that's something you've looked into as far as overhead. Though I mean the overhead 11% is a bit too high I mean it's not cannot be considered as a overhead I mean if you were thinking the first test around 4% it could probably be but 11% is not so and we still don't understand where the the issue is coming from. Have you used huge pages at all or did you set that at all in your? We still didn't we had on the other cluster but these are really early results that we published that we achieved right before coming to the summit. It can improve huge pages but I don't think it's going to improve so much the performance but yeah yeah thank you. So a couple of things just brainstorming with my colleagues over here if there was a way for you to you should be able to pin the hypervisor processes to CPUs that aren't these CPUs is one thing that could get if you are in fact fighting with the hypervisor process. Oh did you already do that? Yeah we already did that so the eyes of CPU parameters so essentially when you put that the that flag in the in the in grab means that from so all the CPUs listed are they don't have to be used by the operating system so they are completely isolated and just Nova can use them so right it's not hypervisor noise as well. Did you shut down the compute process before you started your benchmarks? Did Nova compute? No we didn't. You could do that as long as you don't need to go deploy more VMs. No no yeah that's that's a good idea that's a good idea. Thank you. Yeah that's it I was talking about I don't think that actually works I think what don't think it actually isolates them like it's supposed to so I don't know if that's a kernel version or something. Okay thank you. Does anyone have any other question or anything you want to talk about about this topic? It would be great to to know. If not thank you very much. Thank you.


We're going to see increasing diversification in terms of our cloud providers. When we look at where our customers endpoints are, if you think about all the applications that are on Akama, when we kind of do a study of where those endpoints are, we see over time that they're an increasing number of cloud providers, an increasing number of regions. They're split up, an increasing number of different endpoints, obviously microservice types of patterns. When you factor in growth of edge, then it's really not a surprise that we're going to see more providers come into the mix.


For years, this area around the Kajaki hydroelectric dam in Helmand Province has been a stronghold of Taliban-led insurgents. Citizens have been forced from their homes by fighting. The potential of the dam to power some of the most crucial areas of Helmand had lain dormant in disrepair. In October, during Operation Eastern Storm, 1st Battalion 6 Marines and Afghan National Army soldiers swept north up Route 611 from Sangin, pushing all the way to the Kajaki District Center in the dam. What this dam signifies today, it signifies the fact that the Afghan government is taking full control of the dam and its facilities and will work to provide better services for the people of Afghanistan. Reporting from Camp Leatherneck, Helmand Province, I'm Staff Sergeant Jeremy Ross.


So, I think in the last developer diary, I probably got to the point where I had styled things up. Yes, that's where I'd got to. And since then, it's been a lot of plumbing work, if you like, just to do things like offline. But I will come back to that in probably the next entry. In this entry, what I want to talk about is a relatively straightforward thing, which is lazy loading of images. So I say straightforward. I mean straightforward in the sense that it's not an out there concept. I think many of us kind of realize that we probably want to defer loading of some stuff. And many of us would probably do it on something like a scroll basis and possibly even have a scroll handler to do that kind of stuff. Well, well, I've done it. But I've done it without scroll handling. I've used an intersection observer. So I thought I'd show you that little bit of code today. And you could just have a look. I mean, I said that. I said you could have a look. And then I just said it twice.  Anyway, let's have a look at what it is. If you look at what's on my screen, you'll see that I've got the site up and I've got the network tab open and it's recording. So that's the network you bit there and it's recording. And when I scroll down here, ready, steady, these two poster images here for these two episodes, the lazy load on the fly based on the scroll position where I say the scroll position based on intersecting with the viewport. So let's have a quick look at what the code looks like for that.  And I do a quick check to see where the intersection observer is supported because you'll see what happens here is I basically early on I look for anything which has this JS lazy image class on it. And if it does, then I set things up to go. But if we don't have support for an intersection observer, I just kind of fall back to what I think most of us would do today, which is just go, well, look, I can't do an intersection observer. I'm just going to load everything. And that's okay. I mean, this is a progressive enhancement move. This is something we can use to improve our experiences.  What I'm actually going to do is I'm going to fall back to a scroll handler based one. You might choose to do that. I would suggest typically you want to avoid anything that's coupled to the scroll handling just because it's typically the case that a scroll is an animation and it's very frame rate sensitive. And so you probably want to be careful about doing something that has to fire on a scroll handle a scroll handling. Anyway, done with standing. I've decided that if we don't have an intersection observer, we're not doing anything. But what do we do when we've got an intersection observer available to us? Well, we create one of these and we have it observe all of the images which have the JS lazy image class on it. So if they don't have that class on, we're not bothered. If we do, then we observe them. And what do we observe? Well, we have a threshold, which is how much they have to intersect. By default, it's going to how much does it intersect with the viewport? And I have a 50 pixel margin. So when you imagine the viewport and then add 50 pixels either side, that's the that's the that's the area of interest if you like. And I'm asking, when does an image intersect? Let me know if an image intersects that that viewport plus 50 pixels on either side. And there's a threshold that I need to know. So they what it does is, if you imagine the image being completely inside the viewport, that would have a value of one. If it's completely outside, it's zero. And then as it passes, it's going to go between zero and one. And so there's a threshold. As soon as you pass that threshold, I need to know about the intersection. And that's that's a hard way to hard coded value, which is not point not one. So it's it's one percent, I guess, of overlap between those two areas. So what happens when we do actually see that? Well, you can see I have this dot on intersection, as you can see here, which is called. And we get because it's an observer, we get an array of entries because it's possible that with an observer, you're going to get a backlog of entries because things could go in and out. And by the time you actually get to them, you need to, you know, you need to maybe sort them by date. Actually, I don't know if they already sorted. I should figure that out. In any case, with something like this, it's it's not too bad. I just need to know whether or not they've actually intersected with the viewport. And if they have, I'm fine with just going ahead, you might need to do something a bit more nuanced than your intersection observer callback, where you say, Well, actually, you know, is it, you know, is it the most recent record and all these kind of things? But you can you can see, hopefully that you could do some further work on those records. Right. If I decide or if I find that in my entry that the intersection ratio is less than zero, as in it's it's no longer intersecting or whatever. Don't worry about it. Forget about it. Don't need to know about it. If I have already tagged the image that is now intersecting, if I've tagged it with my handled class, so that's what I do. I tag something. So if something goes in out and then back in again, I don't need to know about it the second time. So the first time it comes in and the first time I see an image intersecting with my viewport, I tag it with this class and say, Right, you're done. And if I see that that handled class is on, then I return. And then the last thing I actually do is preload the image. So how can I put this? We what I do is I actually set a background image and I fade it in. That's just the way I'm choosing to do this particular content. It means that I can have like a placeholder with a little icon. So if it takes a long while to load the image, then we sort of we can just put something up on screen and then fade it in. So I preload the image here, which I have a thing that goes off, makes a new image, sets the source and then does an onload and then resolves the promise. Which you can find inside the utils. In fact, let me show you it since we're here preload images. Then we are making a new image, set the source and then I just use resolve and reject as the handlers for onload and on error. So when the image is actually being preloaded, we apply it. So we find the element that we were talking about, which is, you know, image and find the actual sub element, a child of that, which is as the class of just lazy image content.  If I can't find it, don't worry about it. If I can, I set the background image and then I add this fade in, which is an animation that fades the opacity from zero to one. So the idea of preloading the image is to make sure that because it's a background image, here's the thing. Because it's a background image, there's no guarantee that if I just set the background image and then started fading in that the image wouldn't be sort of part loaded during the fading. You kind of get that bizarre, it's fading, but it's also loading at the same time feel. So I preload it, which is make sure it's definitely in the cache. And we talked previously about my caching headers. So the image should come with good caching headers. Preload it, get it, make sure it's there. And then we can set background image. So when we set the background image, it goes, oh, I better go and get that image. And he goes, oh, it's already there in the cache. Let me just use that pop. And it means that we don't get that partly downloaded. Kind of feel with that images. And that's it. So using intersection observer is a nice progressive enhancement move. I'm enjoying it. It does the job perfectly. If we don't, if we have a long page with a lot of content in and somebody doesn't scroll down, why bother loading those images? No need. So it's not a lot of code, is it? 115 or so lines of code just to handle that? Winning. So there you go. Next time, I'm going to have a chat with you about what I've been doing in the world of offline and prefetching video. And oh my goodness, has that been a journey? Anyway, don't forget you can subscribe. Don't forget you can drop your comments and your questions in below and I will catch you next time. Hello. Thanks for watching. If you enjoyed this video, well, you may enjoy other videos that we make too. So don't forget you can subscribe and you'll get notified when we push out a new video. And there's more videos over there or down there depending on how you're watching the YouTube right now. Definitely click on those.