And here we are in lecture 2, we are going to look at printing, now printing as far as iPython notebook is concerned means printing to the screen. Think of it this way, you use a word processing program and you say file print and instead of printing to a physical printer, depending on your operating system and how it's set up you can print to PDF. The same happens here, and Sympi has various printers, instead of printing an answer to my code I can have it print LaTeC code which I then can then use somewhere else. So there's a variety of printers available in Sympi, here I'm going to do as you can see, this line here is going to indicate to this iPython notebook that I want to do what is referred to as pretty printing. So I'm going to say import Sympi as SYM, just an abbreviation for this namespace and then I'm going to invoke this pretty printing. I'm going to tell this notebook that I want SYM, referring to sympi.init underscore printing open and close parenthesis. Then I'm going to set up my symbols x, y, z, r, theta, now look at that theta, there's a variety or long list of words that I can use in Sympi that will export the symbol for that word. Now remember the variable here, the first one maps to the first symbol there, the second variable maps to the second symbol there, note the commas here, note no commas there, and I have theta here and theta there. Now I'm just simply going to print to the screen theta cubed, let's see what that looks like. Beautiful as if you've written it on paper in beautiful handwriting, that really is pretty printing. So remember we can factorize, I have 2x squared plus xy, it's going to factorize by taking x out as a common factor, leaving me with x and then 2x plus y and I can print that out. So because I've invoked this pretty printing here, everything that gets printed to the screen now will be done with pretty printing and indeed I have the x and 2x plus y. Let's look at integration. I'm going to print the string, the indefinite integral dot and I have some dot integral, now we're going to get to differentiation and integration, but look here I've got integral with a capital I and I'm going to say x cubed comma x, that means the indefinite integral of x cubed with respect to x, so I just have my expression there comma with respect to what I want to integrate and if I were to run that look at that, is that not a thing of beauty, the integral of x cubed with respect to x, I can also do the definite integral, I'm going to do a and b, a symbols a and b and I'm going to say some dot integral, again my expression there x cubed comma, now I have to put this part in parentheses, still with respect to x and I have my lower bound and then upper bound, so if I were to print that look at that, the definite integral of x cubed with respect to x, going from a to b. Now there is a keyword just integrate that we can use here that will actually do the integration for me, but I can also use this formal integral and just put the dot do it with open and closed parentheses at the back and that will actually just solve that integral for me, this is not the usual way of doing it, but I can have the printing and then just say dot do it and it will just do the integral for me, let's look at the derivative, some dot derivative with a capital D there, it's y cubed with respect to y, so this comma y and lo and behold look at that, I needn't just take the first derivative though, what if I wanted the third derivative, so here I have x plus y cubed with respect to y three times, I'm just going to have y y and then just hit enter and lo and behold look at this, so I have a partial derivative because it's with respect to y but I have another symbol in the x, so that is a third partial derivative with respect to y of x plus y cubed, now in this last section of code here I'm again going to do the derivative of y cubed and now I needn't put y, y, y, I could also have just said y, let's just do that three, if I do that it is going to still do the third derivative, so I can also do it here but I can also do the dot do it, so it's just going to do the second derivative of y cubed with respect to y for me and just give me the answer, now just getting back I just wanted to show you alpha comma alpha comma beta comma beta and let's map that to symbols and remember parentheses and my quotation marks alpha alpha lower uppercase and lowercase beta and beta, so we can do that and now if I were to print alpha I have the remember if I'm in a cell and I want to execute that cell I can either press the run button there or I can hit shift enter and lo and behold there's a capital alpha and lowercase alpha there's my lowercase alpha uppercase beta and my lowercase beta done all done now through pretty printing and that is mightily impressive indeed


Uh-oh, we just got a green beast the other way. Oh Might get my full stop out No, oh, I just got enigma'd Well the wrong side of that. All right. I'm closing it Go ahead and taking that small loss Not gonna turn into a big one. Okay. Y'all study what I just did right there Okay, remember when it comes back in your direction, dude, if I'm wrong on it I do not want to mess with that I'm not trying to turn a loser into a winner like if you can take that principle and learn that from me I'm telling you it will be a game changer for you. It's a knot You will take less losses in general. Well, actually you'll take more losses, but you will have less loss overall If it if it digs on you listen a play is either going to absolutely play out or it's not and Why not have the ones that aren't just close it out quickly. All right, so let's let's show this with just my stats Let's do this with just side work stats. We'll be able to compare them side by side This looks like it's two zoomed in for some reason Is that easy level I think 80 is a sweet spot very speedy Okay, when you play hard you work hard don't play at all tell you Roosevelt. I like that I'll go box to the world Let's see this change this filter it just for me. All right, so where am I at for the day? I'm at 81% Profit factor 2.63 remember how I told you guys like 2.7 is that sweet spot you guys have heard me say this right? Now this is a little high on the percent profitable. So I mind you. This is me going on enigma. Okay? Enigma is better. I just I don't know if it's better. It's uh, I mean it's I have to have less thought process involved in the decision-making So my percent profit was probably gonna go up but hopefully my profit factor is somewhere in neighborhood of 2.7 That's what that's my personal endeavors. Is it possible? Yeah, absolutely It's that's what we can do but that means small stops big winners Everybody understand that like you've got if you do if you eliminate the small stops I had a conversation with somebody yesterday. He's new with us We talked on the phone and he's I can already tell you know, he's struggling with it We talked about it Struggling with he's moving the stops.  He thought you could skip through on the videos So I encourage you guys listen I have the videos there for a reason you do need to go through the videos Okay, if you don't you are going to miss out on the little keys in there Like if I telling you like if a play I have an advanced stop strategy If I've got some zone behind the stop area where I can quote play back behind it I'm not trying to turn a loser into a winner.   That's your oh shoot. I'll totally got it wrong Okay, worst case scenario stop it comes in on your ATM, but you should be using that close button Just like I did just a minute ago. Okay, and sure that one that could have turned into a winner That one right down here. Okay, I got sure it would have popped down there And yes, I could have taken out that winner But that doesn't happen all the time right and the majority of the time it comes right back and then zooms on you Almost comes back to your point of entry and then you get wrecked, right? That's what you don't want to do turn a loser into a smaller loser by pressing the close Okay, make that close button something you use often you should be using that close button I don't say as often as you're using the buy sale, but pretty darn close like it should be a two to one ratio Maybe of going ahead and having to close out that position manually by


Okay, can everybody hear me okay? Perfect, so sorry for all that you have to stare at a mask, but it is what it is. For those of you that don't know me, my name is David. I'm with Red Hat and today I'm gonna talk about page table reclaim. I'm not necessarily gonna explain like a real proposal that I have, but because it's complicated matter. Like I wanna raise awareness and maybe we can have a discussion of what might work, what might not work and what the obstacles are of the whole thing. So first of all, like I'm gonna discuss like what we're dealing with and where it especially applies. Then I'm gonna share some fun with page tables, meaning like how you can easily have processes allocate a whole bunch of page tables even in setups where memory over commit is supposed to be disabled and there should not be an out of memory killer. Yeah, sapping your processes. Then we can have a discussion. I have some points on the slide, for example, what do we actually want to reclaim in which scenarios? When do we want to reclaim? How to reclaim and of course how to, we can actually win against malicious processes that we want to hurt ourselves. Kill it. Kill it with fire. Right, let's talk about reclaimable page table first. Just a quick recap, I assume most of you guys here know what I'm talking about. For user space processes, we have these nice page tables. The layout and how they behave are various depending on architectures or configurations. So it depends how many levels we can have if we support gigantic huge pages, if we have transparent huge page support. But of course also what the page table layout is. So for example, if we have the same layout or how a page table looks like on each level, if they differ. And of course we have different base page sizes. On top of that, we have various different corner cases like continuous PD, huge pages, which is weird. PD map, THP and a lot of more weird stuff that over complicates things like page directories for huge pages, which is not a page table, but something in between. But for this talk, I'll be focusing on I guess the x86 layout, which is a base page size of four kilobytes, a page table size of four kilobytes and a page table entry size of eight bytes, meaning that like for each page table we have in our hierarchy, we essentially have 512 page table entries. This makes calculations a bit easier. Now, the issue with page tables is that they are unswappable and unmovable. Everybody who knows me, I'm dealing a lot with memory hot on block, knows that I hate unmovable and unswappable memory because it makes things very weird when you want to hot on block memory. And it's charged like whenever we allocate a user space page table, it's charged as a column allocation. And in the C group V1 version, we actually had a mechanism to limit the amount of column memory that a C group could allocate, which was nicely described here as a way to denial of servers to your system. In C group V2, it does no longer exist as far as I understood. I'm not saying that's a bad thing because usually if you want to set a limit on the number of kernel allocations, the process can have, like you can only come up with matching numbers, so that's no good. So maybe we should much rather try to find out, for example, why we have many page tables and reclaim them instead of trying to use the hammer on the nail here and just say, yeah, we're gonna limit the amount of memory, column memory process can consume or C group can consume. If we take a look at the page table overhead, that means like if you have an M-app in user space, a VMA, how much overhead you actually have with page tables, then in the scenario that I gave, you usually need like for each page you map, you need eight bytes of page table entry. So that's roughly 0.2% of your M-app size. When you take a look at the PMDs, it's even less and once we are at the PUD level, it gets fairly neglectable. But if we take a look at, for example, a single two terabyte mapping, we can have four gigabytes of PD tables, which is a lot. Eight megabytes of PMD tables, which can still be problematic, especially assuming we have even bigger mappings and user space tries to do nasty stuff with that, but on the PUD level, like usually we don't care anymore. And one interesting thing is when we're talking about transparent huge pages, anonymous transparent huge pages, that we always allocate the PTE table as well because we have to, like all the random context, we have to be able to insert a PTE table to essentially split the P and D mapping, which means like you cannot, for example, get rid of the 0.2% just by using transparent huge pages and being lucky that doesn't work that way. So we already have page table reclaimed to some degree in upstream Linux, which is whenever we change our VMA layout, for example, we M-unmap or we M-app with map fixed or something like that, we remove all now empty page tables because they're outside of our VMAs. So we can remove quite a lot of page tables by doing expensive VMA changes, but there are use cases that really don't want to do that, which are essentially sparse memory mappings, meaning you have like a very large VMA space and you will want to dynamically, for example, allocate and deallocate memory and avoid, for example, running out of VMAs, stuff like that. And there are benign user space applications that make use of that, for example, memory ballooning in friends, like with a balloon, with a mem, when exposing memory to the virtual machines and hyperbysy would have like a very large sparse mapping, essentially, the memory allocators that do the same, which really want to avoid doing expensive VMA changes. And I learned that there are fancy metrics algorithms that rely on sparsity in a layup. That's why you have like a very large sparse and map area. And of course, if we consider like a large file that you map into memory, not all pages might be mapped into the page table at the single point in time because they might just be written out to storage and evicted, in that case, there is also like essentially a sparse memory mapping at that point. But unfortunately also malicious user space can make use of that and there are ways to trigger a location of a lot of page tables for a process and you can actually calculate how much harm can be done just by taking a look at what the maximum VMA space is that a single process can consume. And essentially what we can do then is we can break memory over commit, we can break swapping, we can break compaction, CMA is so movable, so it's all a big mess, I would say, if we try to poke it with a stick. And essentially for sparse memory mapping, there are two cases I think that can happen. On one hand, we can have a lot of empty page tables that we can just remove. For example, that can happen with mAdvice, where mAdvice don't need semantics, fLocate with punch hole, or just ordinary reclaim, for example, of file mapping. But there are also other cases of page tables that we might be able to reclaim because essentially they're reconstructable and reconstructable here means if we throw away the page table, we can just recreate the content because we know what we're doing. For example, on certain mappings, we might map the shared zero page on read access and we can do the same again if we were to throw away the page table. And the same is, I assume, with page cache pages because we can just refold them from the page cache and we're fine. So even though we are trying to look at primarily empty page tables and reclaiming them, I want to raise awareness that there is actually much more harm that can be done, for example, by the shared zero page. So what we have upstream right now is a proposal by a USN as a way to reclaim at least the PTE tables, meaning the lowest page tables. As soon as there are no entries left anymore and it works essentially by having an additional counter inside of the struct page and that counter on the one hand keeps track of entries inside of the page table that are not empty and of current page table walkers. It means like as soon as there are no page table walkers anymore, so nobody references it and there are no entries anymore, we can essentially remove the empty page table. Like if you have, I call it unmap or a SAP. So if you would do an M advice, don't need on a single page, it's the last remaining page. We can actually remove them page tables across VMAs or even page tables that don't even lie inside of VMAs anymore. Now we come to the fun part with shared zero page. Essentially, I have a couple of examples. So this is the simplest example how you can trigger a location of a lot of empty page tables. So you have a very large mapping. So I start with 100 rate exabyte, but usually on my laptop it's 64 terabytes. It varies I think on a processor support. You map it readable and writable because we want to write to it and because it's so big and we might have setups that disable memory overcommit, we might have to pass the map no reserve flag here. And then we can create on certainly like on most systems, but not all we can actually create such a large mapping and then what we're just gonna do is we're gonna write to one page inside of a two megabyte fragment. That means like we populate a PMD page and exactly like one page table, PTE table, and we just map like a single entry in there, which means that we populate essentially with this algorithm where we first write to one of these pages and then remove it immediately. Again, we allocate one PTE table and above hierarchy and you can consume quite a lot of memory with that. But the thing is like if you have a system where memory overcommit is completely disabled and even map no reserve won't help you. So in that case, you cannot allocate, for example, on that notebook 64 terabytes. It's not gonna work with memory overcommit disabled. But that's where the fun begins because what about we do a read mapping? Read them on mappings, kernel allows for that. We can have as much as we want even if memory overcommit is disabled because what can happen? What could possibly go wrong? We don't need a map no reserve anymore. It will just work. What we can do then is we just like read a single page and what a kernel would do is it will populate the shared zero page. So we have a lot of page tables just filled with the shared zero page. And yeah, this will work quite fine. We can consume a lot of page tables that way. And we could argue like these are reclaimable page tables because we can just throw away the page table on the map shared zero page. And whenever there would be an access again we would simply refold the shared zero page. But there's also an example I think that's valuable which are we unreclaimable page tables. It's essentially the same mechanism. There is a lot of code like we have userfoldfd involved. And usually when you hear userfoldfd you have to be careful because it relies on for example stuff that has already been folded into a page table to not trigger fold again. And what we do here is essentially even more fun because we have a broad non-mapping. So we don't even need read semantics anymore. We can add as much as we want of that 64 terabytes on my notebook works. And then we're just gonna like place a zero page into that broad non-mapping using userfoldfd and we can trigger the exact same thing. So I have a simple example here that throttles star locations a bit. So up here we can see like we have right now 28 megabytes of page tables. And if I for example, trigger here the reclaimable page tables part you can see that it goes crazy fairly easily like we're hitting 200 megabytes and I'm throttling the locations right now. But on the other hand, you can see that this process it has like 65 terabytes of virtual memory and it essentially consumes no other memory. So I'm like just looking at that. It looks a little bit weird. We already had one gigabyte I'm gonna stop because otherwise I'm gonna run into issues here. But you can see that it's fairly easy to trigger this. But like when I realized which kind of magic you can do with that it gets even more fun because like does anybody here know an interface that you can use to inspect if something has a mapping or not? Like in user space to observe if there's something mapped or there's nothing mapped and there are certain virtual address. So it's broke page map if that clarifies. So with the page map you can identify like it's at that virtual memory a shared zero page map or not. And you can control if there's a shared zero page map or not. So what you can essentially do is you can write a memory allocator that stores information in the page tables. That means like you're gonna store one bit per PTE which is an overhead of one bit for eight bytes which is bad. But you can do it and you can write in this example. So this is just the example I have. You can allocate for example two megabytes what it will do under the hood it will reserve like an area that is two megabytes time the size of a base page. Then you're just gonna poke it with a stick and you can actually store and retrieve data in there. And it's like a proton on mapping. That's I think it's the weirdest part you can store memory in there. What I find like a little bit concerning is that you could hide stuff on the calendar. Like you could hide something from an antivirus scanner. You could hide secrets. I think this is even a better way to store secrets than for example using mlock or something like that because there's no way somebody else can like understand what's going on here unless they're able to inspect the page tables using the page map. So yeah, this is fun I would say. No, obviously the question is what can we do about that? And I mean I started to write up some stuff. I hope that maybe somebody here has an idea of what we can do. But essentially a question we have to ask ourselves is what do we actually want to reclaim? And the thing here is on the one hand we have empty page tables. That's the low hanging fruit I would say for example, using that approach that we have upstream right now. Reclaimable page tables are a little bit more involved. For example when we map to shared zero page and then we have unreclaimable page tables. And here essentially the problem is for example you have a page table that maps to shared zero page but you have user folder D enabled. How would you know that you can actually throw away that page table including that information that's valuable for user folder D? Same applies for soft dirty tracking for example where we store additional information inside of the page table. For example that certain page has been soft dirty. So we have to be very careful with that. Other question like we might want to ask is like do we only want to concentrate of leaf page tables meaning PTE tables as we had in that proposed upstream or do we care about higher level page tables? And I think like we really at least care about P and D tables as my calculation showed but maybe we just want to do it like completely then if we already have to tackle two levels and like reclaim all of them where possible. The other thing is if we would want to care about hot page tables meaning for example if you have a page that page table that is reclaimable meaning it maps something right now like would you want to throw that away and essentially for example refold stuff from the page cache or would you want to have some kind of awareness like which page table has actually recently been accessed or not. We don't have something like I'll allow you for that but certainly it would be interesting. But I guess this would then be one thing to look into once we actually realized that we care about such scenarios and essentially when we figure out when we want to reclaim. Which brings us to the next question like when do we want to reclaim? And I mean the easiest thing is when a page table is empty as upstream showed because you can have some smart logic and then just like detect page tables empty using for example that reference counter and remove the page table. But what about the shared zero page as we seen like how would you want to do that? You would actually have to scan the page tables somehow scanning a process page tables is expensive might need a MF block in read mode. So one idea I had is that maybe you would want to start reclaiming as soon as there is like a suspicious amount of zero page usage. The question is then what is the suspicious usage of the shared zero page? For example in virtual machines we often have like for example cases where we have a lot of shared zero page mapped which is a benign use case and not a malicious one but actually like I have no idea how you would want to tackle this except when special case for example the shared zero page. Other things would be like we would want to reclaim on magic conditions meaning we saw in my notebook like it was consuming like one gigabyte of page tables and it had essentially no resident memory and nothing in there. So maybe that could be an indication that you at least would want to give it a shot to reclaim and once you find out that you actually weren't able to reclaim that much you might just flag the process as doing something special for example that's really a virtual machine that just results in these weird scenarios but I guess it's also like more a way of finding heuristics you could apply. And last but not least one idea I had is like instead of scanning for example processes stuff like that what if your page tables were actually either movable or migratable or at least to some degree you would detect out of memory migration context or memory compaction context that the page you're looking at right now is essentially either an empty page table or it just maps to shared zero page and then remove the page table from that PFN walker essentially so I think there are two approaches like on the one hand we would have to scan the process page tables to find that there are shared series pages. The other hand we could scan the PFNs for example during memory compaction just like once we stumble over an empty page table just remove it and as compaction for example might be triggered once we're low on memory that might be worthwhile to look into. Michael you have a question or a comment? Comment ideally. I think it's working. Okay it's working. Yeah so the first question I would have is would it make sense to kind of categorize those different page table pages those that are clearly reclaimable to even spread them somehow more to those that are disposable really easily and put them on all are you like the page cache and age them as a page cache. You do not have a reference bit for them but you can judge by how many page tables there are actually or how many entries there are in the in that page. I think even on x86 there is an access flag for page tables I think at least in the upper layer PTE so it would be harder to get access to. Yeah but essentially put those pages into all our use and use the general scanning algorithms that we have because that was something that we were playing with as an idea in the past. I don't think that we ever had a code for that but as it seems that for those cases when we are doing unmapping without really calling moon map that would be probably needed anyway. So yeah just put those pages on all our use and see whether that can be reclaimable. At least that would work for page cache and for page table pages which do not which are pointing to zero pages and cases like that. Have you considered that or play? I haven't considered that yet. I mean I mentioned here we don't have something like other you but I mean we could make it work eventually and then like it would make sense to only put like as you said the obviously reclaimable page tables in there and not anything else. And it's especially an issue like once you have VMAs that for example and like have user-fold FD enabled stuff like that it could work but for me one issue is still that like the one example I have with user-fold FD it relies on the fact that like you're not allowed to remove shared zero page and I wonder how you could make that work with user-fold FD like even if we fix the other stuff malicious users space could still go ahead and use user-fold FD to trigger the same thing. Then we might want to think about for example disabling the huge the placing a shared zero page for certain applications like to make it only available to privileged processes so to say. So I completely agree with the reclaimable part or reconstructable page tables to enlist them and to scan them. The interesting thing is then how to actually reclaim page tables because I mean that brings us to the next question how to reclaim. I mean for the proposal that we have upstream it makes use of the fact that each page table walker actually references the page table using the PD ref and your entries referenced that and if you want to remove the page table essentially you have to be the last one that drops the reference card and you remove it which is quite neat because you don't really need the MF block or something like that to make it work. Like you don't need any page table locks except the one of your parent I think but you can look up the parent using the struck page. So that could work the thing is as soon as you want to step something in a page table on your in context just of the page table from a PFN walker I think it could be quite challenging to for example notify and a new notifiers about that if you're like you don't really have the MF lock held or something like that. So that might be quite interesting I think to dig further in because here I also mentioned like scanning process page table needs to MF lock and you can like once you scan you can actually set entries for example shared zero page, page cache pages, whatever you want and you can properly synchronize with an MF new notifiers for example you can do proper MF flushes because you're in the context of the process but I guess maybe it's just similar to ordinary page reclaim where you're also like out of you have your arm at work and essentially your arm at blocks protect you from doing stuff so maybe that could work. The other idea is when we don't look at like the example of the shared zero page which is really nasty but just as like page tables that are mapping page, page cache pages so just the five mapping. Essentially when we are in the memory pressure we already start reclaiming and under reclaim we essentially produce empty page tables when doing it right. Like we un-map the page we set individual page table entries and if we could for example teach VM scan to eventually maybe also consider locality inside of a page table to some degree then you could maybe make VM scan produce empty page tables and the empty page tables could be removed using a different mechanism but of course this won't work for the shared zero page and there are some other corner cases I think once you have M-locked memory VM scan will not set memory in other processes just because like one process has a certain page M-locked but these are corner cases. Then again the other thing is during memory compaction or migration and then the big question about for me is how to reclaim from a PFN walker it's fairly complicated. Recently I took a look at page table migration which is completely insane but you essentially have the same problem like when you hot on black memory you might have to migrate memory away out of that PFN walk you would have to move to a page table which is also quite complicated and especially when it comes to locking and making sure that all page table walkers are properly excluded and synchronized. This brings us to the other point that removing a page table is actually I think the most difficult part and the proposed upstream made it quite nice with the PDE ref and synchronizing against all these. The question for me would be how you could actually make that work for example for PMD tables because it's no longer that easy and the further you move up in your hierarchy of page table for example if you would want to reclaim a POD table it could be that there are some page table walkers scanning over the page table and it is holding a reference for all eternity. I think one prime example is once you tear down a certainly big process it just takes a long time to tear down that whole mapping using the M-lock and it can I think take up to 30 minutes. So during these 30 minutes it could be that somebody is just holding a reference to this upper level higher level page table essentially stopping you blocking you from reclaiming the page table which would mean for me that like once we start reclaiming page tables that are higher in the hierarchy you would actually have to teach page table walkers to let go temporarily of page table references and look them up again because otherwise you might end up with scenarios where like you kind of reclaim a page table for 30 minutes simply because somebody else is holding the M-lock and just doing a big pass over all of the entries that are in there using or for example our page table walk infrastructure. Last but not least how to win against malicious user space I mean obviously if you start reclaiming out of VM scan context or something like that it could be if like user space just goes crazy populating the shared steel pages in my examples that if you do it in a naive way to reclaim once you run into memory pressure that like a malicious user space could still like do quite some harm to your overall machine I would say especially once you think about like running multiple containers and then you have somebody going crazy on unmovable memory it could still be problematic and one idea just I had was that like you throttling might be difficult but what you might want to think about is flagging a certain process as a candidate for example to reclaim page tables because it's doing something that's suspicious like consuming one gigabyte of page tables and once you detect that you could flag that browser as for example to really trigger reclaim before going back to user space and while that is stuck in that mode it would not be able for example to allocate another set of page tables but of course it gets more difficult if you have multiple processes involved to do that yeah yeah sorry I can repeat are there any known workloads like real workloads which have a high ratio of page table size to like physical memory size so the examples I have here is like for example memory ballooning what we have sometimes or even with word or memory you have for example two terabyte mapping and the guest was using memory part of that and then you reboot and essentially you remove all of the memory using them advice don't need and might not want to give it back but it could then happen for example if you have two terabytes of that that you actually have four gigabytes of page tables left so like it looks like oh my VM is not actually consuming memory via this mapping and it isn't but there's still so many page tables lying around but for that use case I mean that would be a benign use case you could say oh yeah we have to reclaim that memory somehow that's okay but the thing is that especially with memory ballooning you could also end up in a scenario where you have populated a shared zero page everywhere that is once your guest operating system inflated the balloon what it will do is which is hand back the memory and advice don't need it but especially older Linux systems in I think Windows what they will do is for example when you trigger a memory dump inside your guest operating system that they just don't care that like a page has been inflated or not there so they end up dumping all of the inflated memory which ends up resulting in read accesses to all of the VMA essentially so your populated shared zero page everywhere so which is also like a same use case in that sense although it's like really a niche corner case and again here the thing would be that like if you have running something in your virtual machine that's malicious it could affect the hypervisor just by reading for example balloon inflated memory so you could end up with somehow the same scenario but yeah like coming back to your question I think like if you have a large MF file that is written back to disk or like evicted and I think you can easily have such cases so the bigger your files assume the mapping the easier it is to get to that point yeah say something about the timing about when to reclaim and scan the table I think multi-generation NLU may be a good timing I think it does scan the table linearly so it may be a good timing to scan the table and find out the reclaimable tables assume we are going to merge the multi-generation NLU That's a good question so what you're saying is if you would apply multi-generational NLU that you could put the different levels of reclaimable into different categories No I mean multi-generation NLU does scan the process table linearly to find out the code page so I think you can, anyway you scan the tables you can find out the reclaimable tables at the same time I guess I mean that's true you could do that I was actually playing with like I was implementing just some naive way of doing that like I was scanning processes and I was sapping the shared zero page to see what I can do, what I can do essentially like as you say multi-generational I love you also, scans the page tables I think to extract access bit if I'm not wrong, under dirty bit what I ended up doing is I didn't hold the MAP log and read mode the whole time but I was scanning similar to what Gupfast does I was disabling interrupts I was scanning one page table if it was a candidate I would go ahead and like do the full MAP log MAP log write mode and would remove the page table but as soon as you have to use like the MAP mode and write mode you're essentially doomed against anybody who is malicious because they can just end up blocking you like and doing crazy kinds of allocations but yeah, I agree but anyway, I think we can find out the reclaimable tables and when to remove the table I think there maybe could be a delay to some time later yeah, the issue is oh yeah I'm gonna answer that and we're done with the talk so essentially we could defer that but even if you defer you always have to issue that like if you do it in a knife way removing page tables is a very expensive operation if you take a look at KUH page D it has to do like take the MAP log in write mode it has to take the RMAP logs in write mode and as soon as you have like a page table that's spanned by multiple VMAs you essentially like you're going crazy essentially because you have to take so many write logs so we have to come up with a better way for example to remove a page table the knife way using it via this PDF I think it works for the PDE tables I'm not so sure if it works for example for PMD tables because once you're essentially removing a PDE table because it was the last one it would like have to cascade but the cascading is problematic with the way it does locking with like taking the parent lock of the page table making sure that that entry can change stuff like that so I think that's gonna be the biggest issue like finding page tables to read them might be easy removing them might be hard but to conclude I think once we solve that problem to some degree we might be able to reuse it for KUH page D to remove a page table maybe more cleanly or more easily compared to this whole MAP lock and write mode dance and what so so that concludes my presentation thanks a lot for your time and we can talk in the hallway if there are any questions thanks a lot


So we're here with the Pudor and you have a new smartwatch. What's the product name on this one? H1. H1 is the design special for AirDrop people. And it's always on display? Yes, it is transverse on the screen. And what kind of function do you have here? Here, SOS. The AirDrop came to send SOS message to their daughter or their son. And the other side, if the AirDrop is done and it has no ability to send message, this watch will perceive that and send message to their son or daughter. So, is it ready for mass production now? Soon. Soon? Do you have a price? Yes. People should contact you or price MQ? Yeah, I think maybe it's better to contact us. And this is an MTK solution? Yes, it's 6 to 60. And how long is the battery life with the display always on? Around 4 to 5 days. 4 to 5 days and then you can turn on the backlight and you have some functions right there. And what kind of software is running? We wrote the system past our Linux. We designed the UI for elder user to use and it has a big display. Just a nice dialer. Homebook and message can type some message here. This is all on the Bluetooth? You can insert a sync card here just like a traditional phone for elder people. And have a synchronized function with smart phones. Nice. Can you take it off? Yeah. Try to take it off. Here is speaker, microphone, GPS, heart rate measure. Nice. And here the charger is magnetic tech. All right. Very nice. And you have many different colors? Yes, we have gold color, grey color, silver color. All right. And so how is it going with the other smartwatch you have? Many people buying the other ones? Yes. This one is Android smartphone watch. Best of all Android 6572. 6572? Yes. We also developed the third software and we are now just finished at independent Bluetooth to this watch. So it can also synchronize with Android smartphone. Do you have Android Wear on M2601 or that's maybe coming later? You don't have this yet, right? No, we currently have no plan to develop the smartwatch best on M2601. All right. We want to make our watch better and better. What you will do next? This one is the main function and then some other product coming later. Yeah, yeah. And this one, the starting point here, the heart rate, it can measure heart rate. All the time? Yeah, in real time. In real time? Yeah. And also the heart rate data can send to cloud. So the daughter or son can monitor their parents' health through the cloud. Wow, so it can monitor the heart rate all the time? Yep. And send messages to the parents and check if everything is okay? Yes, yes, you can. Through the cloud. Nice. So this is your heart rate and how long battery life if you measure the heart rate all the time? No need to push a button to measure heart rate, it's always measured? Right, right. We use heart rate sensor inside so it can always measure heart rate even in strong sunlight. It also can measure heart rate.


Another viewer asks, how many letters of recommendation do I need and who should the letters come from? So that again varies from university to university. Most of them require at least one. We require at least one at Catholic and for the most part they're going to come from your teachers but if you want to have it come from maybe a volunteer supervisor or someone who you work with For me it's always a chance to get to know you a little bit outside of the classroom too so someone who you think can speak to you in that regard and really knows you well and can talk about how you'll be able to contribute in the campus community as well as your performance in the classroom. Thank you, that's great advice.


All right. Let's get started. My name is Greg Fauna, and we are here today to talk about testing CEPH, the distributed storage system, and things that have gone well with that over the past decade and things that have gone poorly. I say the past decade because that's actually about how long I've been working on the CEPH system. It's very exciting for me. Quick show of hands. Who here thinks they know anything at all about CEPH? All right. A whirlwind introduction because the particulars don't matter too much. But what's important is that CEPH is a distributed storage, scalable storage system. It runs a whole bunch of different servers, demons on a whole bunch of different servers, and aggregates all the storage in them together, so you have hundreds or thousands of processes coordinating and working together in one system, and we need to be able to test it. At the bottom level of CEPH, we have the reliable autonomic distributed object store, and that's responsible for storing all the user data and keeping it safe in the event of power outages and server failures. You talk to RADOS using the lib-rados library, and on top of that we've built an S3 and Swift-compatible RADOS gateway, HTTP service, a virtual block-to-base, ball-RVT, and a POSIX-compliance CEPH best-distributed storage file system. At the RADOS layer, there are two main kinds of components. There are object storage devices, or object storage demons, of which you have tens to 10,000 in a cluster. They're responsible for actually taking data from clients, and storing it on a disk, and for noticing when their peers fail and and recovering data back and forth. And then you have a small number three to five monitor servers who are responsible for keeping track of which, for taking in reports and saying, oh yes, this OSD over there failed, and sending out updates to the system so that they can make changes in their recovery and who's in charge of doing reason rights. From a client's perspective, then your application runs, and it uses the lib-rados, and there's just sort of a network pipe out to a cloudy cluster. That's a very quick overview. If you want to know more, I'm actually giving you an intro to CEPH tomorrow. Well, we're going to more details, but sort of from the testing perspective, distributed system, lots of processes running need to be able to test different kinds of failures and different behaviors and cross-processes and the ways they interact live. So, CEPH is short for cephalopod, which is the sort of biological family of creatures including squids and octopuses. So, tuthology is the study of cephalopods, and so that's the name of our testing system. Tuthology started out almost eight years ago now. We needed to formalize the way we're testing CEPH. At the time, the CEPH team consisted of three people sitting in office together, writing code, and we had some, you know, scripts, and we had some tests like PJD or, I don't know, debenture, whatever, that we would periodically take and like run against a CEPH FS file mount and see if it passed, and if it didn't pass, we'd figure out why. But it was all very ad hoc. We didn't have anything that was running on a regular basis. We didn't have anything that was guarding git commits, and so at one point I ran a test, and I said, oh, this test failed, and I tracked down why, and I found an if condition that, and I said, oh, that if condition is backwards, and so I swapped it and I pushed that commit into git, and then a few weeks later someone else ran a different test, and they said, huh, something's not working right in this test, and they debugged it, and they ran through, and they found an if condition, and they said, huh, that if condition is backwards, and they swapped it back, and then I happened to run my test again and noticed that the if condition I changed had been swapped back by someone else, and I went and looked, and happened to notice that, oh, it had just been, depending on which test we ran, we were swapping a less than or greater than back and forth, because in fact the problem wasn't that the if condition was backwards, it was that the test wasn't good enough in the code, the logic test in the code, and nothing was telling us that in our regular testing, and that's not a sustainable way to build a distributed system with a lot of people. So eight years ago, nothing existed for distributed systems. We had a had one of a new teammate who was very excited about building a test system, and he tried out, first of all, auto-test, which I think came out of the Linux kernel community, but auto-test was all about doing lots and lots of things to one server at a time, and so it was unsuccessful, and it was very important to us to be able to manipulate multiple machines at once. So he sat down, and the first thing he wrote was called an orchestra module. It was in Python. It used an SSH system and G event. Nowadays, this is probably not anything super exciting, but it was really important. It was not quite as common back then. Orchestra, it was cool for us because it would it let us do SSH commands to different nodes from a controller, and it was real-time and interactive, and we could do things like, and we could do things like access into a server and run a process and get an object back that represented that running process and send it standard input and receive it standard output. And we could do things on a particular server or on all the servers that are of type x86 or all the servers that aren't of type x86 within our cluster. Tuthology itself was and it was initially just a test runner. We run a task that told the system to do something on targets, machines, and those tasks would map on to particular machines via roles. It would automatically set up the system in a set cluster, it would monitor the health of Seth, it would run the test, it would archive whether the test had passed or failed, and the logs that were generated by the running processes, any core dumps that happened during the test, and then clean the machines back up. Targets, this is all in the ML format, were just a list of, you know, use your names and machines that say Gen2, roles would be a mapping saying, hey, on my first machine, I want to run one monitor, one metadata server, and one OSD. On the third machine, we have another monitor and a client, and the tasks would be something like, hey, I want to invoke the Seth task, I want to mount a kernel client on the client.zero role machine, and then I want to run a work unit, which is located at the file suite slash dbench on all the clients. These tasks are called Context Manager, our thing in Thailand called Context Managers. So the Seth task, despite only appearing once, will set up a Seth cluster, and then when all the other tasks are done, it'll tear down that Seth cluster. The kernel client will mount, the kernel, or the kernel client task will mount the kernel client, and then when all the tests have run, it will unmount it before returning, et cetera. And then the tuthology interface would combine all the YAML fragments you give it on a command line, so I can have a Targets.yaml file that I use over and over again that consists of three machines, and I can have a kernel client roles.yaml and a kernel client dbench.yaml that specify, oh, this one's for running the kernel client, and I want to run this particular dbench test, and I'm archiving it in this folder. And then tuthology just shoves all those YAML fragments together into one file that it uses as its configuration. And when the test is run, it logs everything. We've got here a, oh, it passed or failed, if it failed, here's the line that caused it to fail. We have a log of all the input and output that the main tuthology process got, which pointed out all the standard output of the demons running on other servers. We have the configuration that we started with, and the SHA-1 of what version of Seth we're running, and also all the logs that were generated on our remote nodes. That was in 2011. Tuthology today looks a little different. Some of this was done later in 2011, some in 2012, some of it last year. First of all, tuthology mostly, instead of running on a cluster of like five shared dev servers, tuthology mostly now runs in the CPIA lab. This is a community lab devoted to Seth development consisting of several hundred machines that any committed Seth contributor can get access to. More than a hundred people can get into it right now, and all these machines are just 24-7 running Seth tests to ensure stability and test incoming code. When you have that many different people running, it's very important that you not accidentally use the same machines because things go very wrong if you have two different servers issuing commands to turn on clusters and turn off clusters. So you can lock machines. It's not a very sophisticated thing, it's just a database sitting somewhere that says, oh, these machines are available, I've given you three of them. You can instead of running tuthology directly from your server or from your laptop, you can schedule it. And so you just invoke the tuthology schedule command instead of tuthology and it'll take all the YAML files you give it and put them into a beanstalk queue. And then there's a server running in our lab that has like 50 tuthology worker processes that go, oh, hey, I'm idle. Are there any tuthology jobs available to me? Hey, here's one. Let me walk the three nodes that are required for this job, enter on this job and store the results, and then, hey, are there any jobs available to me? It's super primitive. Second of all, instead of having a bunch of, instead of having whatever random, yeah, whatever random YAML fragments you want, we've organized them all into suites and have a command that can assemble those suites into specific jobs. Suites are just collections of directories or directories of YAML fragments that are combined in various ways. So here's a small example from one of them. We have a verify suite inside of our radius component that runs through a bunch of validation. Now, there's a lot of different ways that you can configure steps. So this just does a couple of different things in this example. First of all, we might thrash, either the default or the no thrashing, and that's doing things to the cluster in this test. Next, we can choose the object store, which lets you decide how data is stored within an OSD. Either our classic, we have an XFS file system to store data in, or we have this newer thing called BlueStore. That's our own custom block-wise access survey. You can figure that BlueStore in a couple of different ways.  We can test the recovery of the monitors, or that the Liberator's API works, or that something called Radar's Classes function. And when you run two-thology suite against this directory, then the first two lines up there just keep your agent and share it all the jobs. It'll go through and it'll select one YAML fragment from each directory to create a job out of that. So this first one is running with the default thrashing and this BlueStore bitmap and monitor recovery. And that's job one. But then job two is with the Radar's API test, and job three is with the Radar's Class, all tests. And then job four switches the object store type to this BlueStore compression and goes back. And so we get a combinatorial explosion of all the different kinds of fragments so that we can make sure that if we add some new feature, we can just add one new fragment that controls that feature and run it in combination with all the others in the system. The QA suite test coverage is pretty extensive. These are some of... Well, I don't know if there's still all the suites we have, but there are all the suites we have when I generated this slide. Most of these things are different set components. Some of them are targeted suites that are used for specific things. I alluded before to the thrashers. So thrashers are sort of chaos monkeys that run while other tests are running, our other tests are in progress, and can do things like say, hey, I'm going to randomly turn object storage demons on or off or change the way that we shard our data across the OSDs or do other mean things, maybe bring them into the cluster or take them out. So that's sort of the basic function of deutology. So here's how we use it. First of all, developers are granted access to the lab and so they can run tests on work-in-progress branches as they're building up new features. We can push a branch to a special sefci.git repository on GitHub and it gets automatically turned into set packages. And then we invoke the deutology suite command saying, all right, I want to run on the set of machines, the Rado suite against my branch, drag one, two, three, four, five tests. And that just sends off all the jobs in the Rado suite into Beanstalk and it turns through them and the results get posted publicly at polpito.sef.com, which is our test result viewer. Second, pull requests get submitted to us on GitHub and the tech leads and reviewers go through those PRs and when they think that they might be ready for merge, then they build integration branches out of several PRs at once and run them through the appropriate suites to check for issues. We have some Python schooling to help with building those branches and pushing them for builds and running the tests and nothing merges to the sefmaster branch without passing these tests and then the results are publicly available and so when a test completes, then the reviewer says, well, something failed. I think it might be this PR so I'm gonna have that person look at it or hey, everything passed, I can merge everything. And then finally, we have a variety of nightly jobs. Most of the suites are run from once a week to every single night and again, all the results publicly posted. You can see what jobs we actually run with that cron tab configuration that's on GitHub. Super nice. However, there are some gaps in tuthology where we have not succeeded. I don't want you to think that we're doing too bad a job so we are actually successful in a lot of ways. We're like, we're pretty good at this, it turns out. Functional coverage is great. We have a lot of specific tools to test behaviors. The Ceph test rate as binary is a thing that generates random API calls and knows what results it should be getting based on what it's putting in the past and we run that against all the thrashers in the system and make sure that the client doesn't see different results depending on what failure rows happen because that's one of Ceph's guarantees is that you don't see that. We deliberately inject failures to test demons like saying, hey, I want you to randomly fail to deliver one message out of 100 to the OSD in the messenger layer. We do things like fiddle with the RADOS objects and make sure that CephFS or RBD on top of them notice and recover those from those failures in the ways that we expect. We can go poke at files in XFS and make sure that the OSD notices and that the total RADOS system notices and recovers in the ways we expect. And do you think that's not the only way we test Ceph? Different sub-components and associated projects like the Ceph Ansible installer or the Ceph volume configuration and OSD deployment tool have their own testing systems. And we have a whole bunch of unit tests some of which are not really unit tests but you pretty sophisticated things that run inside of our make check build target that's built on every single pull request and whose results are posted so you can see if it's passing, those are not. But now that I've covered my butt about how we're great there are some gaps. Tuthology handles demons directly. It SSA just into a test node and runs a process in the foreground which means that it doesn't test our init system configuration scripts. Tuthology does not do performance testing. We have a new per suite that uses a separate thing called the Ceph benchmark tool but there's no analysis up, no automated analysis of it and the tests are very limited. We expect in Tuthology that tests run within hours in as little space as possible so there's not a lot of scale testing or long term testing that the system does. And finally, Tuthology handles the Ceph installs and cluster config on its own so it does not test deployment tools. There's some fixes that we've engaged in and some that we've planned for these issues. We want to expand the API so that we can restart and signal using the init system instead of literally saying, hey, I want to go to the monitor node and run this, fill all that in my command. Which is how a lot of stuff, not everything, but a lot of things happen right now. In terms of performance testing, for several months now, we've been gathering some historical data in our per suite and we want to start guessing at a rough limit for how long is anomalous and failed tests when the times start taking too long, but even that's pretty primitive. So long term, it would be great to track performance numbers and alert when they're out of bounds based on some prediction of how we expect a cluster on these nodes to perform. For scale testing, we just decided, you know what, Tuthology's the wrong place for scale testing, it's an integration test suite. So within Red Hat, we've got tests and a whole separate group that runs tests of long term allocations in their own lab space and there are a few things that do this not too many, but we'd like to start mocking up and injecting age data and large scale data within shorter tests by, you know, running something for a while and then cloning the disk state and being able to copy that in very quickly without going through set and see what happens. This last one is a pretty big problem for us actually. So when we started Tuthology, it was our expectation that all of the set users would be writing their own Chef or Puppet scripts because they were managing data centers and set would just be another thing they managed in their data center and that turned out to be very wrong, but it never got updated. So we need to write a new API to request installs and farm them out to things like so fanciful or sooses deep sea or roofing Kubernetes or whatever is next. And that's what I'm working on right now. And finally, sort of related to that problem, we have a new component in Chef called the Chef Manager and the Manager is a Python service unlike the rest of Chef, which is all in C++ and it handles some reporting from the nodes and aggregating data and things like running a web dashboard. But it's also got one set of cool new functionality coming in called orchestrators that let us do day two operations and things like say, hey, I wanna provision some new OSDs on these servers or hey, I've got an NFS cluster that's backed by ChefVest and I need to deploy a new NFS server. And those orchestrators rely on Chef Ansible or deep sea or roof to do those things because it doesn't talk to them themselves and because there's no integration of those components into Chef, then this is a new key piece of functionality that you can't test in tutology right now. So it's really important that we do that. And it's a good example or a very bad example of how missing testing integrations can have long-term consequences that you don't initially foresee and that you need to be aware of. So that's some stuff that we're missing just in terms of the tests we've written within tutology. But there are also some weaknesses of the framework itself that you wanna be aware of if you're starting to build this sort of system up for your own project. First of all, tutology is strongly tied to Chef. When it was written, it was intended not to be. Orchestra should be usable elsewhere, although I don't know if there's any value to it at that point anymore. And tutology was designed to support other multi-node systems, but nobody else uses it. So we sort of things creep in. In retrospect, we should have found at least one community to use tutology with us. That would have been good. There were other communities that needed things like tutology and that we probably could have worked with if we put the effort in, but we didn't think about it. At this point, we're not gonna worry about it. Orchestra and tutology are reliable code bases that serve our needs. And if you're building a new testing system, you shouldn't build a new testing system, you should find one that exists. You're not gonna get everything you need in one coherent package, but there are things like Zool. There are things, you know, there's Jenkins. There's a lot of different CI and test automation that's available now that you can build systems out of. And there's a lot more interest in distributed systems testing. Second of all, well, big problem. Tutology is strongly tied to CPIA. It's not supposed to be, but hard-coded values sneak in like assuming, oh, all my new packages are located on the set.com domain and not my private downstream repo domain. And because it's a very internal project, then the documentation, while there is documentation, it's often unclear and it's a bunch of different services that you need to wrangle together. Now, the upstream project community is not the only group running it. There are other groups, either distributors like Red Hat or private companies running their own clouds who do downstreams and run Tutology for their testing, but they tend to do a local port where they go through and scrub all these things out. And that makes collaboration between us a little harder than it should be. It also makes developing tests like new tests difficult because we need to get machines in the CPIA lab. We need to image the machines. We need to install packages in order to find out that, oh, we really should have run PyFlakes on our Python code first. It means that you need built-Seph packages to deploy to the CPIA lab. Once upon a time, it actually just built a tarball of the locally built SEPH executables you'd built inside of your source directory and SSH them over to the machine, but that doesn't work when your machines might be different and you have more than four shared dev servers. And that's sad because it means you need to go through a whole package build cycle instead of just testing local source code. And all of this together makes it hard for third parties to compute to mainline Tutology development and for new SEPH contributors to test their patches and that's bad. There are things we should have done to make this better in hindsight. If you don't test something, it doesn't work, so we really should have done regular teardown and set up a part of our practice of the Tutology system. It would have been good when we discovered group sample trouble set up and install it. We'd worked a lot more closely on it because we've had some attrition of people who have the skills to work in the system and that's scary. And when we did some changes like switching to require packages, we didn't think a lot about people who weren't us and that was detrimental to our community. I'm not sure if we would have ended up making a change given the advantages but it wasn't even something we considered. There was a thing called Tutology OpenStack to solve some of these problems. It was a set of scripts. It would talk to a random OpenStack cloud and turn on Tutology and run a test suite that you used to ask for. And that's a great thing to have in theory but it died upstream due to lack of maintenance and interest and because it was running and using I think the actual CLI tools, not some library and all the commands changed. Over the years. At this point we have made some strides to move away from this problem about being strongly tied to CFIA. The file system and some of the RGW gateway tests run against something called a VSTART runner. This is a restricted API, not the full Tutology sub thing but specified subset of it that lets you write a single test that runs in a local dev system or in the full Tutology lab. We're doing a lot more work to try and foster community around CEP. We have a weekly meeting to talk about testing issues and things that people are having trouble with their working on in that part of the project. We've tried to reach out to the known users because we found out about several of them about a year ago at our first CEPLICON that we had no idea existed and we're working to merge enhancements and fixes upstream from those groups. In the future we'd also like to find out what things people have done to do the localizations, what strengths they've had to change, what assumptions we had about resources being available that might not be true and make those configuration options instead of just things that you need to change in the source code and it would be great to rebuild Tutology OpenStack with LibCloud or some other newer, more reliable API. Okay, so Tutology as a framework is good for looking at an individual test and saying whether it passed or failed. We can make it fail based off of all kinds of things that might happen in the system, core dumps, it scans the logs and can say, oh, I see a warn message or an error message and I wanna fail, we can say in the code, hey, I wanna fail if this thing is true or fail if this thing is false. It's very robust about that but there's not a good way to say, oh, this YAML fragment has failed the last 15 times it was included in any test or to say, oh, this test is now taking twice as long as it was three weeks ago and that means that we're weak at certain kinds of problem identification. It hasn't become a critical issue yet so we don't have any specific plans but this is something I hope that one of the AI tracking systems might eventually be able to help us with. A big problem for a while that we've mostly mitigated is that suites have really exploded in size. They are a geometric combination of all the fragments so the RADO suite last time I checked this was well over 100,000 possible jobs and there's not really any way to prioritize or any of those tests to know if 15 things failed which is the most important to say, okay, like here are the fastest tests and I wanna get feedback from them really quickly while working on longer term ones or for resource constrained users to know which tests are most important to run. A couple of years ago, we added something called subset functionality which is a big critical thing for this. So the subset functionality lets you say, hey, I want to run subset one of 500 and it will make sure that we run every single YAML fragment but not every single combination. So when we do this one of 500, we add up to 397 jobs instead of 124,310 and that won't do all the combinations but it covers everything at least once and importantly, if we then run subset two of 500 and three of 500 and four of 500 all the way up to 499, 500 and 500 and 500 then we will have run all 124,000 tests and so this is a good way for us to step through incrementally every time we run in the nightlies while still getting quick feedback of the things that, while still getting fast feedback that something's wrong while maybe not testing every single combination every single time. In addition to the subsets, we also have filtering options that let you say, hey, only run things that match this particular regex in the description of the test. We can say, hey, I want to rerun all the tests that failed in this last indication because I think I fixed it, fixed the problem I found and I want to check. I'm going to do more in the future but this hasn't become a critical issue since we did the subset. More important, scheduling for us is very primitive. Jobs are picked off of Beanstalk and then one of 50 processes that has locked them tries to lock the servers. So if you have a five no job, it never runs because we have a bunch of two no jobs that just always say, hey, do you have two no's and the assistant says yes and then the five no job says so you have five and it says no and the two no jobs always win that race which means that besides those jobs never running we can also find out that scheduled nightlies can be delayed and then run on very old code. So we've come with a number of hacks some of which have patterned themselves in our brain in ways that, in our brains in ways that are pretty bad but we usually but not always notice. So we make all our jobs run in the same number of nodes at this point so that they actually run. But that means we're restricted to jobs that run in the same number of nodes and sometimes we forget that when we're talking about what our coverage is actually like. We have people in our QA team who monitor the queue and say, oh, we've got 7,000 jobs running and 7,000 jobs queued up and 6,000 of them are nightlies that are more than three days old so we're just gonna kill those. One of the earlier things coming up in the framework itself is to update this so that we can actually lock the number of nodes that are required before jobs come off the queue and so we can do that in order to make sure that five node jobs run. We'd also like to write a more robust scheduler but we're not sure we're gonna have any scheduling or any time developer time for that. One thing I also want but don't really know if we'll be able to do is to say, hey, like we've had seven people manually schedule a run on master of rados so we probably don't need to run that one in the nightlies today. And a new problem that's come up recently is that physical machines are first class citizens in toothology. We say, hey, I need to install Samba on these two machines but not this third one and that's okay. And some of our suites do things like say, hey, I want to power cycle that server. And demons, which we run, also belong to the machine that we run them on but that doesn't work for Kubernetes because none of those concepts really make sense for it and Kubernetes is becoming very important to Seth. We're the main storage provider for Rook which is the main storage provider in the CNCF that lives within Kubernetes. So our plan is to define new packages for making those, or new interfaces for making those packages available and starting them but we haven't made a lot of progress in it yet and that'll come right after the deployer interfaces. So that's what we've learned over the past decade. So are we going to stick with it in terms of running our own test system? Yes, we're keeping our tests. It's been a bunch of years of development and it's critically toothology is proven effective at exploring the state space of Seth and testing recovery from failures in ways that other systems we have and that other components work with just don't. At this point it's operating successfully with low maintenance requirements and we have developer experience with the quirks but we are trying to make life easier on new desktop users and update it for the new requirements for development. Should you be doing automated testing? Yes, you should definitely be doing automated testing. Manual testing is terrible. You can never scale it the way you need to. But you probably should not write your own test framework the way we did.  If you wanna build your own, I have some questions you should probably ask yourself. First of all, are you sure that there isn't something public that you can't use? Now it might be that one of them won't solve all your problems but you can use it in combination with others. So there's things like G-Test that are designed for any unit test or Pi-Test or whatever's in your language of choice all the way up to OpenStack Zool which is designed for multiple projects integrating together. And even if you don't see something like that there's probably a software project that does something like what yours does and that probably does testing somehow. And so if you really think you wanna build your own you should know why your test needs are different from projects like yours and why the capabilities that exist out there aren't sufficient for you.  Small frameworks are a lot easier to embed in other systems later on or extend. And that's one of the problems you've noticed with tuthology is that because we built our test framework up around SSH again and running commands on machines then luckily things we do are pretty straightforward like restarting a process or killing it or whatever but sometimes we just like ship off a bundle of Python code and say execute this over there. And that is also a thing we can automate but you know like sometimes it's more complicated build up so of shell commands. And it's good to try and keep your components discreet so they can be replaced later. This is another one of the problems we've had. A lot of our, we've had new developers come on board working on things that are sort of peripheral to SEF and they say wow, tuthology's huge and complicated and I don't want to work and I don't want to have to go through it when I'm testing my system. So they built a whole new system out of completely separate components because it was hard to extract the useful bits of tuthology for them. I've sort of alluded to this in the past but there are some comprehensive changes coming. SEF is going in really big on Kubernetes and Rook and we're actively discussing how to test that. We aren't sure yet how we're gonna run Kubernetes inside tuthology but it's gotta happen and like I said, we're figuring out how to do these deployer interfaces so that tests that we write to test these have kind of functionality can be agnostic to new environments and new ways of deploying things in the future. So, I guess I'm done a little bit early but we have plenty of time for questions. Sounds like a SEF question. Can you resize the volume or is this about the testing? We are using goal quest like questions. Okay, well if everyone has lots of those not testing questions then we can do that but I also will say we have like five SEF sessions tomorrow that are actually about SEF maybe better suited for that. Anyway, I have no idea what it requires. Open stack and poses on changing volume sizes. SEF allows that. I must afford you a lot of tears, I'm sorry. I thought I was gonna run out of time or I would have done some extra questions about community engagement. Yes? Do you know the Jepsen test? Ah. Do you have something similar? Yes, so do we know about the Jepsen test? Yes, Jepsen is a framework for testing distributed systems and the way they respond to different kinds of standard failure scenarios. Jepsen was created after Geutology was and they do a lot of the same things. Not exactly the same things. Jepsen's focused a lot more on specific message flow than SEF is but the thrashers and the RADOS API tests that we're doing are pretty equivalent to exactly what Jepsen does except that Jepsen is mostly black box testing because it doesn't know what the system looks like in the back end. But yes, Jepsen is a great example of a thing that if you're building a distributed system, you should be aware of and either run or know why your thing is better than it. That case, I'll let you go. I'll be around. We also have a SEF booth up in the lobby. If you have specific questions, feel free to come by there and someone should be around most of the time to talk about you and your use cases. Thanks very much.


After fall assessments were completed last month, those with a risk of falling can now attend balance classes. Crow Wing Energize has put together two hour long classes that take place once a week. Over the eight weeks, attendees will learn about the financial and emotional costs that falls have on families. The program's goal is to promote work, play, and being independent. A lot of great education that is given to them on ways to prevent falls, but also ways to gain confidence and be living independently. They also do some chair exercises and different types of things to help promote growth and balance and different musculature things that will support their work. This is only the second year the program has been available to the Breinard community. If you've enjoyed this segment of Lakeland news, please consider making a tax deductible contribution to Lakeland Public Television.


Hey guys, Coach Mack here with this week's Beneath the Surface powered by Microsoft Surface beginning now. Today we're going to look at four excellent plays by the Titans defense against the Los Angeles Chargers in SoFi Stadium. First play, second seven, first quarter. Balls on the plus 25. Chargers are in 11 personnel with Trips Speed Alignment, which means all three wide receivers are to the defensive left. Titans in a four man front. Man to man with single high safety with a low hole rat in Monty Rice number 56. TT game inside with Big Jeff and Nacwon Jones. Simmons makes a nice step outside on the guard followed by a perfectly executed two-hand swipe and swallows the quarterback for a nine yard loss and a Big Jeff celebration. Next play we're going to look at. This is second ten, second quarter, 12 seconds on the clock, scores tied at seven seven at the plus 25 yard line. Chargers are in 11 personnel two by two with top of number splits by the outside receivers on both sides. Titans are in a four man rush, deep sink quarters to protect the end zone. Two outside wide receivers run a take off route. Herbert launches to Mike Williams, but Dupree has a perfect speed to power rush. Does not allow Herbert to follow through on the deep throw. McCrary makes an incredibly heads up athletic play by elevating, seeing Calou behind him and in bounds. Volleyball is the ball to him and Calou makes another athletic play by not only catching the football and being very alert to what McCrary was trying to do in this unusual situation but also getting both feet down. One of the most incredible interceptions I have seen in my national football league career which is a long time. Next play, third and five on the plus 49, 1332 in the third quarter. Chargers are again in 11 personnel, two by two, shotgun formation. Titans are mugging both linebackers in the A gaps with wide rush alignments by the front four. Titans bail out on a five under two deep zone coverage, confuse Herbert with the mug droppers and pre snap bogey coverage. These lines slide to Big Jeff, allowing number 95 to Marcus Walker to power rush to guard and eat up Herbert for a big sack on third down for a seven yard loss. 553 in the third quarter, second and eight on the plus 27, Chargers in 11 personnel, two by two, jet motion by number 13, Keaton Allen behind the quarterback and shotgun making it a three by two split. Titans are in four man rush, quarters coverage, Kevin Byrd playing the inside quarters and gets an excellent read on the dagger route which is a deep dig to his side. The Marcus Walker number 95 gets a tremendous power rush on the offensive guard, makes Herbert move off the spot, Byrd gets an excellent directional delivery key and steps in front of Williams for a tremendous tremendous interception in zone coverage.


OK, so let's try this kind of generalized pH problem. It's a little more complicated because you're actually looking for the mass of sodium hydroxide. So this one says, how much sodium hydroxide in grams is needed to prepare a 546 milliliters of solution with a pH of 10.00? So let's attempt to do this problem. So the first thing we need to know, of course, is that the pH calculation or equation is pH equals the negative log of the hydronium ion concentration. OK, so from that, you can figure out, well, what's the hydronium ion concentration? And that's going to be 10 to the negative 10.00. So you can figure that it's 1, 2, 2, mole like this. So from that, you're going to figure out, well, what's the OH concentration? So remember, kW equals the hydronium ion concentration times the hydroxide concentration like that. And kW is something that's going to be given to 1.0 times 10 to the negative 14. So from that, you should be able to figure out the hydroxide ion concentration. That equals kW divided by hydronium ion concentration. So that equals 1.0 times 10 to the negative 14 divided by 1.0 times 10 to the negative 10. So that's going to equal 1.0 times 10 to the negative 4 mole. So does everybody follow me there? So the other thing you want to know is, well, the ionization reaction equation for sodium hydroxide, so when you put that into the solution, it ionizes to van. So the concentration of sodium hydroxide is going to be the same as the concentration of hydroxide since everybody follow me there. So let's just formally do that. So that's the molar of hydroxide. So for every one molar hydroxide, we have more molar. So the concentration of NaOH is going to be 1.0 times 10 to the negative 4th mole. So we have the volume of solution. And if you recall, hopefully you do, that mol, molarity is moles per one liter like that. So that's, remember, just a fancy conversion factor. So can I erase this portion of the stuff that we've done? Is everybody OK with that? What do we got? We're saying the volume of the solution, we're looking for the mass of sodium hydroxide. So let's look for the number of moles of sodium hydroxide. We've got. So remember, molarity equals moles per volume. So the number of moles is going to be 546 milliliters of 1,000 mils at the bottom, one liter at the top. And then multiply that by that molarity, 1.0 times 10 to the negative 4th moles per one liter of solution. So what will happen is cancel, cancel, cancel, cancel. So we got moles left, right? So well, since I don't have any more space, let's just solve that problem and then we'll go to the next problem. So 546 divided by 1,000 times 4,6 times 10 to the negative. Moles like that, is everybody OK with what we've done so far? OK. So we want to figure out, well, what's the mass in grams of sodium hydroxide? So if I remember correctly, the molar mass of sodium hydroxide is 40.00 grams per mole. Yeah, that's correct. So let's just do it this way to formalize the whole thing. So moles of sodium hydroxide at the bottom, we're going to put one mole of sodium hydroxide. At the top, we're going to put 40.00 grams of sodium hydroxide. Cancel, cancel, and that's what we were looking for, right? So thanks, goodness, after all that, we got to the right unit. So let's just multiply that number we have in our calculator by 40. And this, of course, is going to go to three significant figures. So it's going to be 2.18 times 10 to 3 grams of sodium hydroxide. I may be would have asked that in Miller hands, but I don't know. Does anybody got any questions on that one? Wonderful.


This is stock number 10264. We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used heavy-duty truck and Ram headquarters. Today we are checking out this extremely clean 2013 Ram 2500 Crew Cab Short Box Laramy Longhorn. This vehicle has a 6.7 liter Cummins diesel and from this HD video you will be able to tell just how clean this truck is all the way around inside and out. Black over white gold is the color. We shoot all of our videos in 1080p so if you have HD capabilities on your computer tablet or smartphone device turn them on right now because it's like you're right here looking at the truck with me and definitely your best way to check out the vehicle before seeing it in person. It comes with the factory polished aluminum rims and this one has brand new hand-cooked Dyna Pro LT 26570R17 tires. We put those tires on in our safety inspection and they are brand new all the way around. The frame and underbody looks really good. Didn't see any dents or dings on that front fender. It has the projector lamp headlamps, the LED running lights, the factory fog lights. It has a nice rough country chrome grille guard, bumper guard, chrome trimmed grille. Didn't see any dents or dings on the hood. That's an excellent condition. Bumpers in really nice shape all the way around. Passenger side fender, no dents or dings in the passenger side rim. No scuffs or scrapes. As you go down this side of the truck and see just how reflective and mirror-like that paint is, we take these HD videos so if you are far away or even if you're close by and just can't make the trip down you can still see the truck, hear the truck and have confidence in the vehicle that you're looking at before you even get here so that when you do get here there's absolutely no surprises. Comes with a back rack, headache rack. We even have an option on our website for the FastPass option so you can basically fill out a credit app, apply for financing, get payment quotes, appraise your trade and do all that good stuff right from the comfort of your own home. So when you get here you're going to be 100% ready to go. Back tires are brand new as well. That frame and underbody looks really good. It looks like they haven't messed with the exhaust at all. Coming around to the back of the truck. Rear bumper is in pretty nice condition. No major dents. Has the backup parking sensors. The tailgate is in really nice shape as well. Didn't see any dents or dings on that. It does have a locking tailgate and a backup camera. Spray and bed liner. Bed's in pretty nice shape and it has a, let's see, brand is here. Undercover flex, fold up tonneau cover. And down this side of the truck just as clean as that passenger side. You want to make sure you get the most accurate representation of the truck possible so when you get here no surprises. And the back rim excellent shape as well for full disclosure. Down the rest of this side of the truck just take a look at how mirror like that paint is. Very, very, very nice on that truck. Has the heated fold out tow mirrors. They fold out like so. They also fold in. I'm not sure if they're powerful or not. We'll check out a second. They're not powerful. Power windows power locks and power mirrors. You get the nice real wood grain trim on the Laramie Longhorn package. Speaking of that package gives you nice dark brown leather seats. You get the Longhorn patch stitched on the back, the black piping, memory driver's seat, lumbar. Both these seats are heated. Both these seats are cooled and power. Factory all other floor mats, auto headlamps, power pedals. And as you hop inside the truck here you can see that this one has 56,918 miles. You get the 7-inch multi-view LCD display, digital speedometer, outside temperature compass display, range to empty time. You can put basically anything you want in there. That's the Laramie Longhorn instrument cluster. Special for this truck. You get the heated wood grain trimmed leather wrap steering wheel, cruise controls on the right, Bluetooth and information center controls on the left, and audio controls behind the steering wheel. Six-speed automatic transmission, the 8.4 3C radio with the factory navigation. It also has AM, FM, and Serious XM radio capabilities. You can do your heated and cooled seat buttons as well as your heated steering wheel button there. And this is also where your backup camera shows up. That works really nice. Turned out four-wheel drive. You can do your climate up here or you can do your climate down here if you like the more tactile buttons and knobs. Factory brake controller. Once again, real wood grain, real wood on the wood grain trim there. Brake controller, heated cool seat buttons as well on the bottom here. Once again, if you like the more tactile version. Stability control, tow haul, factory exhaust brake, backup parking sensors. You can turn those off in the light tire load button. 115 volt, 150 watt plug-in, passenger seat is very clean as well. No rips or tears on that seat. Headliners in excellent condition. You have the speakers in the ceiling, part of the alpine sound and the power sunroof. You get your map lights, power slide and rear window, home link buttons for your garage door security systems and lighting systems. And then you also have the alpine badge right there signifying you get that special sound system. It has a remote start as well on the key fob. And we'll take a quick look at the back seats and then we will check out under the hood. Very nice inside. This truck's never been smoked in. Smells very clean. Back seats are just as clean as the front seats. No rips or tears back here. It does have that power sliding rear window, long horn patches stitched into the backrest here. These back seats are heated on the outboard side. What do I mean by outboard side? That side and this side, not the middle seat. And the carpeting is very clean as well back here. No rips or tears. You get in-floor storage bins with removable bins back here for easy cleaning. You can put ice and drinks in there, storage underneath the seats. Got LED lights on the bottom of the seats. And of course the Laramie Longhorn comes with the saddle bags for storage back here. Very cool truck, very clean. Get the wood grain trim on the doors back here. Child door locks. And we'll take a quick look under the hood. I would personally like to thank you for checking out the video today. And hopefully from this HD video you will have been able to tell just how clean this truck is all the way around inside. Now under the hood we have the 6.7 liter Cummins diesel engine base. Very clean, runs very smooth. This truck has been fully safe and inspected by our service shop. Has a fresh oil and filter change. All the fluids have been checked and topped off. The truck has been gone through mechanically 100% and is 100% ready to go. Four brand new tires and really really nice condition. To see more pictures of this truck or one of our other 450 new and used cars, trucks, SUVs, minivans, wranglers, halftons, three-quarter tons, one tons. You name it we got it go to our website www.summitauto.com. Full pictures and descriptions of every single vehicle from two locations all at summitauto.com and if you'd like to check out more HD videos you can go to youtube.com slash summitauto. Remember to like, subscribe, and share on this video and all the videos that you see there. In fact in a second you will see a link to subscribe to our YouTube channel on your left. We linked more heavy-duty truck videos like this one on your right. If you have not been to our website on the bottom link to this vehicle on our website click those check us out and we really look forward to helping you with this Super Clean 2013 Ram 2500 Crew Cab Short Box Laramie Longhorn Cummins Diesel. Thanks again.