Hello, everyone. Yeah, it's already time. Oh, this is the one slide. Sorry about that. Yes, this is the right one. Yes. Yes. Hello, everyone. So I am Taejun and Johannes. We are from Facebook, and we want to talk about resource control at Facebook. So I want to start with this graph. So imagine a web server, a Facebook web server, which is running, like, filled up to the brim. It's saturated in terms of CPU, in terms of memory, in terms of disk. It has hard disk, too. So it's really filled up the brim. If you imagine a web server like that in a fleet like Facebook, there's a lot of management service going on. There's Chef running, which also runs YUM from time to time. And Chrome jobs, all the monitoring. So there's a lot of infrastructure pieces going on in the system. And imagine one of those support services slowly leaking memory, let's say 10 megabytes per second. You don't really notice that first, but it's going to add up pretty soon, and you're going to see problems. If you think about, let's say you run a Chef and somebody wrote a wrong script, and that's kind of leaking memory. And it's not really an essential part of the system. You want it running, but you don't want your website to go down because of that. So it'd be really nice if you can protect your main workload, the web server, from those kind of things. So if you look at the purple line, the y-axis is RPS, request per second, how many requests is processing. That's before we did what we did. So that red marker, 10 megabytes per second grows on both, means that we started a memory leak intentionally in one of those support services. And memory grows up, and memory runs out, and the whole machine crashes down. RPS drops, the machine is crashing now. It cannot really do anything else. And eventually, after like half an hour, the machine gets rebooted, and then it recovers. There's like a 40-minute recovery process. If this is happening at a wider scale, let's say you somebody deployed a Chef recipe, which he did that at the same time. That is taking down Facebook. That's the whole site going down. And the green line is what we did. Green line is the resource control. So we used that for resource protection, and resource control kicking in. And while the RPS drops a bit, it holds on. And we did that three times, and the system generally survives pretty well. There's minor annoyance. People experiencing some latencies in their use of Facebook, and some uncles getting paged, and that's about it. It's not a major event. So this presentation is about going from purple to green, how we did that. So we are a resource control group or team at Facebook, and this is our motto or our mission statement. World-conserving, full-OS, resource isolation. So it's a bit of a world salad, right? So let's unpack that. So the first thing, world-conserving, world-conserving means that we want to use the machine and the resource work to do, right? We don't want to put in resource restrictions in a way which limits the overall utilization of the machine, because that's wasteful. We want control, but we don't want to be expensive doing it. The second part, full-OS, is that if you imagine using a VM for resource isolation, it's a lot easier, right?  Resource control-wise, it's almost trivial. You can easily do that. But then you lose all the integration with the host system, right? You lose access to files on the host system. You can have other workloads, but that's really... Or you can restrict iOS to direct to iOS, right? That also makes the problem easier if you bypass memory management page cache. But these things all adds up to operational complexity. And so what we wanted to do was keeping... To make resource control transparent to the rest of the system, meaning that applications and users can keep doing whatever they have been doing and whatever they want to do, and we want to lay your resource control on top so that it just works transparently, right?  You put resource control on top of it, and it just works the same except for the added resource control. That's what we wanted to do. So if you think about doing that, imagine that we never had that. We never had... And we needed a project. We wanted a project which is simple and which we can demonstrate that this is useful and this is workable. So we chose a project called FB Text. FB Text means that inside Facebook, the text that every machine has to pay to be part of the fleet, or the management and the monitoring overhead that we have to pay, that part is called FB Text. And sometimes, as I said in the first slide, the text part misbehaves and brings down the system. So our initial project was, it'd be great if you can protect the main workload from the text part. So that was FB Text, the name of the project. It became the name of the project. And as you can see from the two parts, FB Text 2, the first attempt just failed miserably. We did something we thought would be effective and it just made the system more fragile, more brittle. Now nobody was happy. So we had to go back to a drawing board and rethink everything. And so what was the challenge about it? I mean, it doesn't sound that difficult, right? I mean, how hard can that be? And the challenges were in almost all areas. Yes, in almost all areas. And the first was that memory that high-end max is more like in C group one terms, right? That'd be memory limit in bytes. That would be the natural thing that people would think about when they think about controlling memory distribution. What they do is saying that you can only have this amount of money, right? Not money, memory. Yeah, but memory is money. So how we can do it for FB Text would be we can put a limit for support services, right? You guys can only use, say, 4 gigabytes or 6 gigabytes. You can use no more. That didn't really work that well. Because if you think about it, right, if you put in restrictions, that means that you are lowering overall utilization, right? When support services has to have more memory to run reliably, if you put in this artificial limitation, it can't. And the thing is that on systems which are running up to the brim, a lot of resources are nominally oversubscribed in temporary terms. So they have to be able to go over this artificial limit sometimes. So this didn't really work. So the moment we put in these artificial limits, the systems started failing more. And we had to inch up the limit to a point where it is not meaningful. So we couldn't lower utilization to protect the workload because the system is already too busy. The other part was that Colonel Oumkiller didn't really work well. The main reason for that is that Colonel Oumkiller kicks in only when the Colonel thinks that the Colonel is in trouble. If the Colonel cannot make forward progress, then Oumkiller kicks in and kills something to make forward progress. It doesn't really know or really care about whether your workload, your application is running fine or well. It doesn't know about that. So the system can easily enter a state where the Colonel is thinking, everything is fine while the application is rushing for 20 minutes. That's what you saw in the first slide, too. That's why the recovery takes 30 minutes because for 20 minutes of that time, the Colonel is not doing anything. He's thinking, oh, it's fine. So that was another challenge that we met in terms of memory control. And IO control, IO was hard, too. Maybe even more difficult. The first one is that we didn't have a good IO control to use. CFQ didn't really work well with SSDs, and even on hard drives, we had a lot of issues in production, so we couldn't really use that. And there's another controller called IO throttle, but that's the same thing. That uses memory IO.highendmax, so it limits the total amount of IO which can be done, and the same story. You cannot limit and survive. And another thing is that the BPS and IOPS is what IO.max uses, and they're just not a good measure of IO capacity. It's just really difficult to find a good configuration with those two parameters because they don't really reflect how expensive these IO streams are to the device. And another issue that we met was that we were not accounting. The IO controller part was not accounting for system metadata operations and swap operations. We get to swap. You kind of need to swap, especially with SSDs. It just makes more and more sense. And we were not accounting them properly, so we were charging them to the root C group, to the system, and that caused a lot of power convergence issues. And power convergence. So for the past couple years, we have been really working hard at this problem, and earlier this year, we thought that we now nailed the memory controller. We nailed the IO controller. It seems to be working, and then we tried to put them together and found out that nothing works. Nothing still works. And the main reason for that was that there are a lot of major priority convergence in the system. For example, also, the kernel basically assumes that everything in the system can make full progress. Otherwise, it locks up. I mean, so if you, like, one process runs out of memory, what the kernel's basic assumption is that I got to make that unstuck. Otherwise, the whole system is going to, you know, stay sealed. So it throws everything, you know, your priority configurations or whatever out of the window and tries to reclaim memory as hard as it can.  But I mean, if you take that approach to, you know, research control inside the system, right, doesn't really work because we want to be able to slow down a part of system really badly while maintaining health of the, you know, a more important part of the system. So the kernel can no longer operate in a way which assumes that everything has to make progress, right? I mean, something is going to make a lot faster progress than some other things. And so this creeps up in a lot of areas, but usually around five system and IO operations. So, for example, EXT4, it can be fixed, but EXT4 in the data journal mode, the default operation mode, can create hard data dependency through its journal. So like a high-priority C group would have to wait for a lower priority IOs to finish before making progress.  And the five system metadata, SOAP IO, those things are the same, right? I mean, low-high-priority, you need to make the distinctions otherwise low-priority can, you know, completely mess up high-priority performance and latency. There's MFM. MFM is another interesting one. So MFM is something which protects a process structure, like a memory layout over process. And one really interesting part is that when you do PS, right, the information, the command line comes from inside the memory of that process, right? So when you do PS, it has to go inside the memory space of the target process and read that, you know, command line argument from there. And so that requires grabbing something called MFM. It's a read-wise semaphore. And it is also a major source of priority inversion because sometimes you end up issuing IOs while holding it. So if a low-priority IO holds it, it's just an idea, waits for low-priority IO, and high-priority comes in and runs PS, then it gets stuck and the whole system gets stuck with it. So it doesn't work. So those are a lot of those challenges. And there are a lot of others too, but those are the big ones. It works now, right? Okay. So, yeah, what do we do about all these things? There's a whole laundry list of problems. So first thing we started out with was to tackle the memory controller because that was the first thing we configured in the first place. And like TJ mentioned, we had to completely move away from hard partitioning that you get this, you get that. It just doesn't work. And instead switched to prioritizing instead the thing that has higher importance. And the advantage of doing this when you prioritize instead of partition is that if the higher-priority thing is not running, the lower-priority thing can just take all the resources it needs. And by definition, that's just a lot more forgiving to configure because the worst that can happen is there's too much competition. And there are no artificial um-kills. There's no artificial crashes or anything like this. If the configuration doesn't fully work, we can just adjust it and detect it while it's running. So for memory, the way this looked like, we implemented something called memory-low and memory.min, which memory-low is the primary thing that we use. So instead of limiting something, you just say the main workload gets this much memory. And if it needs, if it wants more, it has to just compete with everything else that's running. But it's a competition between different jobs and the higher-priority one gets a leg up in that competition. So memory-low is a best effort thing. So what we say is, unless there's severe memory pressure, the main workload gets more memory than everything else. And then if we're about to run out of memory and are threatened to bring down the entire machine, that's when we violate that guarantee and just go, okay, let's just make it work and don't crash. And then we also implement a memory-min, which is for certain jobs where we would actually prefer an oom kill rather than letting an application not work. For example, SSH, because if the machine doesn't respond to SSH, it gets rebooted forcibly.  So that's what we did for memory. Then we also kind of had no insight into how well our configuration was working. If you have multiple jobs sharing a machine, we actually didn't really know or weren't able to tell how much is the sharing of a host impacting the throughput and the latency of individual jobs. The kernel has a bunch of statistics and counters like the page fault rate and reclaim activity from which you could tell, okay, there's maybe a little bit of struggle in that workload, but it doesn't tell you how much longer do your operation if you're running shared versus if you have the whole host to yourself. And literally the only thing you could do would be to have the application or the workload run on a machine on its own first to establish a baseline performance and go like, okay, this is what the workload could do if it had the entirety of the resource available. And then you put it into a shared environment and compare it to that baseline, right? Now, that's extremely cumbersome. It's also almost impossible for a lot of our workloads because they do change all the time. We cannot really say this workload is always needing this much resources because our user activity changes right now. The US is waking up, so the workload is going to increase on our web servers and stuff like that. So that doesn't really work. And even if it did, the only thing it tells you, this workload is now taking longer to run if it's in a shared environment, but you cannot really tell the bottleneck is there, right? You can tell, okay, there's some latency here. This took longer to complete, but you only know the time. You don't really know why. So this is where a thing called PSI comes which is a feature redeveloped for the Linux scheduler. And what it does is it annotates all the points in the kernel where we have events that are associated with a lack of resource. So if the kernel enters a page reclaim, for example, we know we ran out of memory and we have to do some work that is not enhancing the workload, but just kind of making up for the lack of memory. There are other events like you're waiting for a busy CPU to become available, you're trying to run, everything is busy, so you kind of have to wait. So what PSI does is it annotates all these events and then aggregates what it's measuring into a share of total wall time or real time. So it gives you a percentage, so between 0 and 100%, that tells you how much of your overall runtime is the workload not productive because of the lack of resources. So if it, for example, reports 20% memory pressure, that means during 20% of the time that's elapsing, you're not actually doing work, you're just waiting for page faults to come back of recently evicted pages, you're waiting for a page reclaim. And that gives you a measure of how much productivity you're actually losing and it tells you what resource is culprit for that. So the whole thing works against a life system, so we don't have to do control plans or anything. So if we have a setup with multiple shared workloads, we can tell for every single workload at any given time if it's losing time right now on contended resources. So that obviously makes it a lot easier to configure our resource control configuration because we can instantly tell this thing has too little memory. This is waiting on CPU more than it should. And so this helped a lot in getting the basic configuration right. And we can also use something that I would call being functionally out of a resource. So this is kind of tricky. As Tee-Tee mentioned before, when the kernel umkiller kicks in, that's a very specific event, because you're trying to allocate a page, a piece of memory, and it tries to reclaim and it fails and it cannot allocate the single piece of memory. That's when the kernel goes, you're out of memory. But the thing is, even if you're at lower levels of memory pressure where this is not happening yet, you might already be functionally out of memory because you're spending 60% of the time waiting for memory. So you're losing pretty much most of your production capacity, but the kernel umkiller is like, well, you're still making progress. So what this helps with is with detecting when you're functionally out of a resource. It goes for CPU, it goes for memory, it goes for IO. And we can use this for several things. So one thing we did was for load shedding where you have a service that would just go like... The latencies are now too high for every single request and so I'm going to stop accepting requests and let some other machine handle it in order to avoid shinging and completely rebooting or hanging for a while. The other thing we do is... We developed an enforcer of that. This is a project called UMD. It started kind of out as a small Python script that would just go, if you're waiting more than X% on memory, I'm going to kill something. Now in the meantime, it's kind of developed into a much bigger thing. Daniel's here too, he's going to have a talk about that later. And it really does out of resource management for everything. It can monitor IO, it can monitor memory health, it can do all these things. So the thing this does is, as Tita mentioned before, the kernel room killer doesn't kick in for a long time, so it's not really helpful in managing our workload health. And the other thing is if it does kick in, the only thing it tries to do is keep the kernel running so it just picks the biggest task in the system and just shoots it in the head. Now the thing is our jobs are complicated, right? They're multi-processed things and they also have different priorities, and the kernel room killer has no understanding, it just shoots something and then moves on. And when this happens, we actually have no idea what state we're in. So we had a couple of services that could just not continue if the kernel room killer kicks in, so they basically enabled panic on Oom and just had to reboot. It's like if something died, I don't know what died. I'm the entire machine. So this is where OomD kicks in. The two things it does. First, distinction between when the kernel thinks you're out of memory and when we think we're functionally out of memory or functionally out of a resource where our production capacity is no longer adequate. That's completely workload-dependent, right? Some workloads might be fine, they're waiting 20% on memory. Some workloads say, I have a little bit more latency. This is not acceptable anymore. My SLAs are not met. So this is what you can configure OomD to do. First, you configure it. What is my trigger point here? What is my tolerance level for health? And if that's not met, then go do the other thing, which is kill something important. And again, something important is completely dependent on the system. It's dependent on the workload. And what that something is, it's also dependent on is that a single process or is this an entire C group you want to have taken out? So people try to do policy like this inside the kernel over the years. Repeated just doesn't really work to convey all this knowledge, convey what quality of service is to the kernel to convey what workloads are. And that's why OomD sits in user space because you can just configure that much easier. Now for the IO controller, as H.J. mentioned, we cannot really know the cost of IO. So if you can have a couple of IOPS already completely filling up the device, or you could have a couple of megabytes if it's really seekie and it's a lot of IOPS also kind of fully utilized device. It's hard to say in advance how much is this going to cost.  So it's hard to predict. So instead of trying to use these metrics like IOPS and bytes per second and all that stuff, what we do is we track completion latency. So every time you submit an IO request we just monitor how long it takes to complete. And then you can configure per segroup what your tolerance level is, right? So you can say if my IOPS take longer than 50 milliseconds, for example, then throttle everything that has a lower guarantee. So if there's somebody else that has a 70 milliseconds guarantee that thing's getting it throttled. Every time it tries to submit IO or memory that thing has got to wait. So again, this is work conserving. If the high priority thing is not running then the low priority thing can just use the device however it wants. And that works for both hard disk and SSDs. The other thing the IO controller does that we wrote is it supports things like, teacher mentioned, the metadata IO which is shared between all C groups and also something like swap. Because if you're doing memory reclaim and you want to allocate some memory and you have to reclaim something and you're deciding I need to swap out some memory, that memory might not be yours, right? So you cannot be throttled according to who owns that memory that you're swapping. Otherwise you might have the old priority inversions where you wait for a lower priority thing. So what this thing does is a concept called do first, pay later where if there is memory reclaim you're going full throttle using all the disk IO but then you charge whoever owned that memory that you swapped out. And when that guy tries to allocate later he gets throttled or when he submits more IO. So it's put on his budget but you're still moving at full speed to avoid that priority inversion. Which brings us to that. Yeah, that's a whole laundry list of things. There's really no magic bullet. We just kind of had to go through one by one and fix these things up. We switched to Butterfest to avoid the EXT4 journaling issue and then inside Butterfest there were a couple of things we had to untangle. For the MEPS semaphore first we did... The biggest offender was Readahead because every time you access a disk on file it doesn't just read one page the one thing you're looking at if you have an IOP you could do a lot more with a single IOP. So it just reads ahead a couple hundred K. And if you're really heavily under memory pressure it's not a good idea to try to allocate a couple hundred K and start IO when you're already maxed out on all the capacity. So we put a patch in to detect that situation and just abort Readahead and do page by page IO. That helped a lot but it doesn't fully cover all the situations in which we still kind of had these hangs. So Joseph who's not here is also at Facebook and I were working on completely avoiding any kind of IO under the MEPS semaphore which includes swap IO, page cache which includes things like writing to a file in the file system which has file system specific stuff. So it's kind of like slogging through all these things and patching it up. And then shared IOs as I mentioned do first pay later where if you're trying to reclaim it goes at full speed but if somebody else tries to allocate or try to create more potentially memory that needs to be swapped that guy gets throttled instead. Right so how does this look in practice for us? So the first thing we have to switch the EXT4 root file systems on our machines to Butterfest and that was actually kind of funny because we brought this up in the meeting that we have a priority inversion here and we should consider switching to Butterfest and the team that handles these things kind of just took it and run with it and after a couple weeks we had several hundred thousand machines converted to full on Butterfest, which made our Butterfest developers sweat a little bit but it's been pretty solid and we have a seven digit container image all on Butterfest and it's running pretty well. And yeah all the priority inversions are being addressed all the metadata is annotated and properly charged to handle these inversions. The other thing is we have swapped pretty much everywhere so some of our workloads they're like 80-90% anonymous memory and if you get into trouble with that there's not a lot of breathing room for the kernel if you don't have swap because all that stuff is basically M-lock that's everything you get from malloc that's tempFS if you can't manage these and compress them or get them out of memory you might be running okay in one second and you might completely hit the wall in the next second because the distance between being okay and being out of memory is very small if you can't manage memory. The other thing is there's always a competition between your file system caches and anonymous memory and if you can't swap then you might be thrashing your file system caches and constantly reload the same stuff while there might be anonymous memory that is completely unused and you could just swap it once and be fine and the system would run a lot better so having swapped enables us to do all this and makes this much better for memory utilization the other thing that's useful when you do this is as you get into pressure instead of hitting that wall it just kind of gracefully decreases the memory bandwidth and pressure builds up slowly until we hit what I mentioned earlier that our tolerance for production capacity so at some point umdi will notice oh you're spending a lot of time swapping that's something I recognize as a problem and just kill the unmanageable workload and this is really important because I think everybody remembers how horrible swap can be because if it's fine it's fine and then sometimes the machine just goes out for lunch for minutes or hours and umdi is really the key here because it can detect exactly when this happens and it gives us all the upsides of swap and kicks in when things go haywire so we have swapped enabled pretty much everywhere except for the main workload but that's only for right now because it really depends again what is the latency tolerance and right now the main workload's not really tolerating swap bio so the C-group setup is kind of like this we have three major hierarchies one is the workload itself obviously where the web server software sits then we have the system slice which holds all the package management and the remote control of the system the monitoring, the logging and then we have something that's kind of in between called the host critical slice which is neither the workload nor like that lower priority stuff because if those binaries die then the whole machine goes down something like SSHD we detect and reboot the machine if it goes out we also want reliable logging we want umdi to work at all times because that saves our ass constantly and so these are kind of setups the host critical is the only one that has a hard guarantee because we'd rather have umkiller go off on the host level than have any of these not be able to make forward progress then the workload gets the majority of the memory on these machines as a soft guarantee set by memory low and then you can see the IO latent guarantees workload and host critical have the highest priority and then system slice is below that because it's less important umdi we configure in a way that we protect primarily we protect the workload from what's going on in the lower priority system maintenance stuff and we also protect the system thing from itself so if there's any kind of if there's moderate like if there's mid-level pressure in the workload we kill something in the system if there's high pressure in the system we kill something in the system then we have triggers for IO as well just to make sure the workload gets the IO bandwidth needs in order to function and we monitor swap right now to make sure that if we run out of swap real quick then we also have to kill something yeah so these are the users that uh that uh we are seeing let me see oh yeah that's the first one so these are the hard drive machines web servers running you know fully saturated so this is 10 megabyte per second memory lake uh started in the you know in the management uh part of the system and um on the green line like the new one the fb text to protected one right it gets started three times you know ip drops a little bit but you know umdi kicks you know uh memory protection and all the patience is working so they are um protecting workload while punishing uh uh the the memory lake and then umdi eventually recognizes recognizes that uh you know the system is not in a healthy state and goes in and kills the memory uh memory bomb and um as a comparison right the base rps just drops the system just checks out right i mean that line is not you know that straight line is not number being numbers not being reported right the system completely checked out it stopped reporting anything so like the the graph thrower just you know through a straight straight line and the system got rebooted and you know recovered after like half an hour and this is a little bit faster memory lake this uh 50 megabyte per second about the same story but you know the tips are kind of deeper and the timeline more compressed but you know the same thing and this is 100 megabyte per second even deeper dips even faster timing but the same thing uh one thing really interesting in these graphs um i'm gonna go go to that uh later but is that uh the the top one right the the green and the purple ones are the rps the resulting performance of the system and the bottom graphs right uh these small things at the bottom um are memory pressure numbers um and it's a lot more clear in the ssd cases so i'm gonna get that uh get to that later um yes um and this is uh i o protection kicking in so i just uh we just started like um uh untarling a kernel package three times and touching them and removing them in the management portion so it's kind of simulating you know yum going haywire or something like that um and the interesting thing is that without the protection the dips are deeper i mean um but you know it also ends faster right uh the untarling the the bomb workload finishes faster because it consumes more uh resources that's uh uh that's expressed by those green bars green uh green and purple bars so green bar uh shows that on the protected host that bomb takes longer because you know its resources you know getting controlled but also you know they are the main workload suffers suffers less and uh one of the interesting thing here is that even you know even these are all protected they are still showing like these dips right especially on the um hundred megabyte one it's kind of clear i mean there's a fairly dip dip right and the reason for that is because this are uh nominally oversubscribed so we cannot protect it too hard because um system does like uh the system management actually needs to steal some higher capacity temporarily from workload for the system to stay healthy because you know it's just oversubscribed normally because that's more efficient so we cannot protect it kind of really strictly that's why you know you see those dips and going to the um SSD um it gets a lot better it's a kind of same same test but on SSDs and look if you look at the green lines you know they barely dip right I mean they still dip a little bit because the you know kind of needs to figure out what's working said and what's not but uh you know those are like small dips nobody cares nobody would even get paged over that right it's completely fine everybody can sleep peacefully um so you know basically same story right um unprotected the system just dips daily couple faster because SSDs are awesome but you know still we don't want that that kind of dip but uh look at you know this one is less clear look at this one so if you look at the bottom right um these are like really small peaks here and higher peaks there so one thing really interesting is that the blue line which has like this medium peak there's a memory pressure in system that slides so there's a the memory pressure that the memory bomb and the rest of management things are experiencing the the small like tiny bump here is the memory pressure that workload slice is experiencing so what what's happening is that uh uh memory controller kicked in and and made sure that you know um the workload is getting enough memory while punishing system that slice right so it's biasing the memory pressure that way to protect the workload if you look at this um the green and orange lines um if they are still you know a little bit different because they are you know their behavior is different but there's a system and workload for the unprotected system and um the pressure just kind of rises together and the system crashes right um now nothing is being protected there so yep that's the research um and um now that we have so this um it's this might not be you know as exciting as you know hey we succeeded at you know stacking multiple workloads and they will work perfectly we are not there yet but um this does demonstrate that we now can protect or isolate resources comprehensively while maintaining like full OS features um so um if you think about it right um if you can protect your main workload from system management stuff right nothing prevents you from like you know loading like a side workload like a side workload in the system that slides and you know the system would be just happy right because it's fully protected so that's um what we are going to try next um um pretty soon just keep sliding down um and so we're going to try to lose some batch workload uh uh you know next to our main workload and we'll try to make sure that you know workload is not disturbed at all so that would be our next step towards uh workload stacking uh with you know full dependency and eventually um we we still need some features especially on the iOS side but you know we will uh start experimenting with uh you know uh putting heterogeneous workloads on a single system while fully controlling how resources are used across them without using utilization so that's um what's our on our to-do list all right um and most of these features are already upstream some of them are not yet um but like in in in the next version next release or the one after that the kernel release um they should everything uh should have everything upstream so you guys can use it the same way we use it um and all of this uh is documented on this website you actually don't need the the the hashtag now we we drop that so you can go to i mean it still works with hashtag but it looks kinda worse um you can go to opensource.fb.com slash Linux and you know it lists like it has these icons um and you know there's a cgroup to psi bpf and all these things and then you know you can go to the website and there's a fairly good quality documentation so you can learn more details about them all right so that was it any questions we have a time okay


Live from Austin, Texas, it's theCUBE. Covering Dell EMC World 2016, brought to you by Dell EMC. Now, here are your hosts, Dave Vellante and Stu Miniman. We're back here at Dell World, Dell EMC World in Austin, Texas, 2016. This is theCUBE, the worldwide leader in live tech coverage. Andy Rhodes is here, he's the vice president and general manager for IoT at Dell EMC. Andy, good to see you, welcome to theCUBE. Great to be here, thanks. So IoT's hot, it's the next big wave, you guys are riding it, give us the update. Yeah, it's definitely at the top of the hype cycle right now, that's for sure. So we get a lot of calls and people ask, what is it? That's the first question, what is IoT and how do I buy some, how can I buy me some IoT, is there a skew for that? And I jest a little bit, but the reality is that it's really a series of use cases. Everything from the consumer wants that people know, like a Fitbit, which is monitoring you as an individual and taking sensor data off you and pumping it to the cloud, all the way through to things like predictive maintenance on heavy mining equipment, through to cold supply chain to energy demand response. So IoT is really just a series of use cases where our customers and the world is collecting data of things rather than just people and applications and then really analyzing that data for some sort of business value, whether that's driving efficiencies, if you're doing building management with smart lighting and smart air conditioning, you could be wanting to save electricity. If you're doing, the smart hotel room of the future, it could be for the customer experience, no keys and great environments. Or where it gets really exciting is when our customers are starting to use it to generate new business models and new revenue streams. And so a lot of them, it's all about this digital transformation. We have one customer who is in a building business and they used to build pumps and valves and thermostats. And now what they're doing is they're using some of our products and services to build a building appliance. They sell as a monthly fee and manages the building as a service. And then the next stage in their evolution is now, if they manage hundreds or thousands of buildings, they can compare those buildings together and they can say, hey, look, I got two strip malls in Houston, they're both retail. I can evaluate the use of energy in that building. I know it's because one of them has an old air conditioning unit. I can now sell that data to air conditioning resellers in the vicinity of those shopping malls. So that's where it gets super exciting is this business transformation effort. I want to test a premise because when you hear things like that type of transformation, you think disruption. And you're talking to a lot of customers. When you go into any retail organization, they've all got an Amazon war room, they're getting disrupted, they're getting Uberized, they're Airbnb, okay. The premise I want to ask you, put forth is, it feels like IoT is transformative, but not necessarily disruptive. It's the existing infrastructure players that are taking advantage of that. Is that a fair premise? What are you seeing? I've become known for this saying in the last 18 months of this job, which is it depends, right? So it's my new buzzword. It depends. You should become a consultant. It depends. And so I think that it can be disruptive. I mean, look at things like Uber and Airbnb, they've become verbs today. You've been Ubered or you've been Airbnb. Well, I mean, there's many out there. I find that massively disruptive on the business model side. But there's still this sort of transformation that is everybody else in the world. So if you're running a factory and you want to make it more efficient and smarter, then you don't have to throw away everything you've got to get there. Sometimes it's just unlocking these islands of data that already exist, but don't sit on an IP network. And so when you unlock those islands of data and then you can mash that data sets together, then you've got new use cases, you've got new ways to analyze this data and all of those sorts of things. That's more transformative. So I think there's both in the environment. Andy, can you drill down a little bit for us and where does Dell fit into this big ecosystem? The hardware you're selling, is it solutions, software, all the above? Yeah, so I think maybe I'm biased because of the company I work for, but behind every IoT project is an IT project. If you think about it, it's about extracting data and analyzing data. The first problem statement is much around connectedly unconnected. Most of the things in the world already exist. They are production lines, they're elevators, they're air conditioning units, they're lights. They exist out there. No one's going to throw them away just because they got religion about smart environments. And so the first job Dell does is connectedly unconnected. And that's why we set up a new division about 18 months ago around IoT where we're actually building what's known as an IoT gateway. And an IoT gateway basically connects to multiple different protocols. Whether it's old, serial, like RS485 or 232, or brand new stuff like ZigBee and Z-Wave and other technologies, we can ingest all of that data from multiple of those protocols. Second part is, well, how do you analyze and where do you analyze the data? So if you think about this notion that all data will go from sensor to cloud, to public cloud, that's a very consumer-esque mindset. And we know from hundreds of deployments now that that's never the case. And so, and not all the data has value all the time. And so you've got massive data sets. If you had to ship them back into the cloud on sort of cleanse, you'd have just this massive data. It's not going to happen. Yeah, no way. Our CTO, David Floyer, has actually written extensively about this because data has gravity and just the sure physics of what you're talking about. There has to be a lot of edge computing in what happens with architecture. Yeah, and this is predicated on the fact that 82.7% of statistics are made up on the spot, but I'd say that 90% of that data is useless in its raw form. So I see the gateway as one piece of the architecture and it's almost like the spam filter to the cloud in an IOC setting, right? Think about it, it's cleansing data on the way, extracting the valuable data. The second part is you do have a lot of data back in the branch or in the data center and you do need to do big data analytics on that. And then Dell, especially now with Dell EMC, we have massive of capability to store and analyze big, big data sets. So we're the underlying infrastructure provider on all of this, and then we have great partnerships with platform vendors, whether that's Microsoft and Azure or SAP and their platforms or PTC with ThinkWorks.  And so partnerships and a really strong ecosystem base are absolutely critical. So where we don't have the capabilities, we partner, and then we bring all of that together in what I call use case blueprints. So here's how you do predictive maintenance on heavy equipment. Here's how you do cold supply chain with perishable goods. Here's how you do automated energy demand response in factory environment. So I think that's what the world needs is more of these how-to to a level of specificity but not all the way down because we don't know what sensors they're using, what data they're collecting and every one of those use cases.  And give us some examples. What do some of those solutions look like? Well, predictive maintenance is a great example. So you look at people that are operating heavy equipment, then what they tend to do is they have to maintain that. They're doing it right now on a time stamp basis. Imagine mining equipment in Calguli in Australia, so a big mining area. While you're in the middle of the desert, you've got heavy equipment, maybe shake and bake, separating rocks from other rocks. That stuff breaks all the time. A lot of those manufacturing equipment providers today, they will go service that every three months, whether it needs fixing or not. You then wire all of that up with sensors and you can start to extract the data from that and then do sort of predictive maintenance algorithms and you know when it's going to break and then you send a man or a lady in a van or a helicopter in many of these cases at the right time, not just on time intervals, and you can save a heck of a lot of money on your own bottom line because you're not shipping people around the world all the time and you can keep your customers up on whatever their environments are. So that's a great use case around predictive maintenance. You were talking earlier off camera about the cultural issues. Talk about OT versus IT, the operations folks versus the IT folks. Do they talk? Should they talk? How do they converse? What's that cultural divide and how is it coming together? Yeah, this is actually one of my, the cultural side of this is my biggest learning and one of my favorite topics because very quickly you realize the tech is there. So the tech's not that hard. Well, don't do it yourself, we can do it for you but it's not that hard. But it's available today. It's available and it's commoditizing and the price is there. And it's not a mystery anymore, right? It's not a mystery. Now, connecting some of these things with the non-IT protocols is tough but the industry's solving that problem. No, the OT and IT, OT by the way is operations technology. OT people don't call themselves. You won't find that on a business card. They'll be called manufacturing VP or head of logistics or fleet management. Facilities. Facilities only, yeah. And their world is much more of a business world. I mean, they are the business. Their metrics are the yield of my factory, the yield of my agro crop, their, the energy efficiency in my building, they're the uptime of my manufacturing process. They're very different metrics where if you look at IT, their metrics are more around security and scale and cost of deployment. And so these two people have to come together because my view is once data hits the IP network, it is the CIO's responsibility to secure that data, to make sure that it gets used or thrown away in the right way. And the CIO, she or he, is absolutely responsible for it. But it's useless on its own, right? Data stored is a cost center, data analyzed as a profit center. And that's where the OT people have to come back in and actually make use of this data. They're the ones that say, hey, I'm going to use this data in a certain way, but IT already have big data, you know, analytics systems. They already have sensors of excellence. So what we've seen in the most unsuccessful deployments is where IT kind of sit around saying, I need to have an IOT project. I don't know what that is, but I need one. Or OT say, I'm going to go do it on my own. They have to come together, work together, understand what the use case is first, understand what the value proposition is and what the business value is, whether it's new revenue stream or efficiencies, and then they have to work it together. So that's number one in the cultural interlog. Then if you think about, you know, a lot of customers are using this to change what they sell and how they sell. So IT and OT can come together and create a new product. But if your sales people are trained on how to sell maybe an OPEX model versus a CAPEX model and your marketing team haven't caught up or your CFO says, I don't know how to count that revenue or I'm going to see a revenue dip for two quarters, you know, how do I explain that to the street? So I'm now seeing that successful IOT projects start with the use case, have OT and IT come together and then bring in the rest of the business functions. Andy, one of the biggest topics we've been talking about recently has been security. So, you know, what do you say to people that say, well, IOT is going to greatly increase the surface area. We've got all these news protocols. It's highly insecure. Oh my God, just stop. Yeah, I mean, look, the first thing I'd say is that's absolutely, you know, getting paranoid about security is the best thing you can do. So getting paranoid to the point of not doing anything is also not the right thing to do. And I think that the industry is coming together to help solve a lot of those problems.  Using a gateway is a great example of starting to bring all the data into one device that you can secure. You can lock down the ports. You can manage it like a server. You can put, you know, network security. You can put encryption on the data. You can do all of these things and it's in the IT's domain and they know how to do that. Now, you also have to partner with some of the OT security vendors. GE, as an example, has got some great assets around OT security with WellTech and that whole portfolio. So I think like anything on security, it's not one thing. You know, it's not one security solution. You have to do it at the things level. You have to do it between the connection of the things and the gateway. You have to secure the gateway. You have to secure the data going from the gateway into the cloud. You have to like just deploy good network security protocols as well. You know, one of the ways that you help security is not do dumb stuff to begin with.  So again, I think it's coming to a company like Dell who understand all of those aspects and bring in the right partners or our own assets to help customers. But you know what's interesting there, Stu, is the model, we always talk about it's not about the perimeter. It's not about building the mode. It's about, you're going to get infiltrated. It's about using analytics to find out who, and what to do about it, how you respond. Now, while that sounds good, a lot of organizations still put most of their investment in the perimeter. With IoT, there is no perimeter, right? I mean, there's a series of perimeters, right? And so, and I feel like Stuxnet changed. I mean, that's a good example of an IoT worm, if you will. That sort of changed the mindset. So, are you finding that customers in your world are more tuned to this reality? And are able to better respond, or are they not as security savvy as say, traditionally in paranoid as the traditional IT folks? I'd say that most customers, whether they do IoT or they do traditional IT, given the high profile of attacks, given the catastrophic business losses, the consequence, I'd say that 95% of the customers I speak to, it's one of the top things in the dialogue, is how do we secure the stuff, and how do we manage the data, how do we secure the data? I haven't seen a lot of them paralyzed by it, to be honest. I think they realize- It's not, it's safe. No, I have not. That's good. I think they realize that the advantages of going off and putting these use cases to work in their business, and the fact that the world is helping with the security issue, just look around this show and you can see a lot of security providers. I think that they aren't paralyzed by it, but they remain very, very paranoid. And it's the mindset that they need to have and continue to have, because the last thing that they want is an attack that will stifle the first or second IoT use case project they put in place, and then it all sort of breaks down. Right. Andy, we talked about how all the data doesn't necessarily go to the cloud, we have the edge component, but what about the ties to the cloud? Dell, for the most part, does public cloud through partnerships, so how does the public cloud fit into the overall IoT story for Dell? Yeah, I think Michael articulated this, and I hate to paraphrase him incorrectly, but I think he said that the cloud computing is a way that you do computing, not a destination, right? So when I say the cloud, I think it depends on what the customer's requirements are. Some of them just, they absolutely cannot have their data go off-premise for security reasons or latency reasons. Some are fine with that. Some it's for national boundaries, and so we have a very open approach to that architecture. If you want to go and do computing in the public cloud, then we have partnerships to do that. If you want to go and do hybrid and put some of it on-premise in what's now called the fog, we have a new term, it's the fog, so you have the edge, which is the gateway, the fog, which is sort of the branch or the middle, and then the cloud, we can do it that way. I come back to this terrible answer of it depends, and it's use case specific, and it's company specific, and so if you're only a cloud vendor, guess what, everything looks like a cloud. If you only build edge gateways, guess what, everything looks like an edge gateway. If you only build the stuff in the middle, that's what you sell. The beauty of Dell EMC is that we have that whole portfolio, and we can actually sit down and advise customers based on their needs of what is the best solution for them, and it sounds like a sales pitch, but I firmly believe that, and if you go speak to our customers, that independence of architecture is hugely valued. It's all open. Now, we're always going to recommend that you buy it all from us and that our stuff works better together, but if you want to uncouple it, then you can uncouple it because it's based on open standards. And it's a great topic. We got to leave it there. Thanks very much for coming to theCUBE. Really appreciate it. Great hanging out, great chatting. All right. Keep right there, buddy. We'll be back with our next guest. This is theCUBE, we're live from Austin. Right back.


One of the most common questions I get is what happened to your foot once they amputated it? What if I could answer it better than with just words?  I definitely stole Shane Dawson's intro because I've been watching his conspiracy theory videos all day when I came across this piece of information This is not a conspiracy theory video, but I feel like it's just Creepy enough that it falls into that category. Maybe not I could be way off base anyways I have been combing through medical records all day and I am way too excited about this I fully recognize that I will admit to that right off the bat But I came across this little tidbit of information while going through medical records I have an appointment set up with a new surgeon this Wednesday, so tomorrow and He needs my surgical records I have never bothered to look through my surgical files because why would I until today and as I was combing through them I found something I thought you guys might be interested in now real quick if you are grossed out easily or even not that easily Just brace yourself. I guess this kind of makes me feel a little bit queasy But the same time it's so fascinating to me that I kind of power through it One of the most common questions I get is what happened to your foot once they amputated it because you know This situation is going on where did it go like what do they do with amputated legs now? I could just answer that question for you But what if I could answer it better than with just words that is what this document made me think could possibly be Possible now. I doubt that that will actually happen, but I'll just let you read it. All right here. We go received in a biohazard bag without fixative label without fixative labeled Don't you think it should have a label? I don't know anyway designated right below the knee amputation is a below the knee amputation of the Right leg. I'm glad that that's what they had in the bag It measures 23 centimeters from distal toe tip to heel 20 centimeters from heel to interior soft tissue margin and 13 centimeters from heel to posterior soft tissue margin Here's the part that gets a little gross though again is Really interesting 20 centimeters of tibial bone extends proximal from the interior soft tissue margin Essentially meaning that there was like a piece of bone sticking out from the rest of it Which makes sense because they want to keep a lot of that tissue on my body not sawed off So they cut the bone off, but then they're gonna like keep the tissue and fold it up Which is now here because like the bones here, and I'm not doing a good job explain this But do you get it like they they fold in my calf underneath? It makes sense the bone would stick out of the amputated part But that's really gross to think about just sitting there in a bag The skin soft tissue and bone at the margin appear visible or five toes That's good all with intact nails the skin over the dorsal foot and interior leg contains writing a blank ink Yes, because as you guys know if you watch my channel my friends and I and my parents Had a goodbye to my foot little ceremony where people wrote goodbye messages to it So I'm glad that they noted that that tells me that it was indeed my foot They wrote this message about here's the part that is really interesting to me because I know this exists Somewhere gross photographs are taken there are pictures of this Somewhere so many people have asked me like did I get to take a picture with my foot? No, because I was unconscious and they were doing the amputation or did I get to keep it? No, because there wouldn't really be a reason for me to some people can keep their legs for Religious reasons something like that. There are also stories of people who have kept them Just because I'm not really sure under what cause but this one guy had a foot party And made tacos for his friends out of his foot meat. That's really gross I would never do that to you guys, but also kudos to him if his friends knew what was going on Because you can't make your friends eat your foot meat with I'm just gonna stop right there This is really gross photographs of my foot were taken well after it was amputated Which means those photographs are somewhere those photographs are not in my medical records Which honestly I'm grateful for because I really would not like to have come upon those unexpectedly Yeah, because that would be very disturbing. I honestly wonder if I could request those I don't know if I would put those up because I feel like that'd be super gross But also I feel like a lot of people would be really interested to see them and you all have been a part of this journey so far already I don't know that might be a little too far that might be a little too up close and personal I'm guessing it probably would be but at the same time the internet was interested in everything. I Don't know maybe I'm just in a weird frame of mind today guys But this was so Interesting to me because you take photographs for a purpose right so they have to be saved somewhere I wonder if there's just like a whole file of like photographs of amputated limbs sitting on someone's computer Up at the hospital. I'm sure that there is that is a weird thing to think about I would love to get access to the hospital and take you guys on like a tour of where Amputated limbs go because honestly that is probably the most common Question I've gotten on the videos that have like gotten millions of views people want to know what happens with it Because that's something that you don't really think about until you're like, oh wait someone's an amputee in surgery What happened to the limb what happened to the foot or what happened to the hand?  I would be curious. I was curious I know this is not at all related to my foot, but when I woke up from surgery I told you guys that I was in a lot of pain because the nerve block didn't work So basically I felt everything when I woke up from my amputation surgery I got to see the records of that too like all the pain medication. They gave me holy crap, they gave me a lot of fentanyl and Hydro-morphone, which is delotted and ketamine in a very short period of time trying to get that under control Like I knew that because they were worried that I was gonna stop breathing and Brian was really freaked out But also it's weird seeing it all written down and kind of having some understanding of How much that is so it's been a weird day of watching Shane's conspiracy theory videos and then realizing that there are photographs of my severed foot Sitting somewhere in the internet system of the hospital slash someone's computer that maybe I could get some day Do I want to get those does the internet want to see those would I show them questions to be answered on another day? I am a little nervous to ask but I'm curious if Anyone would want to see those obviously I would put it out with lots of warning But is that something that is interesting to people again? I'm not promising that I would do it if I could find them because that might be a little too personal I have a feeling it might be too person maybe I wouldn't want to see it Maybe that'd be a little too gross, but then again it is my foot So I don't know anyways. I'm gonna go back to watching Shane Dawson conspiracy videos. Shane. You're fantastic Thank you for entertaining those of us stuck at home. I will talk to you guys soon I look forward to your comments on this video your queasy. I apologize for making your stomach churn I too feel a little queasy knowing that there are photographs and that the bone was sticking at the top But still it's really interesting. All right guys, that is it for me. I love you because I care of yourselves I'll talk to you soon


Alright, so let's take a look at finding the areas of regions using what's called a Riemann sum. Important pronunciation note, this word, this is actually the name of a 19th century German mathematician whose, because he was German, pronounced his name Riemann. This is not pronounced Riemann. If you pronounce it Riemann, everybody will know that you didn't learn it from somebody who knew what they were talking about. So the basic problem here is we want to find the area of a region, and so here's a general approach to finding that area. So here I have some regions. So there's my top, bottom, left, right. So it's a bounded region. I can talk about what the area is. And so a typical approach is something like the following.  I'm going to, in each of these pieces, I'm going to draw the largest rectangle that fits inside the sub-region. But here I'm drawing the rectangles that fit inside these sub-regions that I've formed by my partition. Why draw rectangles? Well, the reason is that I know how to find the area of a rectangle, and so I'll find the areas of all those rectangles. And when I add up those rectangles, when I sum those rectangles, the sum of the areas of the rectangles will be less than the actual area of the region. And so in other words, this sum of areas gives me an approximation to the area. And because it's less than the actual area, it's what I'm going to call the lower sum or the lower Riemann sum. And likewise, I can find the upper sum. And it's not going to be too much different. I can find the upper Riemann sum in almost exactly the same way.  This time I'm going to draw the smallest rectangle that contains the sub-region. So imagine that you're shipping these pieces. You want to find a box that fits. So I'm going to find the smallest box that I can use to ship the pieces, and those boxes might look something like that. And again, these are rectangles, so I know how to find the area of a rectangle. And again, the significance here is the sum of the areas of the rectangles is going to be greater than the actual area of the region. And so I get a second approximation, the Riemann upper sum. Well, what do I know? The area of the region is less than the upper sum, but it's greater than the lower sum. So the actual area of the region will be somewhere between them. Now there's an important qualifier here, which is that we have to have a region that actually has an area. And that's a topic that you will discuss at a later point in mathematics. For right now we'll assume that we have areas, we have regions that actually have something that we could call an area. And in that case, we know that the actual area is going to be someplace between the upper sum, which is a little bit larger than the region, and the lower sum, which will be a little bit smaller than the region. Well, let's find it. So let's find a Riemann sum. Let's approximate the area that's bounded by y equals x squared in the x-axis over the interval 1, 2, 7. And for variety, we'll use the upper sum using n equals 3 subregions. And we'll also find the lower sum over the same subregions. So let's go ahead and sketch what that region looks like. So this is y equals x squared under y equals x squared above the x-axis from x equals 1 to x equals 7. There's top, bottom, left, right. We have a bounded region, so I can at least ask the question, what is the area? Now I want to use the upper sum with n equals 3 subregions. So I want to divide this region into three pieces. So I'm going to divide it out of how about here and here. So I need what are called partition points. So maybe my first partition, I'll occur it, how about x equals 2? That's a nice number. And then I need a second partition point someplace between 2 and 7. How about x equals pi over the square root of 11 times, how about x equals 5, just to make our lives easier? So we'll partition the region at x equals 2, x equals 5, and to find the upper sum, I'm going to find the smallest rectangle that contains each of the subregions. And again, the idea to think about is I want to ship this piece here, this piece in the first subregion. What's the smallest box that I can use? Likewise, to ship this piece, what's the smallest box I can use? And likewise for that piece. So what do those boxes look like? They look a little bit like that. So now I need to know what the areas of these rectangles are. Well, how do you find the area of a rectangle? Well, I need to know the height of the rectangle and I need to know the width of the rectangle. And if I knew both of those, I could find the area. So the first thing to notice is that these rectangles will generally have one vertex that's on the curve. So here, this very first rectangle has a vertex that's right here at this point on the curve of y equals x squared. So if only I knew the coordinates of that point. If only I knew what the x value was at that point. Oh, here it is, x equals 2. I know that x equals 2, y equals x squared. So that tells me that this point right here has coordinates 2 comma 4. And so there's the vertex of that first rectangle. There's one of the vertices of that rectangle. And so while the y value is 4, so that tells me the height distance above the x axis is going to be 4. And the width, this triangle, this rectangle goes from x equals 1 up to x equals 2. So since it extends from x equals 1 to x equals 2, the width is going to be end minus beginning. That's going to have width 1. And what's the area of a rectangle with height 4, width 1, that area is going to be 4. And I'll go ahead and mark that down on my first rectangle. Well, that worked out pretty well. Let's take a look at our second rectangle. So that's going to be a rectangle that covers this region here. And the top point of that rectangle is going to be right here at this point, which is on the graph of y equals x squared at the point where x is equal to 5. So that vertex is going to be located at 525. And my second rectangle will look something like this. Height is 25. Width is, I'm going from x equals 2 to x equals 3. The width is going to be 5 minus 2, 3. And so I have a rectangle with height 25, width 3, in the area 75. And finally, that last rectangle. So that last rectangle is going to have the vertex right up here on y equals x squared at the point where x is equal to 7. So that vertex is going to be the point 749. And the area of that rectangle, height is 49. Width is 7 minus 5. And so the area length times width 98. That's the area of the last rectangle. And if I sum up those areas, I will get an approximation to the area of the region. So 4 plus 75 plus 98 is 177. And that will be my upper sum, which again, is going to be a little bit larger than the actual area of the region. Well, how about the lower sum? And for the lower sum, I'm going to do almost exactly the same thing. I do want to use the same subregions, but this time I'm going to find the largest rectangles that can fit into each of those subregions. So those rectangles look something like that, and I go through the same process. So that very first rectangle, the vertex occurs at a point on y equals x squared, where x is equal to 1. So I know the coordinates of that first point, I know the height of that rectangle. And I know the width of that rectangle, so I know the area of the rectangle. Likewise, second rectangle, the vertex is going to be at a point on the graph of y equals x squared, where x is equal to 2. And I know the coordinates, so that gives me the height. I know the width, and so I know the area. And then that third rectangle, the vertex, is going to be at this point on the graph y equals x squared, where x equals 5. So that tells me what the coordinates are. I know the height, I know the width, I know the area. And the sum of these areas is going to be my lower sum. And a nice picture to keep in mind. I am interested in the area of the region. So here's the area that I want. Well, it's going to be greater than the lower sum. Again, these rectangles that form the lower sum are all completely contained by the region. So whatever this area is, it'll be less than the area of the region I want. And equivalently, also, I'm also going to have this area to be less than the upper sum. Because these rectangles that form the upper sum are larger than the region. They contain the region, and then a little bit more. And because I know those two numbers, the area of the region itself is going to be some place between 63 and 177. Now that leaves a fairly large region interval of possible values for the area. But we'll see if we can make that approximation more precise in the next video.


Similar to in classification we can use the logs to evaluate how model parameters change. So we say have our model dot make inspector training logs. I'm just going to immediately turn that into the data frame where we extract the number that should be quotes. The number of trees, and I'm once again going to do that sort of for loop where I say log dot num trees for every log in logs. And this time I want the root mean squared error. So I want log that valuation. That rooming squared error, or every log in logs. And so here we can see the number of trees, and we can see how the rooming squared error is. So if we plot this logs df with a line plot. Our x value is the number of trees. And our y value is the root mean squared error. We can see that as we increase the number of trees, our root mean squared error decreases. And eventually a plateaus. And so once again, similar to the classification analysis, we can use this to see if maybe we need more trees or less trees. And in this case, it seems like 200 might actually be a decent amount of trees. Perhaps an argument could be made that we should drop that down to 150 if we're trying to improve the speed of our model. But it certainly doesn't seem like we need any more trees in order to gain any more accuracy in our model.


Good afternoon. I have with me today Mr Rupert McPetery, CEO AIPAC at EssenceMedia.com. Thank you so much Rupert for joining us. A real pleasure and great to have the opportunity to talk about the agency and can and the future. You took over as CEO in May last year in the midst of the merger process at EssenceMedia.com. What were the big challenges that you faced in the run-up to this merger, which finally concluded in January this year? It's been an extraordinary journey bringing the two businesses together. I think we set out to create something very, very new and because of that we wanted to take our time to do it properly. So although there were challenges, some of them around logistics, some of them around how we set up our organizational design, some of them around keeping up with what our day-to-day business is whilst trying to imagine the new agency. We were very lucky that we took the time that we did because that allowed continuity of business, but also to manage any small challenges here and there. And I'd say probably the biggest challenge that we had was trying to find the balance between taking the time to do things properly with full engagement, participation from our people, keeping all our clients up to date, so taking some real time and care about what we're doing. But at the same time, we don't want to lose any time. We wanted to move quickly, move decisively. And of course the industry, the market, our clients and our people were very curious about what will Essence Media.com look like. So the only real tension we had was that finding a balance of moving fast versus taking the time to take care of everything properly. So what have you achieved in the short term and what's the long-term agenda? So when we launched formally, Essence Media.com, the new agency in January, we felt we'd already achieved quite a lot. So we have a new vision, a new proposition, we have a new way of working with a new operational structure. We've integrated our people into one team. We're still in the process of moving some people into co-location. So that's sort of step-by-step getting the estate sorted. But I think what's really heartening for all of us, what we take really positive is the feedback we're getting from the industry is very positive. So if you ask what we've achieved, I think we've positioned ourselves well as the agency for the future to help clients navigate the new communication economy, to elevate our work, to take it upstream, to deliver what we call breakthrough work. And it's working. It's working well. And we've had many proof of concepts when we've been pitching of how relevant our new offer is, how attractive our new offer is. Because we're looking to go beyond just the sort of simple find an extra half of 1% advantage here or tune up a strategy there. We're after something much richer and fundamental that will genuinely change the game, create a breakthrough for our clients' businesses. So we've achieved a lot. Longer term, honestly, there's still a lot to do. We're at the very start of this journey. It's only 155 days since we launched. So whilst I'm fiercely proud of what our people, what we call essentials have achieved so far, there's a lot to do as we roll out our new vision and proposition, the tools, the technologies, the capabilities, and we scale these across the region and in market. So we're proud of what we've achieved, but we're humble enough to know we've got a huge amount to do. You use the term new age agency. How do you define that? Well, look, I think we've been very fortunate in taking the brilliant capabilities. And by that, I mean people as well as the tools and the tech of Mediacom and of Essence. You know, you've got a Mediacom, a classic, strong agency model with a really diverse range of clients, very strong in communications and strap planning, brand building activities, you've got Essence with data analytics at the heart, performance, digital centric. So the raw ingredients of what we have, I think are quite unique. And in our new proposition, what we're trying to solve for and why these people capabilities are so important is really trying to address what we see clients are asking for now. And this is no longer really just optimizing individual silos of activity. Of course, we're trying to do the best we can in every platform, every media, every plan. But clients these days, they want to understand the co-relationship, the co-dependencies, social and commerce of content to data analytics, as well as classic brand building activity. And so that's the agency of the future that we're building a new agency to navigate what we call this new communications economy. And critically, I think quite unique for us is we're able to solve for a whole broad range of questions that clients are asking in one go. Okay. So looking just at the agency business in the AIPAC region, what has been the client's response to the merger? And what are you looking forward to the year ahead in terms of growth? So clients in AIPAC, I think have responded really well. The feedback's been very positive. I think what we're standing up in the region from a capability point of view, really does solve for many of the key business challenges. So I think we really want to be clear to our clients that the breakthrough work that we deliver, it's not just nice work. It has a measurable impact on their business or on the KPI that they sell. So I think we're in a really, really good place to deliver on that. The feedback's been positive. I think there's also a really, really high level of curiosity about what it means, what it means for their brands, what it means for their businesses. So since we launched, we've been working with all our clients on very bespoke, specific application of some of our new technologies, some of our new tools, or even bringing in different profile of talent from the broader agency. You can imagine how powerful it is, let's say, for one of our former media clients to get exposed to some of the talent profiles that come from the former essence side of the business, and vice versa. I think marketers these days are just so curious about not just what their direct competitors are doing, but what adjacent sectors, adjacent categories, and other businesses are doing. The more we can bring different profiles of talent, diverse talent and capability to bear, the better. So we're working on these plans for all our clients to introduce some where irrelevant, where it's going to make a meaningful difference, some of these capabilities. You've preempted my question, my next question. I was going to be asking you, how has the talent profile changed in the organization, and what would you define the culture of the organization today? So I think from a talent point of view, the profile of talent is changing quite quickly now. I think for us, we're obviously still maintaining that sort of classic media capability, but where we make a difference for clients' businesses, and therefore the type of profile of people that we want to have in essence media is in more upstream areas. So I think we're seeing a higher proportion of our talent pool being really well equipped in areas like data analytics, digital and data consulting, commerce, social content, on top of that classic media planning and buying. So it's changing quite quickly, and we're driving this very intentionally, because we think that's the profile of talent that we want as essentials to deliver on our proposition. That's what clients needs, what clients want. And that comes in two areas. Number one is how we educate the learning and development programs that we have for all our essentials in Asia Pacific. So we're rolling out multiple programs right now, very bespoke for individuals to help keep people on topic, on trend, and really at the forefront of the capabilities that they need. But it's also a question of when we're out in the market hiring, we're probably looking for a slightly different profile of talent than perhaps we would in the past. What is this different profile? Well, I think it's absolutely about the specialist expertise in some of these core areas. So of course, we still need classic account managers, account planners, you know, the buyers that will activate campaigns. But we also need data specialists, we need people who are very skilled on e-commerce platforms and marketplaces, people who are very comfortable understanding the role of content, be that, you know, original IP or DCO on a client's schedule. So it's a very broad spectrum of talent. So increasingly specialist talent, increasingly upstream, but honestly, increasingly higher value to customers. A word on the Indian leadership? Brilliant. Okay. Now this merger has also created scale for Essence MediaCon, particularly in India, where it's now among the top three agencies in the country. What are your plans for the market? The Indian market, I think, is terrific for us because we've got a very strong Essence MediaCon, we've got a very strong GroupM and a very strong WPP. So there's extraordinary infrastructure in the market from a talent and a resource and capability point of view. So post merger, I think we're in a really, really strong position. We've got a very strong overall leadership. We've got a client list, but I'd say is the envy of everybody. So what we're doing now is we're just really concentrating on elevating the work, just really making sure that our promise, our commitment to clients around finding breakthroughs for their brand, breakthroughs for their business, we actually deliver them. So it's a bit sort of heads down. Let's concentrate on the work. Of course, we do want to grow, but I think the growth will be a function of our capabilities and the work that we do. Having said that, we've also had a really, really strong year from a new business point of view, year to date in India. So I think we've got a high level of confidence in the Indian market and our place in that market. And we're looking with a level of optimism for the future. So with this increased scale, are you looking to add on more new clients in the next six months? Any pitching in the work? Any pitches in the work? Yes, absolutely. I'm a little bit greedy, to be honest. So whilst we've had very, very good progress at the start of the year, with an extension on our P&G assignment. And by the way, we won an amazing award at the Cannes Lion last night at Bronze for P&G Whisper, the missing chapter, which again is testament to the quality of the work that we're doing for clients in India. So the extension on P&G, we won the consolidated Vivo business. So we've got good first half year momentum on new business. We've got quite an interesting pipeline, but of course, what we want to do is make sure we take care of current clients, do breakthrough work for current clients and then step by step grow through the acquisition of new clients. So again, we feel optimistic. We've got a plan. And I know with Navin leading the business there, we're in good shape. You mentioned quality of work and the quality of work goes back to the talent in the country. So how do you assess the Indian talent? From my experience, I've been running Asia Pacific for just over a year now. And I think that talent we have in the Indian market is world class. I really do. And that P&G can win is a great example. And that work was also rewarded and recognized by war. So I think the work is coming through. And that's a function of the talent, as you say. We're increasingly investing a lot in the learning and development of our people in India so we can deliver on the proposition. But I think the raw base of talent is extraordinary. And if you imagine that breadth of talent I was talking about earlier, the different specialist capabilities around data, tools, analytics, insights, commerce, social content, that's all there in India. Our job is to make sure everyone has the tools and the capabilities that they need to thrive. So that's the learning and development piece. And the way we operate empowers and enables them. So I feel very confident in the talent that we have in India. And one of the things we're very positive about from an essence media con point of view is giving all our essentials, all our people, mobility opportunities. So we'll be seeing more and more talent exchanges, mobility movements, just to give people that career building opportunity. How important is cross learning across the region then? It's huge. I think it's huge. I think one of the challenges agencies have is sometimes we try and reinvent the wheel in different parts of the world or on different clients even in the same market. So one of the initiatives we've been driving over the last year is to really share and reapply work, share and reapply best practices more quickly, more practically around the region. So sometimes that's, you know, in a leadership forum, sometimes that's in town halls, sometimes that's in very specific practice based sharing sessions. And sometimes it's very client specific, just so that we're moving best practice and information and ideas around the organization as fast as we can. The question which I was actually really curious to ask you is, you're a big proponent of diversity, equity and inclusion in the workplace. Why should the DEI be at the forefront today? For me, DEI and I is just so fundamental to how we go about our business. And for me, the reason is really, really simple. It genuinely matters. It makes a difference and it matters to our people. And I think sometimes we talk about our talent and our teams. So they're one. They're not. They're 2,500 individual essentials in the region. And we need to respect all of their individuality. We need to create an environment where they're all comfortable. Everyone is comfortable to be their authentic selves at work. And it matters to our people. It matters to our clients. So many clients now in a review and a tender will want to understand for real. So not just some nice words, but for real, what are we doing for our people in the DEI space? And beyond that, and I take this really, really seriously, it's the right thing to do for the communities that we operate in. I think as a leading agency in the world's largest marketing services organization, we've got a responsibility. We have a genuine responsibility through our work to drive change. Everybody's talking about DEI now. But what was really interesting is that you are Essent Media Comm's Fate Diversity Lead. Can you tell us more about this fate in inclusivity? Because you also operate in an area which is extremely diverse. So can you just tell us something about this? Yeah, sure. And this is a topic very close to my heart, as you'll understand. When we set up, if you like the founding principles of Essent Media Comm, we did want to ensure that DEI was at the very heart of who we are and what we want to be. And to that end, in addition to having some really specific policies to ensure that people can be their authentic selves at work, we think there's a really important point around ownership and accountability and responsibility. So in our global leadership team, every one of us is responsible for advocating a strand within DEI. And I thought given the diversity of Asia Pacific, the really wide range of beliefs and thinking, faith, that it would probably be the right thing for me as the APAC CEO to advocate for. Because it just comes down to the fact that many, many, I guess, views on DEI tend to be on what I call more traditional areas of it. Is it race, ethnicity, social class, et cetera. And faith is often the forgotten, the forgotten piece, right? And I think, again, when we look at DEI, I'm a big believer in this intersectionality, because you could be many parts of DEI. My core beliefs could straddle a whole range of things around gender, ethnicity, social class, neurodiversity, as well as faith. So I didn't want faith to be left behind, particularly given its significance in our region. So we're at the start of this journey of really trying to advocate to ensure that faith is not the left behind strand of DEI. This was so interesting because every, like I mentioned, everybody's talking about DEI, everybody's talking about LGBTQ, but nobody, very few, or I haven't heard many people talking about faith. Exactly. It is the forgotten thing. And we're doing a lot of research as we're sort of preparing our founding principle. And there's a lot of great work being done by many businesses around DEI. But if you look at DEI in company reports, for example, it's very rare to see faith covered, very rare. So we felt it had to be an integral part of what we're doing for our people. I think we have to show equal respect and create equal space in faith, as in other areas of DEI. It's that simple, I guess. So I'm really proud to be advocating for this faith element of DEI. So we're just trying to, you know, one, raise some awareness around it to make sure we're creating that space so that people can be their authentic selves. The way to drive that, as with everything, is that someone owns it, someone's accountable for it. And that's me. Finally, there's a lot of talk about recession and uncertainty about the economic scenario today around us. What is your assessment of the APAC region? And are there any green shoots that you're seeing? So APAC is, of course, vulnerable to the global headwinds, global ups and downs.  Some of the consequence of that is seen in interest rates to try and manage inflation. So there are concerns, of course. And I think in various parts of the region, markets will have their own specifics, you know, that perhaps accelerate that or slow it down or add or detract from it. When I look at the region as a whole, I remain still cautiously optimistic for 2023. I think, you know, relative to some of the other parts of the world, we still have significant opportunity in some parts of the region. India last year was terrific growth market for us. This year, not quite as terrific, but still relative to the rest of the world, I think it's still quite interesting. One of the big hopes we have in the region for 2023 is the recovery in China. I was in Shanghai just last week talking to a lot of our customers and our team. And I think we feel quietly confident that there'll be a steady recovery. So not a rapid ramp-up, but a steady step-by-step recovery in China, which of course is a significant part of the region. So I think there are a few green shoots, particularly if we look at China, because China, of course, is not just about the domestic China business. You know, when China grows and when Chinese consumers feel confident to spend money on brands, they do it outside of China as well as in China. So I feel cautiously optimistic for 2023. Thank you so much, Rupert, for your time and have a safe flight back home. Thank you so much. A real pleasure to talk to you today.


Kelly Kumalo on conquering everything meant to destroy her. Songstress Kelly Kumalo has mastered the skill of turning the lemon's life gives her into lemonade and she's still determined to emerge victorious from her recent battles. Kelly's latest track, Impini Written and produced by Mondlin Kobo of Inca Music fame also talks to the wars she and people are facing. When we were in studio creating the song, we wanted to talk about the woman who has conquered everything that was meant to destroy her and that's how the song came to live, she told Schwisse L.I.V.E. The song is not just about my personal war, it's about the wars that we, as individuals and as people, are facing on a daily basis. If you look at where we are right now, there are so many battles that we are faced with. From COVID-19 to GBB, to racism and the list goes on and on and on. Kelly said each time she released a new song, she realized once again just how much love she gets from people. A refreshing reminder, she said, because on any odd day it can seem like the world is against her, especially on social media. It is crazy for me sometimes, the amount of love I get having such a positive reaction from people always warms my heart and it truly validates me as an artist. If my life is anything to go by, I think I am blessed. What I've learned is that people fall in love with authenticity. For me, no matter what life throws at me, people continue to support me. I think I have also mastered the rare skill of making lemon juice when lemons are thrown at me. I never see negatives, instead I always make things work for my good, Kelly said. No matter how most people feel about Kelly Kumalo, there's only a few people willing to deny the fact that her vocal prowess remains unmatched and the songstress keeps proving she deserves the vocal highness nickname Zancy has given her. In Penny, Kelly's latest single which translates to at work, sees Kelly talk about some of the battles she's had to fight but the songstress explained that the song isn't actually about her. Kelly explained that she realized a long time ago that her songs belong to the people that listened to them and since then, she's been singing with the hope to uplift, inspire, or help people who might be in situations described in her music. I've learned that it's not always about me. As much as I'd want to talk about me and my life and battles, as an artist and as a person respecting my calling, I get more out of a song when I talk about things that other people can relate to. It makes better sense to me to have a broad message in my songs that people can relate to.


ఇరరాన తారితి fountain నిమరా సంపలిటిమూ eyelashes మనిటినంర౯ నిమనిమతినిలౚి స౐చెంప్త్ సంకిమిలారకాలు.  Representatives are getting away by coming here to kids over their problems They are happy They are happy They are giving all they should be prendre rewarded A scene has just come we must get准 États To recognize thatถर Quantity You have come to see whenever a kid performs That is so happy People used to come to普通 Pram pod But now they show themselves And tell others You are just so happy You are hugging everyone Now even the undertżers remember  lotus dhupote  endang Yes, in Amrasarth i can do so How was you building this small monitor fort in India ever? Great And now at that many start 노래 Raht Lets fly I know Prachete you have 박un How do you feel about this monitor? ୂുੈ૒༱ૉ༱ૂ, ୂു൜༱ૂૂ༱ૂ༱ૂ, ୂു༱ેૂૂૂ༱ેૂૂૂরૂેૂૂૂ༱ૂૂૂૄ༱ીૄ૜༱ૂૂૂ. ୂുു༱ૂૂૂો༱ૂ. 搭ning siya may ofidab by planteware SDHDU  這個人  恃 ちょ  に 小  に           可以  あ года and uò N      L         Let  Hua  地 思選 bdaya Bdaya     ね  Ok Si Tu  Mu Say Wa Len Ka Ra 50 Delete nosis