Hi, welcome everybody. Thanks a lot for coming to my talk. What you are about to see here today is pretty cool stuff. I had lots of fun working on that project, so I hope you'll find it cool too. So in the nutshell, I'm about to show you how to set up a cache timing cover channel so that two VMs collocated on the same physical box, same socket, can talk together without being detected. Without being detected is important here because this talk is all about practical implementation, not just some proof of concept or theory stuff. And so just a second. So before we begin, usual disclaimer. So this is a research project that was done on my own time, my own network. This talk reflects my opinion and not the one from my employer. Information and code provided here should be used for educational purpose only. All right. Let me introduce myself and also give you guys some context around how I ended up working on that project in the first place. So my name is Etienne Martinot. I'm from Ottawa, Canada. I work currently for Cisco system where I do Linux kernel and KVM stuff. As a kid, I was fascinated by electronics and radio, especially the concept of modulation. It was just much later during my engineering study at Laval University that I finally realized what was going on. That was cool. But then I got a job, ended up spending several years hacking in the Linux kernel. And as you may imagine, I kind of forgot all about modulation. Up until recently, where part of my work doing some low level performance analysis on KVM, I noticed something strange. Basically, I observed some sort of cross talk going on between two virtual machine. Obviously, the cross talk was very subtle. But the tool I was using was smart enough to detect it. So doing more investigation, I realized that the two VMs were being assigned to the same physical core, but different threads of execution. So this threads of execution concept is typically known as SMT or hyper threading on Intel. There's more research and I found that nice picture from Intel. So here we can clearly see that the two execution threads are actually sharing some common functional unit. And some operation have to be serialized one after the other. So that stuff explained the result I got. But then I had an idea. What if on one of the hyper threaded sibling, I modulate a contention pattern over the execution pipeline? Let's say a long instruction is a one. Let's say a short instruction is a zero. Then what if on the other hyper threaded siblings, I try to detect that amount of contention over the execution pipeline by executing an instruction and measuring the time it takes? If it's slow, then it's a one. If it's fast, then it's a zero. Then I realized that with this technique, I could do pretty cool stuff such as sending information from one VM to another, assuming that the two VMs are being assigned to these hyper threaded siblings. So I ended up spending quite a bit of time on that project. And so one of my goal was to see with my eyes the quality of the signal being the quality of the communication channel. So naturally I decided to try to transmit an image so that I could see the result thing output on the other side. Obviously the very first image I used back then was not this one. It was a picture of my kids which I am obviously not going to show around here. So I used the Defcon logo this time which has been reformatted at 640, 480, SVG quality one bit per pixel. And this is what I got on the other side. So that was pretty cool because I was able to see all the noise present in the channel and so on but still I was able to get some information out of it. So then this is at that time that I essentially realized the problem related to security and I said to myself there is a big issue with this type of stuff and this is basically why I am here today. And by the way before we move on to the core of this talk just for the fun of us here at Defcon I have a recording that shows what happened in real time when we take that image and we send it over and over again at 15 frames per second. One thing I did in that recording is that I have my noise generator that I start and stop in the background which is essentially a compilation of the Linux kernel. And then you will see the effect on the communication channel. One more thing is that I am running the mp4 encoding software on the same machine where I am running this experiment. So on its own this thing is generating quite a bit of noise. All right let's take a look at the video here. We see that when the Linux kernel compilation is running the channel is completely saturated by noise and that's expected because the pipeline is just running so many instruction at the same time so many context which are coming that nothing can go to the other side. All right let's take a step back here. My goal was to come with a practical implementation not just some theory stuff. Why? Well because I wanted to prove that this is a real issue and that we need to fix it. So now in this talk we are going to go over the design and what it takes to basically come with such a cache timing channel. So basically we are going to go over the shared resources on x86 multi core. I am going to show you guys how to encode and decode data using cache line. And doing that we will see the effect of the hardware prefetcher and I am going to show a trick we can do to get around it. Then we will also see that the encoded data that we put in the cache line doesn't stay there for very long time especially with VMs because with VMs there is lots of noise. Also we will, I will show you how to find cache lines that are shared across VMs. And at the end of it I will show you my phase lock loop implementation which essentially enable two processes running from different VMs to synchronize together very, very precisely. Finally we will do some detection and mitigation aspect and also towards the end of this talk I will basically measure, do some sort of bandwidth measurement on this channel and we will also go over a reversal example. Alright. So when you have IPertrading enabled there is lots of possibility for inter-VM modulation assuming that the two VMs are being assigned to the different, to the two IPertraded siblings on the same core. You can do pipeline contention. This is the first example I show you guys. But you can also do modulation in L1 cache or you can do modulation in L2 cache. Now if you have IPertraded disabled. Alright. Maybe this one. Alright. So as you guys are all very familiar with by now we have a fantastic tradition called Shot the Noob. Has this guy been doing good? That's exactly what I like to hear. Alright. We're not going to hold them up too much longer. To new speakers, new attendees and do DEF CON 23. Alright. As I was saying now if we have IPertrading disabled which is typically the case because this type of issue has been reported way back then in 2005. Link at the bottom of this page we can still do modulation but this time in L3 cache. This is what this talk is all about. Obviously if the VMs are being assigned to different socket this cache timing modulation won't work because the cache are not shared across the sockets. But now you see there is that buses that connects all the cache together. The co-eventsy module that can potentially be used. So this is also interesting but this is outside the scope of my talk today. Alright. Now it's time to understand how we encode data in the cache. So a cache line typically holds 64 bytes. So when you read a byte that is not in the cache it's the whole cache line that is brought in from memory. So now the basic of this trick relies on the fact that we can measure very accurately the time it takes to read a byte from memory. When we get it from L1 it's very fast. L2 a bit slower. L3 even slower. And in memory in main memory it's very slow. So now the way we do for encoding a pattern in the cache line is we load or flush a particular cache line. Let's say when it's loaded it's a one, a flush is a zero. And for the decoding part what we do is we measure the time it takes to read a byte that corresponds to this cache line. If it's fast it's a zero because that cache line was loaded. If it's slow then it, if it's slow then it's a zero, sorry. If it's fast it's a one, if it's slow it's a zero. Sound simple? All right let's take a look at the practical example. So when I started that stuff I wrote a simple client and test program. There is no VM in the picture at this time. This thing is running just on Linux on the host directly. And the cache lines are coming directly from shared memory. So here the client is encoding a pattern. This is the graphic that you see at the bottom left here. And once the pattern is encoded in the cache, basically the client wakes up a signal on the mutex and the server wakes up. And do the decoding. So but now here there is something weird. This is what I got on the other side. So but there is clearly a pattern when you look into this. This is just not pure noise. All right then I decided to take a step back. I'm just going to write a simple test that flushes all the cache line from zero to 100. And then after that I'm going to measure the time it takes to load them back. And I'm expecting long latency for all of them. But there is obviously something else going on here. Some of the cache lines were exhibiting long latency but lots of them were very fast. What's going on? This is at that time that I learned about prefetching. So prefetching means bringing data or instruction from memory into the cache before they are needed. So on the processor that I'm using which is a Xeon 5500 there is more than one prefetcher. There is prefetch for L1, prefetch for L2 and there is also different algorithm out there. But at the end of the day this thing is trying to predict what address will be needed in the future. All right. Before I move on sorry this hardware prefetcher stuff is one of those things that you can control at the BIOS level directly enable or disable that. This is obviously not what I'm doing here because we don't have that type of access over the machine. So we have to work around it. So I came with that idea that I'm going to just randomize the cache line access. So I'm going to basically randomize the cache line access within a page. Fair enough. But it turns out that we also need to randomize the cache line access at the page level. So in other words you cannot just go with an incremental pattern at the page level because the hardware prefetcher will kick in and will detect that and will try basically to load your cache line in advance. Doing all that apparently to manage the hardware prefetcher at least on the machine that I was running on. Then I face another problem. So basically what happened if you end up waiting longer before doing the decoding? So right now I have a client it basically encode, signal the server, server wakes up and the decoding. It's very fast. But let's say what happened if you wait? Wait more? And wait even more? Well, we clearly see that the time from when you encode the data to the time from when you decode the data has to be very small. Otherwise the other stuff that is running on the system will kick in and start to pollute the cache and essentially erase your data. So in other words the encoded data and the cache evaporates pretty quickly. And this is even more true for us when we are running in the VM because with VMs there is lots of noise. So talking of noise. So I've done a couple of experiments to try to characterize the noise and so on. So I have a test program that basically it's basically using a calibrated software loop that takes exactly two CPU cycles to execute. And I'm running that loop 100,000 times. So I'm expecting that the execution time in cycle is going to be 200,000 cycles. And I'm repeating that test 1000 times. So when you are running on bare metal kernel with all interruption disabled there is no noise, right? The loop is always taking 200,000 cycles over and over again. But now if you are running in user space, right? There is processes running and so on. Well there is some noise. And actually the noise is coming from the host operating system that is doing interruption handler and all that stuff, right? By the way, the small spike that we see there are the timer interruption on a per CPU basis. So this is a six core machine. And the bigger spike is actually a network interruption that is running on CPU zero. Now if you are running in the kernel inside the virtual machine with all interruption disabled there is still quite a bit of noise down there because you know the host kernel is running, all the interruption is running and you have some more noise because of the hypervisor layer. And finally if you are running in VM user space, which is the case when we will do that communication thing, there is quite a bit of noise. And that is because the kernel that you are running on has its own timer interruption, all the stuff is running there, the hypervisor layer and then on the host. By the way, it looks really bad here but if you make the mat, the gradation comes down to about two percent which is about what we expect for a compute load when running on a VM. All right, now we understand the noise, we have a way to trick the hardware prefetcher. Now it's time to put the client in VM one and the server in VM two. Remember the first test we did, all that stuff was running on the host directly. So I put my server in VM one and VM two. But then I realize there is another problem. The cache line that I was using initially are not available anymore. Basically the L2 and the L3 cache, those things are tagged by the physical address. But in a VM, the physical address that you see has nothing to do with the real physical address that exists on the host that the cache is using. Why? Well, because there is that other translation layer, that other translation layer, sorry. And basically the VMs in the virtual machine, they don't have access to that information. It's a tricky problem to solve, but I don't think it's impossible. But fortunately for us, we don't have to worry at all with this issue. Thanks to KSM. Well, at least as long as KSM is enabled on those systems. So what is KSM? KSM is a kernel thread that runs on the host kernel that basically scan the running processes and compare their memory. If it finds identical pages, KSM merge them into just one single pages. So, but obviously if one of those program wants to modify, let's say those pages, KSM kicks in and basically do the unmerging. KSM is pretty useful because it saves a whole lots of memory with VMs. Especially when the guest, especially because the guest operating system image from one VM can be shared with the other guest operating system on the other VM. All right, coming back to that slide, the idea I had was to create a unique pattern on a per page unique pattern in memory that is the same across the client and the server. So that the idea is that on hostOS, KSM will look at those things, do the scan and it will eventually do the page deduplication for us. Note here that the per page is important. And because if different pages are identical in content, KSM will detect that and will merge them one on top of the other. And so you will end up overlapping your cache line, which is obviously not what you wanted to do. Side comment, with KSM you can do pretty cool stuff. Such as in identifying the operating system or the application that are running beside you. All you need to do for that is to load in your own memory the image of what you think is running beside you. Then you wait a bit because KSM deduplication process takes time and then you write to some of those pages and then you measure the time it takes. So if the time it takes to do the right is much longer than the normal right inside your virtual machine, it means that you got KSM involved all the way down from the host. It did the page deduplication for you and it means that you have a match. You basically identify what is running beside you. All right, coming back to that picture again, I realized there was another problem with my design. Basically there is no synchronization primitive across processes running in different VM. Remember when I was running directly on the host, server was signaling a mutex, client was signaling a mutex server, was waking up and doing the decoding. But here there is no such thing. Well in reality there is things to do that. For example on Linux there is an KVM, there is IVSHM where you can basically from one VM to another signal is a MAP4 and so on but that stuff is not enabled in production. We need something to replace the mutex. Why? Well because we want the server to run right after the client so that he's going to pick up the signal, right? Remember what happened? If we wait too long all the data is gone so we have to be fast. So then I ran a couple of options I did not really know how to attack this thing. So one of the options I had was basically I'm just going to forget all about that synchronization aspect and kind of hope for the best. With some with error correction ECC we can achieve some data transmission. You know obviously there is there is space between the client and the server and it's come totally random so this is going to give us very low bit rates obviously but the CPU consumption is low and now that's kind of cool because we need to be we don't we don't want to burn CPU because everybody's going to detect us right? So another option it's basically I set it up such that there is a busy loop on each side and the client is set up to run a bit faster than the server so at some point in time there will be an overlap and at that point the server will pick up the signal and the transmission will happen. So this is giving a no K bit rate but the problem with that one is that the CPU consumption is very high and this is no good for us because we want to remain undetected at hopefully well and we would like to be less than one hundred one percent CPU usage sorry. Option number three so this is basically my phase lock loop implementation that I was talking at the beginning. Let's define a common period on the server and the client and let's let's have the client and the server lock into phase. How I did that? Well at the beginning of each period I have the server that is sending out a sync pattern. This sync pattern is very similar to what you found in those analog TV for the vertical sync that's the kind of the same concept and then you have the client that is running a swept scan over that period and tries to detect that sync pattern. Once it detects it, it basically lock on it. So what obviously once the sync is detected the client is just shifting back the phase and now we are ready for transmission but there is a there is a tricky problem for that to work. We need a monotonic pulse. We can in theory tolerate some jitter but not too much because in VM there is lots of noise and the data evaporates out of the cache very quickly. So in practice all that stuff looks fine but in in theory all that stuff looks fine but in practice it's a bit more tricky. How can we achieve a monotonic pulse? First thing comes into mind it's to use timer. Well timers are good because anyway we need to sleep in order to avoid detection. But the time with timers there is a big problem so this is a graph that represents the latency in microsecond the this the latency distribution in microseconds of a timer that is running in a VM. This is a log log scale. So we see that from that graph there is there is lots of jitter. I mean it can range from all the way down to 20 microseconds all the way up to almost 200 microsecond. And now if you factor in the original design this timer this jitter from the timer is going to come from both VM at the same time because they have the same type of distribution. So there is just too much jitter for that to work. The data will not persist and transmission will not happen for sure. Okay so the idea I had was to basically compensate this timer in software to some to some value above the maximum jitter so that in theory this should give us a nice monotonic signal. Well here we need to be a bit careful because this compensation thing is going to be subject to noise. In other word what I'm trying to say here is more time you're trying to compensate something more noise you end up accumulating from the underlying stuff that is running beside beyond you. Right? And the other thing that needs to be that we need to be careful with is this compensation is burning CPU. But on that aspect it's not too bad because all we have to do is to stretch the timer period to some higher number and we will still stay under 1% CPU usage. All right it's a tricky problem but I believe at the end I got it right. The compensation thing I'm using is basically a calibrated software loop that is kept in check with the TSC at every single point in time. And this is the result I got. So my machine is a 2.4 gig machine and when I'm running Hydal this graph represents the jitter that I have on my compensated timer and it's in cycle. So I have roughly I have 50 cycle of jitter on my timer. It corresponds to 20 nanosecond. Even on a loaded system I've got roughly 300 cycle. It's 120 nanosecond. It's pretty fast. Pretty accurate sorry. If you compare that with the original timer I mean there is obviously no no comparison here. Even if we put the latency back into perspective with the original graph it doesn't even it doesn't the jitter doesn't even show up on that scale because the original timer was in microsecond scale and now I'm working in nanosecond scale. And as you may already understood this synchronization aspect is the key of this design because it basically enables the communication to happen with very low noise because it's very very precise across the two process running and at the same time it consume low CPU. Okay let's recap what we have so far. So we have an encoding and a decoding scheme that is based on memory access time. Slow it's a one, fast it's a zero. We manage to get rid of the hardware prefetcher without disabling it by the BIOS because we just randomize the cache line access at the page level and so on. We also found some we also found cache lines that are shared across VMs and that is thanks to KSM by the way. And we managed to design a phase lock loop that basically gives a very high precision across two process running a different VM. Time for a demo now. Okay so I basically with that technique I basically repeated the original experiment which consists of sending that DEF CON logo from one VM to another. We can see that this techniques is offering pretty high quality and kind of low noise communication channel at least if you compare that with the original pipeline contention example I show you at the very beginning right? So there is no error correction that is running on the transmission channel there is no retransmission nothing but as you can see if you look carefully you'll see that in the picture there is a couple of bits that are flipped here and there and that's expected right that the channel is kind of noisy a bit. All right then in that experiment again it's another recording I repeated that exact same streaming experiment with my noise generator running on and off in the background. The first thing I want to mention and that's kind of cool is that when the transmitter is not running the receiver is picking up the noise from whatever is running on on the on the operating system. So to me this could potentially be used to fingerprint the operating system that is running underneath. Also in this recording here same stuff than the previous one I have the mp4 encoding software that is running on the same machine and so on. So on its own this thing is generating quite a bit of noise. But still you will see the effect when let's say I move a window around for example and of course you will also see the effect of when I compile the Linux kernel you will see the noise going on and off and so on. One last thing I may have mentioned it before there is no compression no retransmission there is no protocol what you see is essentially the raw capacity of that channel. Let's take a look at the video so that's the noise I was talking about. You see the compilation of the Linux kernel totally saturate the channel so on the left it's the source and on the right obviously it's the destination of the payload. Alright so now we'll make the mat real fast here so that video was transmitted at 60 frames per second interlacing interlaced four time 15 full frame per second one frame was a VGA quality 640 by 481 bit per pixel it's if you make the math it's roughly 4.5 megabit a second and the CPU both sides were 50% CPU utilization. Of course if you if you utilize more CPU you can crank up the bandwidth on this one. Alright now let's focus on something a bit more useful than trying to stream a picture from one VM to another, a reverse shell. First thing I'm doing here and again this is the same two VMs that I have originally is that I'm running the client in I'm running the server in loopback mode which basically essentially display whatever was sent by the client and the client in that mode is right is sending a bunch of exact a small A and in the background I'm running on and off again my noise generator and you guys will see the effect you will see that the synchronization will happen at some point. Alright now transmission is going on. I'm going to hit the pause button for a second here so in my program the reverse shell stuff I have a way to basically turn on ECC on the communication channel because as you can observe there is some bits here and there that are flipped right so now I'm just going to unpause that and I'm going to run the system with error correction turn on this time. Alright now we see that the output is clean right? Obviously sometime it will happen that there is some more there is more bits that are flipped than what ECC can support and so in my program is actually displaying a couple of star right there the servers is displaying that obviously we can we can we can crank up the error correction I'm just using I believe a 16 bit ECC over 240 bit payload so that can be increased easily so now I'm going to basically remove the loop back mode because I want to send command on the other VM that's the reverse shell mode right so that was just a test mode here so I'm just going to unpause that thing so again the lock was pretty fast right now the lock is when the phase lock loop is synchronizing so now I'm sending command to that other VM commands are coming back this is the process that are running in the other VM just going to hit the pause button again well to be imaginative here I named my stuff timing channel so the timing channel is the program that is running in the other VM as you can see that the time that this program has consumed so far is very minimal and yeah that's the that's the reverse shell here I'm looking I believe at the source code of this program yeah all right so you see that you know the reverse shell is not super responsive and that is because the server has been configured in a mode where I don't want the server to burn too much CPU on the other side in theory we can we can dynamically crank it up or crank it down right now I believe it's using half a half a percent of CPU in this demo so you see the response of a responsiveness level all right so that's it for the demo so what can we do to prevent that stuff so the first thing is obviously to disable that page de-duplication thing or to set it up on a per VM policy not on so that it doesn't basically cross the VM boundary that's one thing you can do it's gonna it's gonna take care of this those enter VM shared with only pages it's gonna it's gonna remove them all so this flush and reload technique won't work essentially and and then it will take care of that side of the M and O application fingerprinting thing I was mentioning earlier obviously this is at the cost of higher memory one of the other thing that can be done is on the x86 today this C flush instruction is not it's not privileged so maybe we can make it maybe Intel can make it privileged or something don't know if it's a micro called update obviously you have to revisit your call location policy let's say what what what are you putting on the on the core what you put on the socket what you put on the box basis personally I'm more of a fan of of trying to detect this type of communication and for that to happen there is many there is many let's say way we can do one of them is there is a bunch of hardware counter that are available for performance reason on the on those chips so one could do some sort of pattern and noise analysis to try to to detect you know some some spike and some very you know precise noise the other thing that can be done is you know it's it would be too to try to detect those enter VM process scheduling pattern meaning that let's say let's say you have two process running into different VM and they are always scheduled at the same time somehow they always overlap so that could be detected obviously and one of the other thing is abnormal TSC usage for for that for that stuff that I'm that I've been doing there is lots of there is lots of call to our DTSC because this is what the compensation is doing and so on so one could would could monitor and put some your stick around on some usage pattern of our DTSC instruction and try to detect that so this is really what I'm working on yeah and so basically that's pretty much all I have the source code for this in this cottage code is going to be on my Github I'm just going to send an update on on this slide deck shortly thank you very much everybody


I want to encourage you in this little piece to camera to keep an eye on your relationship with doubt, yours and your attitude towards other people, especially these parents with your kids, leaders, doubt, doubts are the foothills of Mount Wisdom. You have to go through them, clamber up through those foothills to get to higher levels of consciousness and awareness and clarity. But if we demonise the foothills, people will stay there, quit there, and leave the mountain and never discover the wisdom that can only be accessed by journeying through the foothills of doubt. I think we need to revise our relationship with doubt. All right, I hope that helps.


So welcome everyone, did you notice the fog last night that kind of played into my cards? Because what I'm talking about today is something that has been dogging open stack for quite a while I have seen it myself. I go to customer sites regularly and one question that I get by asked by pretty much every management member at one of the Sessions is so and how am I going to see what the cloud is doing? I'm going to pay two million bucks for this. What do I get for it? And this is something that opens like at the moment cannot answer so I Came up with a forecast system initially I Just started as a Just an exercise for myself to make my work a little bit easier And then Colin my co-worker who is unfortunately not able to come today. So He has a big part of this He came up with the idea of actually casting that into code and asking the cloud for advice So clouds are being built by architects and engineers. It has been the way since I joined Mirantis four years ago We have in the beginning we were talking to mostly all technical people and we were by building relatively little clouds that were let's say test To sort of dip your toes into the pool of cloud computing nowadays we are talking to very large cloud customers that are building million-dollar clouds and There with these clouds obviously have to be paid for somebody and Our managers want inside and we are what we want to give that give it to them So this is what I've seen some quite a few customers You make a little extra spreadsheet you get a couple of values from the different days and Then you make your curves you to come up with all this might this looks about right and If you present this to management management is going to say yeah, and who tells me this is right This is well, it looks good, but it's I Don't really have confidence in that this is going to be the right Forecast so well you want half a million bucks. You only get 150,000 And the other thing is obviously you just put a couple of values in for in wrong and the whole forecast is going to be Unusable so next thing who would know who would have the values that you are looking for Values, of course are in the cloud the cloud knows what is being used how much Of the resources that you have that you theoretically have Actively being used how much of the resources are open how long it will take to get there To the point where your resources are not going to be enough anymore The most important keyword here being obviously automated because the last thing you want is something that is very complicated to use And that obviously nobody is going to use for that very reason So big challenges. We have a challenge in tooling. We have a whole bunch of different Software that helps us see what the cloud is doing right now. We have one Miranda's open-stack has Stacklight there's a couple of other companies who also make tooling for that But this is also always only a snapshot you have a An operator who looks at the cloud sees everything is healthy or this is broken. We need to fix that Yeah, okay, this is not good and so You cannot really use that for reporting other than again copying values out of it and putting them into a spreadsheet At best case you will get a little bit of history, but it will not help you with the forecast So what's in reality what happens is this? You just take the value you say, okay, this year's budget was X next is what we want a little bit more so that's how we come come up with the number for next year and In many cases, you know, okay, this department is going to be onboarded they have a lot of compute resource requirements and Putting that into your forecast makes the thing exponentially more complicated. So you have that curve that you're plotting in Excel and then you have to somehow add that workload jump to make sure that you will get to the point where you can again have an accurate forecast or reasonably accurate forecast and So what this leads to is that projected budgets usually are wrong and I mean you probably most of you probably have seen that you Pro you forecast X amount of money Dubai something The forecast is was incorrect you actually need more and then it goes to rush order Instead of a six-week procurement cycle you have to have this in a couple of weeks costs a lot of money and Obviously management is going to be rather unhappy about that and The impact of that is almost always only seen when it's too late unit, you know Okay oops, we are getting very close to the Ceiling but when you know that it's already usually already after the procurement cycle the last procurement cycle is over So the business effects, of course, and this is what the managers again are going to be interested in is Budget overruns you spent more money on stuff than you would have otherwise if you had ordered it in time late delivery of products or services if Department can be onboarded three months late because you don't have the hardware to onboard them then you will get to the point where You simply they simply cannot Do their job anymore until the onboarding happens outages, you know You run a whole bunch of systems at peak the risk of an outage Goes drastically up You have Arrows due to overstressed operation staff, you know things like somebody pulling the wrong heart, this things like somebody Nuking the wrong server for for onboarding and Then at the end of what this all leads to is lost or unrealized revenue Nobody really Wins in this it in this scenario management does not you do not and and then we are what we're getting to is Everyone is angry at everyone without knowing what really went wrong So this is the ancient history. This is what I built myself to Calculate clouds for customers Can see at least somebody is kind of interested in that this is Basically this got me found from doing calculations for a week down to a couple of hours or less to project reasonably accurate Cloud model for a customer this obviously only works at the very beginning when I'm starting out At a certain point in time when I'm when I calculate something for an existing Requirement so what do we do? We Introduce weather station And this is Going to address or is this supposed to address the problem that we have here Stacklight is great for monitoring But what we really need is a stack light over time and this is what weather station is and as we have clouds the weather station came from we are Monitoring clouds forecasting clouds and this is what the weatherman does so instead of building for operators and technology people we are now tooling for business we do the We project cost for the cloud scale out we predict when hardware will be necessary. I'll show it slides to that later on We can compare scenarios and obviously also integrate with ERP systems and provide the reporting data for Okay, you need to prove to management that you hit your 99.95 99.99 SLA so you can just pull the report say, okay, this is how long we were down This is why we were down and this is still less than we are allotted to to meet our SLA So here you go. How does it work? We have a core Unit, this is a flask server that takes data from the Influx DB from Stacklight and this is Was something that was rather useful for us when we started developing this Instead of deploying another agent to each and every one of your hundred two hundred five hundred cloud nodes We are just using the data that is collected anyway to for our Monitoring and take them and use those as Historic data on one side and then predictive data on the other side So we keep values longer than Stacklight does we have we have our own database to Put data. This is a thinned out values So we do not have a database that will explode and become terabytes and terabytes of Reporting data, we've only keep the values out of the Stacklight database that are going to be necessary for our different data point reporting reporting and We have configuration of course and that this all Ends up in a graphical user interface. I Initially thought about doing a live demo today. Unfortunately. This was a little bit Late and it's getting a cloud on the plane was not doable anymore. So we just basically just said, okay I'm going to provide Pictures of what what this looks like first of all we have the input output cycle of weather station We have of course the project manager and the architect the architect It communicates with the weather station by adding on for instance at a certain point you expect a raise in utilization and He also gets the values out of the cloud to analyze for future purchases For instance, it will show you that you are getting very close to the limit of Vcpu, but you still have plenty of memory left So what you can conclude out of that is that the servers that you have may be may have more memory then They need to it to be Balanced optimally you have for instance a server with 768 gigabytes of memory and only 24 CPU cores So you can see okay our workloads are not that memory heavy so we cannot so we can buy cheaper machines with less CPU we have on the other side the project manager who Obviously knows about the requirements for the cloud and also there's a direct line That's missing here between the project manager and the architect where data comes in and goes out and We have on the other side we have our Operations center and the all the data into the into weather station comes from there So the idea is to support business not the technology the technology is supported enough by other tools by Stacklight and We want what we want to do is we want to be the monitoring cloud monitoring for business and important is a head of time planning and Configuration tuning as discussed you have a command line interface if you like or automating things and you want to pull data from the From weather station you can do so or you can use a graphical user interface First of all pre-deployment capacity planning there is weather station can actually do that you can plug in a new cloud and You can plan or what what the What the cloud? Parameters that you need to give to the to the customer are going to be so You have To do that you have to be able to also store and compare cloud configurations. You can see okay. I If I do this Compared to that. This is what this is the result of that and The downside to this of course is if you go back to go back to that or if you remember the spreadsheet that I showed before You will see that the configuration for that is not trivial It looks pretty easy and I have given it to a bunch of co-workers who wanted to have it to do their own planning And then found out that if you change one parameter This can have very far-reaching consequences and there is no safety net. There's nothing in there So we have put some some of the safety net into the capacity planning for weather station, but we have not This is not to the point where you can just give that to anyone and say let's say to Well the project manager and say okay plan my cloud Okay, so this is a input panel for the cloud we have As an input we always have an Assumption of what the workload is going to be like. I mean In many cases, it's not as easy as it looks you can say okay I expect most of my workloads to be something like this you have To vcp use per instance and four gigabyte of memory per instance, of course this bird is varies very much by what you are actually doing with the cloud we have we're currently working on a configuration that uses hadoop and where there's Average instance is two-thirds of the size of the machine that it's running on so it really depends and it pays to sit down and With all the shareholders stakeholders that are going to be in this cloud environment and Come up with a reasonable estimate of what the average is going to be so One thing of course and this is something that we have also seen time and time it again is that we have shareholders who have very Love the ideas of what this is going to entail Let's say, okay. We have we need a thousand instances of the 16 vcp use and 64 gigabytes of memory and one of the Things that you can do then is you can actually Figure out There's a slight missing that sorry sorry about that. There's a slight missing We will figure out a rough estimate of how much this is going to cost So the idea is to say okay Yes, we can support your thousand instances with X vcp use and Y memory but You this is part of this the cost goes to your cost center and as you are taking up Two-thirds of my cloud you're going to call to cover two-thirds of the cost and in many many cases You will find afterwards that the Estimates become quite a bit more realistic and this is where this tool is good for being able to predict a rough estimate of what It would cost if you have a realistic estimate and if you have the lofty estimate and say, okay This is this is the Delta. This is what you need to cover and we have shown that to customers and in many cases The result was that people were actually sitting down. This is the result that I actually want to see not they're not People, you know saying, okay, we are replacing one rough estimate with another rough estimate, but actually sitting down and Examining their workload. I mean in most cases you have pre-existing workloads You have let's say a server farm that you are running your Instances on they go back to that and actually look at what is this going to look like in Real life and do I really need what I what I said I would need so this is a set of comparable cloud Configurations that we have made and you can see here There's this is this is a little bit hard to read but I can point it out a bit So you have three configurations that you're comparing in terms of a lot of metrics that you want to see you have For instance, you If you have a different configuration see you would see for instance, you need 58 node 62 node 70 nodes and Instances based on CPU just tells you okay in this case you can actually see the delta you have Okay, and this case the estimate for CPU was different between those Between those three configurations and so you can see in these cases you could run 15,000 instances on this cloud With the vCPU forecast that I have made versus 9,500 instances with another for forecast metric and The important thing of course are the constraints values for instance you look at My instances I have a value for total vCPU. I have a value for total memory and I have a value for Total storage capacity value for storage performance and then you look at those four values You have for instance it says on vCPU. I can run 10,000 instances on the cloud for on memory I can run 6,000 instances for on the cloud on Storage I can run nine thousand on storage performance. I can run 5,500 so what you find out is you have two parameters that are low and the other parameters are high in Which case you can say okay our storage is going to be too slow for what we are doing here so we need to either come up with more spindles or we need to come up with another way to make my storage faster and The same thing goes for memory I have more vCPU value more CPU capacity Then I have memory capacity So if after I fill up my cloud with workloads I will see that I still have overhead on the CPU side, but no overhead on the storage on the memory side So if I say okay, I add I'm adding X memory to in this configuration then I can Remove that bottleneck, but I still have the performance bottleneck so in other in Conversely, it might be useful if you do not have the requirement for these 10,000 instances for instance to say okay instead of boosting my storage performance. I just give the cloud a new Ceiling 5,500 instances and instead of raising my memory. I reduce my CPU count and they can say okay and then later on when I have I have a certain balance now and later on as Simply scale blocks of this out. I scale out my storage and I scale out my my CPU and computer like for instance, I put in a fourth rack a fifth rack and as I Maintain the balance. I can see okay now we have the money is Better spent than if I spend it spent a lot of money on one on optimizing one parameter and Later on hit the other Limit before I can use my expensive Expensive CPUs for instance, especially with CPUs. That's very important because high performance CPUs are usually very expensive Like we compare a mid-range CPU is half the performance of the top-end CPU But only costs 20% of that of that value Okay, so the next thing that we want to do is The capacity planning and reporting after deployment. This is where the meat of the matter comes in you have a Cloud that's already running and you want to know when will I had hit the ceiling? so we can in this case we can compare hypothetical scenarios what will happen if I That let's say I want to see What happens if I change my memory over commit or I? I change the sorry When I change my memory over commit when I change that this disk over commit or something like that and It will tell you what? When you will need to do your upgrades This is at the moment that tie into ERP and workflow systems is still in Development we have We are working on this very actively at the moment. The project is six month old and it is in some Respects still a bit incomplete, but it is already very usable and this is something that where we will work on The deficiencies and and build up from there I don't think any of the projects and I mean you a lot of you know from the early days of OpenStack How long it took for some projects to reach maturity? I think we are going to be quite quite a bit faster than that So this is another interesting thing this is where you have Your SLA reporting you can find since in this in this cloud We had no downtime we had We were obviously hitting our SLA and all the individual APIs all the individual opens the API's reported no downtime during the cycle that we are Looking at we have 87.5 percent CPU used and about 50 percent RAM used what you can again see here that is in in our cloud is CPU is going to be the bottleneck and So we in the future and expansions we can Either use smaller CPUs or my or more memory or less memory of course bigger CPUs or less memory okay, there we are and The interesting thing is also this pie graph when you see downtime and unfortunately I Slide that I had that does not have downtime. It would show for instance that a quarter of the downtime it was as with the three API and a quarter of the downtime was Cinder API and so on so you basically have an Explanation to what went wrong and where you in the future can Can or should make adjustments to improve that Of course, we are also looking at this can this the pie chart is visible on my monitor But not on the projector. There's another patch out there Unfortunately, it is held a light gray and apparently the light gray did not show very well and so this is a As we are not really doing anything with this cloud. This is basically just a demo cloud that has a couple of workloads that are Have no bearing on Realistic cloud configuration you can see this is a this has seven instances and Fmrl storage for seven instances and so This is this would be Let's say more useful if you have a larger cloud So the next thing is we can add Capacity we can say okay before we do actually do our Expansion we can predict. Okay. We are adding on the first of January. We're adding two new servers 17th of July. We are doing three new servers and then the graphs will show that differently I will show the graphs in a minute The same thing as we can add a usage plan list where we can predict onboarding of instances We can say okay on this state. We are going to in on what 50 instances on this state We are onboarding 20 instances. So you will see that the graph will also change for that and here we are Capacity planning so we have at the left here. We have the threshold we have how many these total VCPUs are available and this obviously changes as we are adding servers to the configuration and We have the actual use of that VCPUs. You can see this is climbing here You can change the prediction model in the software. This is This is some an average graph and you can change to linear that you can have different different models there and what this shows us in this case is that at At the allocation threshold, we have this is the amount of CPU that we have and we have let's say 80% of that We want to know when we hit that we hit we will hit this according to this prediction on this state this time and So you obviously know That you need four weeks to get a server in all of the server get the server in Could be longer and So you have to order ahead of time and you have at this point you have to add CPUs simply because you are going to hit your set threshold that you do not want to Pass so The other thing is of course you are planning for this duration. Let's say this is three months and You want to see what happens at the very end of those three months because that will tell you how much you need to order to not hit To move the threshold that's currently here to up here so you get out of this you get the order quantity, so you will have the delta delta of 1525 this is that 15 About You need about 13 vcp use it in addition to that of course this will the numbers will be much louder for your cloud But it is for demonstration purposes That's fine So how do the workload onboarding? predictions Come in this was actually This is the one so you have if you have more workloads if you start onboarding your curve will change then in a straight curve, it will also move from a straight it will get steeper and so of course in this case you are Predict that passing of the threshold is earlier than it was would then it would be in an environment where you have Where this workload was not on boarded? so What you can do for it do here is you can take the new value to see it Predt projection is different and we also may need to pull forward your orders, so you do not overrun your Current capacity so if we do not add here This is the adding of the capacity if you do not add the capacity then at this point you will hit capacity will be not only Overrunning the threshold, but you will actually hit the capacity and your cloud will be Fall you will not be able to onboard anything more unless you change something so in this case We have added workloads in this case. It's the other way around. We have added services We have added service at a certain point and this is Where we at the allocate the capacity we say okay instead of There at a certain point We will The other threshold line is missing there still this is we have we have that now So basically this was the old old threshold line and the threshold line would have been it would have hit here so we would have gone over the 80 percent and Instead we are adding capacity so the new threshold line is here, so in we are hitting the threshold line in the Father out because we have added a capacity to the cloud Okay, so But we are planning still planning to do is One that's going to be a lot more parameters at the moment. We have CPU capacity. We have memory capacity We have storage capacity and storage performance and that's going to be more parameters that are going to be managed and We have We are going to also come up with a graph and this is that something that has been a bit more of a Graphical user interface problem than a technical problem where we say okay We will have we will have both memory and CPU in one graph where you can see where you can compare okay on CPU We still have Headroom but in the on memory we are going to hit the threshold earlier on and you can and then you can adjust your configuration accordingly But this is not done yet and this we have we are working on getting this fixed the one thing to remember is this is meant as a this is meant as Add on to weather at a stack light because we use the stack at the moment we use the stack light Infrastructure to extract the data from the cloud in the future. We will also be able to put other Metrics engines into that so we will be able to use other Monitoring tools, but at the moment this is what we have to live with and The the other thing is that when you are interacting with weather station It is important that the people who are not Let's say Engineer is not supposed to make the forecast for for architecture We are basically when you are logging in you can get different different levels of Access to ensure that we do not that people do not project for produce forecasts and calculations that are not That will not reflect the true state of the cloud Okay, thank you very much. I hope that it was useful for you I hope that somebody wants will be interested in trying out weather station and also giving us feedback on what should be done better What what should be improved in the future what? what additional predictions you would You find necessary and And so thank you very much for coming and have a great open stock summit