Hey, it's Alex here from AlexFrigus.com and behind me, I've got a bunch of body panels, real-life therapy body panels. So if you have seen my 2019 body panel showdown series, you would have noticed or you would have seen round three, which was quite controversial where I measured power irradiance at zero engines, zero engines right at the surface of the light. Now explain why I did that in the video, if you haven't seen that, go and watch that first. I'll put a link below. I've since done a comprehensive update on my blog, which I'll also link below, explaining why I measured at zero-inch and the issues I had with measuring at six-inch. Despite all of this, people, including the companies that supply these panels, readers, viewers, yourself have reported and said, look, we want to see the six-inch and even 12-inch readings, despite the issues, the limitations. So I thought, you know what? I'll just do it. I'll do a video. I'll test at six-inches and share the numbers with you guys because that's the whole point of this video series. I wanted to show as much data so you could go and make an informed decision yourself. So I've got my meter here and I've measured out six-inches and taped the ruler to it. So what I'm going to do is just literally go through and measure the readings at six-inches and I'm going to film it and you'll see the fluctuations. One other thing I should mention is I've actually figured out a little bit more about this meter, like how to use it, the measuring rate. I can lock in a measuring rate there and I'm getting a lot more consistency with the numbers but you still see that variability from hotspots, beam angles and all that sort of stuff. But anyway, now let's get into measuring these numbers at six-inch. So one thing I will mention is I'm only going to measure total power, all right? So I'm not going to measure infrared, the near-infrared and then the red light, I'm just going to do total. I've already spent enough time on this and you can go back and see some of the numbers on there. No one really has asked me about red versus near-infrared so I'm just going to do total. When I do the 2020 review, body panel review, hopefully there's some new panels coming in and I'll do this whole series again, I will test that six-inch showing red and near-infrared. I'm also going to be doing a mid-size panel like these are the half-size, so the 360 red rush, the Mito Mini I think it's called, the Biomax 300 or 450. I'm going to review all of those for that review. I will measure red and near-infrared, sorry, I just had a brain fart there. All right, so that's it, let's see how we go with the recording here. Oh, one other thing, so all the companies, with their testing and all the YouTube videos, they use solometers, right? And what they, they set those solometers to measure at peak hold. So that will simply record the absolute peak number, the highest wavelength that's picked up, and that's what they show and reveal. I've got many issues with that, again, like hotspots, what happens if there's one real big concentration point and that's your peak, you know, like that, and then all the rest is like half that number. There's lots of issues there. I'll link to the article I've written about this issue, but that's what people want, that's what people are asking me to do, so I'm going to do it. I don't know if I can find a peak hold number on here, so instead you'll just have to, hopefully I can get the screen on here. Actually, you're not going to be able to see the screen because it's going to be so red. You'll just have to trust me on this, okay? I'll read out the numbers as I see them. Anyway, what we're going to use for this comparison sake is the peak number, all right, the highest value we see, all right, because it's too hard to average it, like, it's just ridiculous. There's so much fluctuation as I move around, so we're just going to go with the peak, and I hope you've run tight with that. All right, let's get started on this. So I just double-check, there's no way you can see the screen on here as I am, as I have the red lights on. It's just too hard to focus in because it's so bright with that red light. So yeah, you're just going to have to trust me on these numbers, okay? What I have done with this meter is I've locked in on the bottom right here. I've locked in the refresh rate, so it's going to be the same refresh rate for all of them. That's how fast it's measuring. It can be anything from, like, I think it's 100 microseconds right up to, like, five seconds in terms of how much data it clicks. So I'm going to put it 350 US. I believe that's microseconds. Anyway, let's go. And what I'm going to do is I'm just going to read out the numbers. Whoops, I'm just going to read out the numbers, and then I'll go back and write down the P number that I see. So this is the Mito Red, Mito Max. So we're going to start off panel here. Sure, I've got the settings right. Yeah, all right. So I am about two inches off. I'm at four, and this is milliwatts per centimeter square. 7, 5, 10, 12, 15, 20, 26. So I'm now in line with the panel. 35, 43, 41, 49, 46, 54, 54, 58, 54, 62, 55, 60, 65, 62, 68, 60, 61, 68, 60, 60, 62, 57, 56, 44, 40, 29, 23. I told you there was hips of variability there, right? What was the highest? 72, I think it was. That's a lot. That's pretty good. I'd say 72. These guys do claim to have the most power. They say they've got five more LEDs. Their numbers at the source were pretty low. So hey, that's pretty good, 72. You can see that variability there, and that's why I was so nervous about tracking this, because even when I was in front of the panel, it was 40, it was 50, it was 60, it was 70, and then dropped down again. I didn't get to 70 until I was way over to the side. I mean, it's all over the place. And if I average that, how do you average it? Do you just pick random points? But what happens if you pick five points and three of them are hotspots and two are really low points, and then you pick five points here and you don't get any hotspots? Like, there's so many issues. But anyway, 72. All right, I think that's what it was. 72, I have to go back and watch it. All right, let's do the same. So now we've got the red therapy, red rush. So I'm turning the near infrared light on, starting this up. It's so bright. All right, here we go. So we're starting off panel, seven. So this has got a much wider beam angle, because I'm way off and I'm still getting power on the side here, which I know I talk about. 10, eight, 14, 19, 19, 26, 30, 33, 44, 46, 41, 53, 57, 50, 58, 50, 60, 61, 51, sorry, 57, 49, 49, 49, 48, 51, 55, 54, 44, 39, 41, 52. It's dropping down. OK, I don't even know what number that was the highest there. Is it 60? Sorry, I'm going to have to get back and watch all of these. All right, let's move on to the next one. This is the Mito HQ Mito panel. Yeah, so one quick thing about the red rush there. I know they talk about having a wider beam angle, wider treatment area. And you saw that straight away because with the Mito red, when I was just off the panel, there was like, what, three or four points. Whereas here, I was like an inch or two off and I was already at 10. So this is a lot more focus. This is a lot more wider. I think red like rising, which we'll do at the end is the most focus of the lot, which again, it's like it's so confusing, right? I'm hoping you guys can understand why measuring powers is such a complex thing. Anyway, let's do the Mito HQ Mito Max. By the way, all these panels discount code Alex, you'll get a discount savings. Let's go. So this is one thing I don't like about the Mito HQ is you don't know if the lights are on. So fire and for it's on. Since fire and for it, it should be near free. Interesting. All right. And I read like, let's go. Start off panel. Hit the start button. Just checking this is working. Yeah. All right. So we're at three, two, four, six, 10, 14, 20, 23, 26, 30, 34. I'm just going to check something real quick. Yeah, 34, 35, 37, 39, 40, 41, 43, 45, 45, 46, 46, 45, 45, 46, same pretty steady here. 46, 43, 45, 42, 40, 41, 37, 34, 32, 24. So it's dropping down. So what was that? 40, 45. All right. So I just went back to check the numbers. Mito read the very first one we had, peaked at 71. Red rush, the second one, the red rush, 720. Peaked at 60. Mito HQ Mito panel here. The black one peaked at 46. So so far the Mito read is as well in front. All right. So now we're going to do the Biomax. Platinum LED therapy Biomax 600. Okay. I think this came in second overall with power at zero inches. So let's get this going. Okay, we're at five, six, seven, 14, 24, 25, 38, 48, 50, 58, 56, 59, 60, 61. Oops, 61, 64, 55, 53, 50, 52, 55, 58, 59, 60, 66, 54, 54, 44, 28, 17, 12, 7. So I think the other peak was 60. So I mean, based on that, what did we say? 66 for Biomax, that puts them in second place with Mito read at first, but that was second place behind Juve. So if Juve come out number one here, Biomax would drop down, but Juve very well may not, because I know Mito read say they got the best on the most powerful panel on the market. And Biomax have said they're more powerful than Juve. So let's find out. All right, Juve solo firing up. Let's do this. So we're at zero, zero, three, three. Oh, just check that. Yeah, 35, five, six, eight, nine. We're in line with the LEDs now, 19, 21, 25, 28, 30, 33, 37, 41, 42, 44, 45, 43, 46, 48, 47, 43, 46, 44, 45, 44, 43, 40, 38, 39, 41, 40, 34, 30, 22, 19. There we go, that is crazy, isn't it? I don't even know what I said, 48 I think it was. So that puts them, that puts them after testing five panels, remember Juve were number one at zero inches, testing five panels that are in fourth place. Minority is 71, Biomax 66, Red Rush 60, Juve 48, only just beating MWQ at 46. Interesting, huh? So I mean, yeah, there was definitely a flaw in my zero inch testing because the Juve came out like double the power. So yeah, I apologize for any confusion that has resulted in this.  Juve though, like MWQ, were pretty consistent. Biomax and MWQ, the two most powerful will seem to have the most fluctuation with the numbers, whether that's just because they have more hotspots, whether it's something to do with the beam angle, whether it's something to do with the LED spacing, or whether it's simply because they're much more high powered units, so there's more variability in it. I don't know. All right, so yeah, 48. Yeah, again, I want to apologize because obviously my zero inch readings were very inaccurate. I mean, minor read have sent me independent testing and they said they were the most powerful panel in the market. I haven't finished all the testing yet, but yeah, that's my meter here showing, yes they are, 71 has come out on top. Biomax have done their own testing and said they're higher than Juve. That's also what I saw, 66 versus 48. Yeah, I should have done six inch testing from day one. The reason why I didn't want to was because of that fluctuation because I didn't know what was fear. I mean, after speaking to a few people, they just say, oh, they use peak hold, right? Whatever the highest number is, that's what they're on with. I still have many flaws in there. I have flaws with all of these measurements, beam angle hotspots, all of it. You know this now. You know what, I'm doing this now, and we've got one more panel to go. So red light rising, I haven't forgotten about you guys. Let's see what happens here. I think red light rising actually do have quite a focused beam angle. All right, so red light rising full stack. I'm going to go down on an angle here because I've got the red lights in one row and then the neofrid underneath. So I don't want to just get red or just get neofrid. I'm going to come down and stake it, all right? Let's go. Start at this point. So we're at 3, 3, 3, 5, 6, 7, 10, 11, 16, 21, 26, 32, 39, 40, 43, 43, 46, 48, 53, 51, 54, 56, 55, 56, 57, 57, 58, 55, 56, 58, 59, 57, 55, 53, 40, 48, 45, 29, 16, 5. What was that? 50 something, 58. Well, that was the only ones in the 50s. All right, so we've got a total. We've got a ranking. First, let's go bottom up. Last place for power audience at 6 inches tested by this meter, which has been customized to also test in up to 10, 50 nanometers, because the base model only goes to 850 or 750. This is what I'm looking at here. All right, so I see the power levels with the graph at the bottom. I see flicker and whatnot. You want to look at that EE measurement. So that was 15. That was the last measurement I took. And so that's changing. The graphs are changing. It shows pretty cool stuff like the peak nanometer. There was 660.9. You can flick it. It's a pretty cool machine. Last place, we have the Mito HQ, Mito panel, 46 milliwatts per centimeter square. Second to last, we have the Juve. All right, 48. So they weren't far from rock bottom. They came out number one by a large margin at zero inch. I'm going to have to go back and change all this. I'm so sorry. I feel horrible for this. No, sorry. Fourth place, we have red light rising, the full stack. 58. And third place, red rush at 60. And second place, the Biomex 600 at 66. And first place as claimed, down the end there, very first panel I tested. Mito red, Mito max, 71. They're also the cheapest panel. That makes them a really good value. I mean, the most power, cheapest price point. Sure they don't have all the bells and whistles, but interesting. So let's run through the updated totals now using the six inch power, peak power readings. First place with 25 points was the Biomex. And second place with 32 points was Mito red, Mito max. And third place with 37 points, red light rising. Equal, shared between Mito HQ and Juve, they both had 40 points. And in last place, sixth place, red therapy, red rush, 720, 41 points. Thoughts, comments, feedback, please leave them below. Please don't give me a thumbs down for this video. I've tried my best. I hope you can understand why I did what I did. I've written about it, talked about it. I don't need to bring it up again. Still though Biomex are clear number one. You can't go wrong with Biomex, like nearly the top power. I mean, there are only a few points below the Mito red. Great performance all around, best warranty, great support. Five wave lengths instead of two. You get the modular capability, low EMF, low flicker or decent flicker. Good amount of LEDs, good price point. This can't go to Alex saves 5%. There's a reason why they're fast growing. They've become very popular in the biohacking space. And they might not have the marketing might that Juve has for instance, but their panel is a good panel. And haven't had any issues with, like I did with the Juve you saw at the start of the video. So there we go. Mito red, if you're really tight on the cash and you just want a high-powered good value panel, then Mito red, Mito max. I mean, they scored really well in the second place. And they are a lot cheaper than Biomex. So there you go. You don't get all the same bells and whistles as you do with Biomex. And it's still overall good, good panel. This can't go to Alex who also saves 5% on their panel. So can't go wrong. The third place, red, like rising. Yeah, quite a unique panel, different in many ways. They don't have the modular design and all that as well. But they have no flicker. So if you're worried about flicker, then they're the ones to go to. And they also had decent power. I mean, not the best, but it's up there. So you can't go wrong with any of those panels. Mito HQ hold fourth place. So if you're in Australia, it goes Mito HQ. You're going to save a ton of money. If money is not an issue, it's sure we'll go with Biomex. But Mito HQ just can't go to Alex. I think it saves you 10% as well. And Juve, I mean, now they're in fourth, fourth equal. It's kind of hard to justify that high expense. So previously, you know, I thought they were really high powered. So I was like, well, if you just want pure power, go with them.  I guess that's what I'm trying to say. That's the whole purpose of this review. There's a bit of panels. They're high power. You get a bit of warranty. Bit of whatever it may be. And it's so much cheaper. And they don't do a discount code. I'm wrapping up now. What I'm going to do, be sure to subscribe and jump on my email list. Because what I'm going to do is I'm going to do a mini review. I'm going to do a handheld one. And then in 2020, I'm hearing whispers that, you know, there might be a new company coming on the market or there's going to be a new panel coming out. So what I'm going to do once things have settled down a bit, I'm going to do this whole review again with 2020 to find the best body panel for 2020. And I promise not to screw up the power and numbers this time around. Please don't hate me. All right. Thanks guys. Thanks for the support. Give me a thumbs up for this. Share this with your friends. Comment. And yeah, you can always support me by using discount code Alex on any of these panels. All right guys. Bye.


At Trinity Harvest, our goal is to love Pikeville, love on Pikeville. We love community events. We love sharing with the community. We love bringing people in. We were tossing around ideas for fundraiser, for a local charity. We decided on Judy's Place and what better way than a 5K. Everybody's into fitness, into getting healthy. So we thought, good, another 5K for Judy's Place would be great.


You asked me what are the four most trained in the OH industry and what we can expect this year. So, good part about the OH industry is when there is a huge discussion point about digital, the emergence of digital is actually hitting the other media channels, auto home is immune from that channel because most of this digital space people I would say Facebook or Google or Netflix and Amazon they are the people and Apple they are the people they are using maximum out of home. So, it is very clearly evident that we are out of home we are out of that competition. I believe that more and more brands typically I will give you an example Netflix has gone and bought a company in US out of home company in US. So, more and more this kind of brands would be needing tactical campaign would be needing more top of mind campaign they would be using out of home. So, out of home has a great what do I say future provided out of home community understand the value the cleaner they put their act together in terms of cleaning up the board aesthetic beauty, operating in research monitoring. So, proper do's and don'ts if they follow I think it has a great future. So, thirdly I think in India the best part happened is 3 years of I and I way they have come together to set the norms of the industry and we are pretty much maintaining the hygiene of the industry as of now which is pretty much in control. There are changes visible in the out of home industry like Bangalore is going through new bylaws change there are place based digital out of home advertising opportunity coming up in Bombay. So, they are looking at destination based digital out of home you know permission the authorities are thinking of giving. Airport started looking stunning I think digital out of home is looking fantastic in Bombay Delhi airports and Vinaigrettings very good accolades I would say all the other modern airports too including Bangalore, Hyderabad, Calcutta so they are looking good. So, I would say India is seen in a very different light by the world as a fantastic market to do business and from out of home point of view also it has become a very very important market because it started looking good everywhere started looking good. We have the opportunity we have to just put our app together. Next is I think the handset manufacturers and OTT players are the new game changer they are doing pretty good. I am sure real estate as a local business model your James jewelry real estate coaching classes this will stay a sunrise sector for the out of home as a local business point of view. So, all in all I am seeing 2019 will be a great year because all will problem will be behind us problem in the sense that recent events which are coming up your look your election will be in place and I am sure the second half of 2019 will be great for out of home.


Hello everybody, my name is Lasse, one of the co-founders at Ivy, a Danish climate impact software startup based in Copenhagen, sitting around 60 people today. Most of us in Copenhagen, a few abroad as well. Let's jump into it. I think most people are probably aware that in order to live up to the net zero ambitious target set out in the Paris agreement, massive reductions in greenhouse gas emissions are needed. And I think the interesting fact here in our perspective is that companies actually account for more than 80% of those greenhouse gas emissions. So it needs to start with the companies. That's the real driver of change here. And I think thankfully for many and for this you can say overall agenda regulation or specifically the regulation called CSRD by EU sites is now forcing both big corporates and banks to measure the full scope of the greenhouse gas emissions and also to set out a very clear plan for reducing those emissions over time. And I think it doesn't stop with the big corporates and the banks because the big corporates are pushing this pressure downwards to their suppliers and the banks are pushing this pressure or responsibility towards their business customers. So luckily it doesn't end with the big corporates and banks that are being impacted directly by legislation. A lot of other companies are also being impacted indirectly by this. But in our perspective, the real driver of change here will not be legislation. That will be consumer behavior because we are seeing in most statistics by now that consumers are increasingly wanting to purchase, you can say, more responsible products. And we would even state that sustainability is becoming the third competitive criteria beyond price and quality. So it's really something here that you can say that's here to stay. And I think the very interesting part here is that it doesn't only impact the consumer facing brands. It actually puts that pressure downwards throughout the entire value chain of that company. So in this example, it doesn't only impact the retail of the soda that a consumer is purchasing in a grocery market or at a retailer. That responsibility is actually pushed from the retailer of sodas towards the manufacturer of sodas, towards the manufacturer of the labels that is put on the soda can or soda bottle, and then again towards the manufacturer of those labeling machines. So this is just one example, but I think it's a really good example of how that pressure is actually pushed throughout the value chain and hence impacting most companies even though they are not directly impacted by legislation yet. So what is the problem then? I think the problem that we see today is that most companies struggle in measuring their full scope of their GSD emissions. And as a result, they naturally also struggle in sharing those emissions with their customers, which is something that is now being required or even demanded by customers. And naturally, they also struggle in reducing those emissions when they don't have the baseline in place. So I think that's the fundamental problem that we are seeing in the market. And another thing you can ask yourself, why is it that's difficult for companies to measure emissions? Well, let's start with having a very brief look at what they have to measure. So I think many of you are perhaps aware of the greenhouse gas Scope 1, 2 and 3 emissions in very brief. Scope 1, the emissions coming from the assets you control or own. Scope 2 being related to emissions from your purchased energy. Most companies are able to measure that today or at least should be able to measure that today because the data maturity is quite high. The challenge here is that Scope 1 and 2 emissions combined only represent around 20% or even less of total emissions. So the remaining portion is actually coming from Scope 3. And again, Scope 3 is the emissions coming from your value chains, both from your supplier base or your upstream activities and also from your customer activities or your downstream activities. So representing more than 80%, and this is really where companies are struggling today. And why are they then struggling with Scope 3? I think what we typically see within Scope 3 is that the IT landscape or the data landscape is very fragmented and complex. So it could be many thousand different suppliers that you are purchasing from, many hundreds of thousand different unique products that you are selling to your customers. You might have a bunch of different ERP systems or the relevant data sources and you might even also be present in a multitude of different countries. So in other words, your data is not compiled, it's not normalized, typically it's not cleansed or categorized either. So it's not meaningful and it's not useful. And I think the result of this is that you don't really have the data foundation in place to even start measuring emissions. So I think that's the fundamental problem that they are struggling with. So what does it take to get there and how do we go about it at ID? Well, if you start in the middle in the white section here, Fundamentally to calculate emissions, that's relatively simple. You need a relevant data point for, for example, a specific product. Could be the quantities of products that you purchase or sell. And then you need a relevant emission factor that you can couple it with. That's not the hard part. I think the hard part and what most companies are struggling with is the left-hand side of this equation. So going from back to my point before, going from a very fragmented and complex data landscape to one source of truth centrally, which is cleansed, normalized and also categorized to a very granular level. Because if you don't categorize it to a very specific and granular level, you don't have the foundation for coupling that with a relevant emission factor. And then your results end up being skewed. And I think the implications of that worst case can be very severe. It can lead to greenwashing issues. It can lead to reputational risks.  So what we do at Ivy is that we actually carry our customers throughout this process from initial data sources to calculated emissions in our software platform in order to ensure that quality level and that granularity level which is needed in this agenda. I think what it enables us to do is that instead of making this a compliance game, then for us, this is way more. This is actually a method for companies to drive business value through climate change. And maybe just to specify, so it doesn't sound as fluffy as business value, what we mean here is that, yes, we do offer them a carbon accounting product which enables that they are fully compliant, they are audit ready, and so on. So that's in place. But perhaps even more importantly, we also offer them what we call a product carbon footprint product which enables them to get insights on the greenhouse gas emissions of their individual products and all the way into their underlying parts and components. This enables them to identify hotspots and drive carbon reduction opportunities within their materials and even also reduce their cost to those learnings. And then thirdly and lastly, we also enable them to share these product level numbers with their companies to drive more revenue and gain competitiveness because this is something customers are requesting now. To all in all, compliance, yes, but more so reduction and commercial benefits driven through climate change. So that's what I have for today. And thank you for your time. And of course, if you want to learn more, feel free to reach out through our website. We are also here today, a couple of people. And thank you for your time.


Kubernetes infrastructure so that it, you know, as Kubernetes native possible as a way as possible. And so, by doing this, the virtual machines are actually inside a container. Some solutions out there, such as Conna containers, I believe, maybe actually modify the container run time. That's something we're explicitly trying not to do because we don't want to be modifying that ahead of time. Now, in the future, that might be a restriction that's lifted because dynamic container run times is something that may come to Kubernetes in the future. But for now, that's kind of a hard and fast rule and one of the reasons that we're doing it the way we are. And, you know, by leveraging our existing ecosystems, teams are allowed to use the proper tool for their solution. Whether that's a virtual machine or a container, they can put that right into their CI CD pipelines. So, for the way we implement this, we actually are using a custom resource definition. This is, you know, I've got an example of one over on the right. It's basically just a YAML file for those who haven't seen this before. For those who have seen Kubernetes constructs before, this should look pretty familiar. In this case, the only thing special about it is the kind is a virtual machine instance. So, virtual machines here have their own kind. And this gives us the ability to express all common virtual machine parameters, such as memory, CPU, and the like. Because we're implementing this as a custom resource definition, we also inherit RBAC rules. So, users are only allowed to modify things in the namespaces that they're designed for and what have you. So, here's a little bit of the workflow. This is a busy slide, so if I could take a minute to explain this. When the user implements this custom resource or posts it to the system, that's a virtual machine instance. And so, that's actually just a record on the, you know, in the CD cluster. We've got a controller, a vert controller, which is monitoring for changes to custom resources or to virtual machine instances, in this case. And when it sees one, it actually schedules a pod. And that's all it does at this point. We just schedule a pod. And you can see that, you know, this is the third step here. And then, now the vert controller is a cluster level resource. So, it's only job is to schedule pods. Then on each of the individual nodes, vert handler is running. And that's another controller we have. And it is looking for these pods that have a special label on them so that it knows that it owns that pod. And it will then schedule starting the virtual machine inside of it. Now, there's a little bit of hand waving there, of course, because I said start a virtual machine in a container that's already running. So, what we're actually doing is we've got a daemon called vert launcher inside this pod that's actually doing that work. You know, just some full disclosure there. So, as far as scheduling, as I said, vert controller is scheduling a pod. That means that we're actually literally using a pod and pod rules for where we're going to end up placing virtual machines. That means anti-affinity, affinity labels, selectors, all of those constraints that you can put on a Kubernetes pod still work. And you can even use a custom scheduler if you needed. Now, the applications within the virtual machines, because they are leveraging a pod, all existing Kubernetes constructs such as services and routes still work, and we'll get a little bit more into what those are later. But we actually use labels on the service itself to designate which pod the service belongs to or where to route the packets, basically. So, virtual machines live in pods. Now, that's transparent to higher level management systems. But, you know, technically, that's not worse than it currently is before we did this project. Now, virtual machines leverage pods. When we have a new virtual machine record, any labels that are on that will be translated over to the pod. We're going to need that for scheduling, of course. And to match things like services. CBQ and memory resources, they're actually matched from what the virtual machines definition has to the pod, so that we're not over allocating how much we're requesting. And, of course, affinity and anti-affinity, I talked about that. As far as the storage, that's where the real rubber meets the road. We're using persistent volumes for this at the production level. And so, what that basically means is that any existing storage backend that you already have for Kubernetes cluster, we can take advantage of. And that's a one-to-one mapping between the persistent volume and your virtual machine disk. And so, by doing this, of course, we can benefit from all existing ecosystems that are currently out there. And there's a lot. So, for the disks, and this is actually not a Kuber project, this is a Cyster project, the containerized data importer. I'm actually going to take advantage of that and use it in the demo in a few minutes because it's just that awesome. This is something that allows you to take an existing virtual machine image, a raw disk or something, and actually import it directly into your Kubernetes cluster on the fly, which is something that, obviously, we're going to need if we're going to be able to pull this off because that's been something that's been complicated for us in the past. So, this is also, like Kuber, a declarative Kubernetes utility so that you have controllers and operators monitoring for resources that will take action when those resources show up. And so, the two use cases here to either designate an HGP URL that you would use to download an image from, or a second use case that I won't be showing is to actually use a read-only namespace within Kubernetes to copy golden images to your user's namespace. That way, they wouldn't modify the original and all the goodness that comes with that. So, as far as the network goes, we're actually using the pod network for the virtual machine. That's both bad and good. The good, of course, is that you're able to communicate with any existing container resource as it currently exists. So, we can also expose these services from our virtual machine using services and routes, as I've mentioned, to expose specific ports on your virtual machine to the outside world. We're looking at alternative networking options such as multiple networks or different variants, but right now what we're using is just a tap device inside the virtual machine. The unfortunate part about that is we lose the ability to do live migration, because in the beginning we actually had Libvert outside of our pod and at the cluster level. Or we had one Libvert per node and that allowed us to do migrations between a virtual machine between different nodes on your Kubernetes cluster. The trouble with that was a little bit of a rabbit hole, but we had some issues with PID namespaces and the like that we were violating assumptions and we just really couldn't do that. It wasn't a good model. So, instead we're actually doing one Libvert per pod. And so, Libvert actually lives inside of the pod that we're deploying our virtual machine in. What that unfortunately means is Libvert has no network access to the cluster or to other nodes. So, we lose the ability to do live migration for now. Once we implement other networking options, we can reintroduce that. So, looking at the virtual machine client tool, this is vert control. One of the things that I sort of skipped over or have glossed over at this point is we're looking at virtual machine instances versus virtual machines. These are two different kinds of records. A virtual machine is kind of a static template for a virtual machine instance. Point being, in Kubernetes world, if you start a pod or stop a pod, you're basically creating or deleting a resource. And so, that's what our virtual machine instance is. It's kind of an analogy to that. But, we recognize that that doesn't really translate well to the vert world of Rev, Overt, and the like. People coming into this ecosystem or the tools that we're trying to translate to this system, that doesn't really work well. So, we created the virtual machine object and that's where, when I'm talking about starting and stopping, that's what we're doing. You actually issue a vert control start command on a virtual machine and it will kick off an instance. It also allows us to connect to the console or use VNC in order to be able to interface with your virtual machine and kind of get a snapshot of what's going on, because you're obviously going to need that. Now, there's two ways to do vert client and that is either as a standalone command, which is what I'll be using, or you can actually use it as a cube control plugin, so I'm straight off of cube control. And time for a demo. Real quick, before I do that, I'd like to explain what my system looks like. So, this slide, other than being an example of something complicated, is what my development environment looks like. Inside the physical machine, we're actually running, and I'm sorry Dan, I have to say a docker cluster. Nearly got away with it, but it's on the slide. Inside of the D word, docker, we're running a vagrant instance, and the reason we're doing this is for streamlining development so that everybody's machine looks the same. We're getting consistent builds and the like, but unfortunately adds a little bit of complexity that I can't get around when I'm showing this as a demo. We'll be using cube control commands directly from the physical machine. We've got a little sleight of hand where we're actually proxying these calls through the different layers here and down to node port. But when I start working on the networking, the edge of the light gray box, node 01, is where your node ports actually terminate, and so I can't reach them from the physical machine. So, I had to explain that before we get into this. So, if we start this off, I think I can just hit... So, I'm actually just running a QMU instance here. Let's see if I can make this bigger. No, I don't want to do that. So, I'm just booting a QMU instance here. We've got two gigs of RAM and just a network device. Just a standard CentOS machine. It's logging in to show what the environment looks like. It's a standard run-of-the-mill CentOS machine. And all I did was take a gig image ID deed from DevZero and then run a QMU install on it and, of course, reskin it with the DevConf logo so that we would have something recognizable.  And I'm actually going to start a simple HTTP server here using just Python, which I wouldn't really recommend, but it works great for a demo here. So, I'm going to use Fort 9090 and, of course, when you run this, it's exposing all the files in this directory. One of them, of course, is disk.img over Fort 9090 as a web server, which will become important in a minute when we start using the containerized data importer. So, here, this is the containerized data importer. And what just happened was I'm using the get tree, as you can see. There's a little bit of cruft there from the video. Ignore the second argument of the last part of it. All it is is a pointer to my get repo with containerized data importer. And all I've done at this point to that is run make manifests. So, it's just a straight get tree you can check out and run directly. And that's all I did here is just deploy these different pieces. So, I've got a service account, the cluster roles that are needed to actually do these actions, and, of course, the controller that is monitoring for persistent volumes or persistent volume claims that match its they're actually annotated. I'll show that in a minute. So, I'll show you what the persistent volume claim here looks like. We're using annotations and that's all that the containerized data importer needs in order to recognize persistent volume claims that is supposed to be taking action on here. So, as you can see, I've got port 9090 disk IMG. This contrived IP address actually points back to my bare metal machine when I ran this demo. And, of course, the key value pair, the key is the Kubevert IO storage import endpoint for telling the containerized data importer where to go to fetch this image. The commented out secret name, obviously I don't need because I'm just using Python simple HTTP server, so it's just going to serve up whatever file it sees. So, I'm going to go ahead and create that. That's created the persistent volume claim already. And we'll look at the pods here real quick to show that that's the case. Now, there ends up being a little bit of lag here and that's an unfortunate side effect of our current implementation. We're in, as you can see, it's now running. We're running the upload for containerized data importer directly through the Kube ABI server. Now, in this case, that's a 10 gigabyte image. We're moving 10 gigabytes through the Kube ABI server. That's causing lag. I'm sorry about that. That is what it is. In the future, we're going to be re-implementing that as its own service endpoint and using different solutions in order to be able to authenticate so that we don't have that particular issue. So, checking the logs here, you see the import has begun. And so, we're going to look at the virtual machine instance itself here. This is what's going to actually be using this persistent volume claim that we're creating right now. So, this is basically the bare minimum that I would really want to define in the first place. I've got two gigs of RAM. I've got a persistent volume claim in this case, which is mapping back to devconf-pvc, which is the persistent volume claim that we're actually creating with the containerized data importer as we're going live. So, highlighting some of that here. Of course, two gigs of RAM. Sorry, I shouldn't pause at that point. I'm basically repeating what it's doing. So, quit out of there and then we'll check on the persistent volume claim again to see if it's up yet. Still not. But for now, let's look at the services. We don't need to wait on the persistent volume claim itself to be instantiated in order to be able to instantiate the service. So, this is what it looks like at this point. We're basically just going to use port 22 as our service, SSH, because it's already running on the CentOS box. And we're going to be exposing port 30,000 as a node port, which means that on the outer level of the light gray box, we're going to be using port 30,000 as SSH. So, over here in devconf.yaml I'm highlighting the incorrect thing. Actually, what I was wanting to show is the label here is devconf.usDemo. And that's just an arbitrary label that I put on. And so, the selector on the service here devconf.usColonDemo that key value pair is what indicates that this service should match that virtual machine. That's all you have to do. So, we can go ahead and create that and it's up. So, I'll show that. So, we've got a cluster IP of<PHONE_NUMBER>44 and the port that we're exposing is 30,000. And we can check the endpoints real quick. So, services always have an endpoint, which is where they map to on the other end. And at this point, of course the endpoint that we're mapping to is none because we haven't created the virtual machine that maps to this yet. So, let's check again and it looks like the import has now completed. So, let's check the logs real quick to make sure that everything went okay. And it did right there. Import complete down near the bottom. And we don't need to worry about the learning for file because we didn't use that. We used HTTP. And here is the persistent volume itself that we're bound to. And it is mapped to the DevConf PVC. We don't need to worry about its name because that will be looked up automatically. So, let's go ahead and create the virtual machine instance now. And show that we have a new pod for VertLauncher and it is running. It's been up for three seconds. And here's the virtual machine instance that it goes with. So, let's check the VNC console here. Login through VNC. And the screen saver kicked on. Don't know how to get out. Actually, no, we ran out of power on the laptop. We never plugged it in. So, yeah, the power ran out on the laptop. Here is a plug. We never plugged it in. So, at that point we were showing the virtual machine, you know, missed it right there. VNC was going to boot up and actually show that we were using the exact same image that we had started with on bare metal, which was, of course, well, we've got a charge now. But it would take a minute for the machine to boot at this point. So, but that was basically, the only other thing that I wanted to show was the service and actually how we mapped from that virtual machine exposed that service because the endpoint that I showed you a moment ago was mapped to that pod for that virtual machine. And because we're mapping the virtual machines, there's the pods IP to the virtual machine, that then terminated at the point itself. So, then we were able to SSH in from the node IP itself. So, something outside the cluster could then SSH into that box, which obviously if you're going to have a cloud based virtual machine is a very essential point. From here, the, yes? Yeah, we're just going to talk about the next steps, of course. One of the things that I glossed over was that we don't, or that I was using local storage on this because it's a single instance machine. You can use other back ends, however I chose not to do that because of the complexity of actually doing that on a single node. One of the things we'd like to work on in the future, of course, is making that a little easier to do and multiple networks, you know, is another thing that I had mentioned that was on our major wish list. But from there, I guess it's time to take questions. Here. Do you see the cloud mainerized? We're bringing in Mike one second. All right, can you guys hear me? All right, I want to make sure I understood the containerized data importer, is that what it was called? Yes. Okay, it's basically just a utility that will go grab an HTTP image of a disk image, right? Yes. But it runs as a container, is that what it's called, containerized? It is a container basically running as a run once container. And that, you know, because as you see, it said completed when it finished actually doing the import. And so what that pod's job was to do was just simply to load the local storage on one end and connect HTTP on the other and just move the bits. Move the bits into the PV. What's that? Just dumps the bits into the PV. Exactly. Okay. And it's doing it in the namespace that you want on purpose. So whatever namespace you mentioned in your virtual machine is where it's, you know, the containerized data importer is going to start its pod and then that, you know, we're not running into permissions issues by doing it that way. And is that the only utility you have today? You don't have anything that'll pull? I was talking to Steve Gordon here and I thought there was some way to pull an OCI image that might be embedded with a kernel and all the things that you want to be able to do that. So pull an OCI image with it's already got all the bits you want? It's the abuse of OCI, but it also would make it nice in that all the infrastructure would be the same, right? I think what you're talking about in that case is using a different, excuse me, container runtime. No? No, no, no. I'm saying like dump a VM image in an OCI image, stuff it in a registry. Oh, right. Pull that in instead of just a random disk image. Or another option would be just pull street from Cinder. Like I always wondered why. So we actually do have registry disks as a possible option here. So, you know, you absolutely could do that. I think that in a production environment, I would imagine maybe that the persistent volume claim would have just a more universal appeal. But yeah, we certainly could put it into a registry as well. A container registry, right? We're actually doing that on our dev images right now. So we created Docker registry, sorry, a container registry. And we, you know, when we instantiate the development environment, we stock it with an Alpine image and a Cirrus image and a couple others, just so that we have base images to do all our testing infrastructure in. And so yeah, we put that directly into a container registry and utilize those images inside Kubevert as well. Yeah. In that case, we actually have a, there was one other pod that was sitting on the, you know, what I would was listing the pods. That's what was doing that. Yeah, one more pod. Any other questions? So this project has been going on for a couple of years. Are there examples that are using this for like primary security or are they just migrating existing virtual machines to make it easier to their existing workflow? And if the former, can you give an example of where it's better than just... I'm getting a lot of echo. I couldn't hear the question. I'm sorry. So one of the benefits of running virtual machines on top of Kubernetes is you get more than just like LXC and C groups to do like sandboxing between different environments. And I was curious like, are there either customers of Red Hat or Red Hat that's using it in an area that they have like pen tested this on direct containers that they have and this was secure enough? And can you talk about that a little bit? So in terms of what I think the question is is address security concerns or with security one of the things we were trying to address when we set this up. Yes, it is more secure. That wasn't necessarily our stated goal in terms of something we were setting out to accomplish. But you're right. There's a lot stronger process isolation when you run services inside of virtual machines like this. However, I would still point out that between other containers, there's no isolation at that level anyway. So you're just really, I mean, if security now if you're looking at untrusted workloads in the virtual machine, yeah, this is great. If you're looking at not trusting anything on the cluster, you're going to need stronger guarantees. Does that answer the question? Yes. So the question was how is it different from in the virtual machine versus just in a container and maybe who cares about that? Okay, so in terms of what's the difference in level of security between a virtual machine and a container? Level of process isolation. That's still gay, are you? I'm sorry. The level of process isolation? The level of process isolation? Yeah, so that is one of the reasons that you would run a virtual machine is because you have that stronger process isolation than you do with a container which you see groups, namespaces, SeLinux. I mean, those are strong guarantees but in theory you might be able to do something. I don't know. So yeah, I mean the virtual machine is a stronger guarantee but that's not what we set out to do when we do that, when we set this up. Thank you. Anybody else? If you don't mind I might try to answer this question. I don't understand it. So I think you're thinking about it reverse like you're thinking about it from a security perspective he's saying it exactly right. Don't think of it from a security perspective but we should tell him what to think about it. Think about it as a tool to pull a VM into an application similar to in a Kubernetes EMO file like that's phenomenal, right? Because now I've got a database living in a VM I've got front ends living in a container it's a way to scope the entire application with a single application definition of Kubeberg. On the other side of that I would say something like Cata Containers is where you're now running a container in a VM it still pulls the VM image it still uses all of the I'm sorry the container image it uses all the container constructs that you care about so now it's a packaging format in that scenario you're adding extra isolation around a container I think that's the way that you think about it a security construct in that now it's an isolated container but I still get all the packaging format advantages you get bringing old stuff that's in a VM advantage which is like the converse essentially Thank you First you owe me about 50 cents Can you spot me? So one of the interesting things I would like to see with Kubeberg is to sort of be able to handle the Cata Container use case in that Kubernetes is running along and finds that it needs additional resources an application figures out that it needs additional resources so it calls into Kubeberg and launches additional VMs that Kubernetes could take advantage of those new VMs to launch more containers inside it Is that being considered or is that So the big limitation to what we're setting out right now is that we're attempting to modify the Kubernetes cluster or not require it to be modified ahead of time and so in order to be able to run a virtual machine as a container that requires a different container runtime than the default so we can't ask an administrator to necessarily do that because that's going to increase the friction in the future if Kubernetes does allow dynamic container run times to be injected in on the fly then we can explore using virtual machines directly or using them in the Cata Container sort of approach where the point is for process isolation. Does that answer the question I'm not looking for necessarily the isolation to be great but just being able to launch more at a certain point and applications run out of resources inside of the existing VMs and it needs to launch more VMs so usually Kubernetes breaks at that point and has to fall back and open stack or some other tool for launching more VMs I just thought that Kuber could be a way to actually oh so resource overrun right I hadn't thought about that that's an interesting angle three more minutes maybe less is there a way to pass like a cloud config as part of the virtual machine instance object cloud init data yes I did not show that absolutely yes we inject that into the virtual machine and you know basically cloud init is its own standard so we're not doing anything special there we're just injecting that data and making it available we do that usually through a through its own volume mount so just to confirm the Kuvert has no access to the VM itself right it just brings up the VM and you're expected to know how to make use of it that's a loaded question no what I mean is like there's no for example if there's no like injection in the cloud config of like an SSH key well you can do cloud init data you can do an SSH key that way absolutely right right but I mean that's up to the user that's up to the user to do what they had to set out presumably you would know that you wanted to do that however so we're exposing the ability to do that and yes you can use that or but you have to know to do that right what I just mean is the Kuvert just brings it up and that's it like there's no management inside of the VM itself in August 2018 that's the answer is yes that's true we are looking at other possibilities in terms of doing a monitor application that would be available but of course you know how do you do that in a generalized fashion if you're building up a generalized virtual machine suddenly you're building you know your own VMware sort of infrastructure so yes we're looking into possibilities for limited cases last question maybe let's get started I never gave this talk in 35 minutes so I probably would run over time but nobody's speaking after me so if you get bored just leave but thanks for coming thanks for staying long on a Saturday evening at DEF CON Graebiots vs Nightmare how could we need this and container changing the world Graebiots they fight Balrogs and for Linux distribution so when I came up with the name this whole systemd drama was going on and Debin got forked over systemd and I thought if they fork over systemd what they're going to do with containers in kubernetes kubernetes is actually systemd done right in a way so that's what the title came from like first time I gave this talk in bruno and I actually didn't shave for quite a bit but I can't pull that off here so what are we talking about so traditionally I'll go a little bit into the history of the role of the operating system and why I think these things matter if you look at inside rathead you often can have this discussion like what is the operating system a lot of people say well rel is the infrastructure I never really agreed with that from my point of view infrastructure like network cables and hardware and things like that and the point of the operating system is actually to run applications it's the application run time because no one runs an operating system because they want infrastructure to run an operating system run an application and they want to abstract the infrastructure so they don't have to deal with it but it's true like there are two views there's the hardware center we're thinking about how do I make the hardware work how to bring it up and there's the application center view which is how do I how do I give an application a common run time and if you look at the historic role of linux specifically in our world enterprise linux coming from a world where like everything was about the hardware computers were big and scars you call them mainframe and they came from a big window who controlled the hardware the operating system and the software you could run on the operating system and you used least hardware ran black box services in not everyone like not every mainframe worked that way but the biggest dominant players in the market worked that way you didn't really have much control and everything was vertically integrated and it was like least hardware and like if you wanted more performance you called them and they locked in we unmode them made funny noises and then activated more capacity in your mainframe and then we had linux and mini computers which I will ignore here to make this shorter someone will call me out on it with linux we had a little bit less vertical integration you still had the operating system and the hardware from the same vendor usually depending on the vendor more or less control on the ecosystem some of the linux vendors had very tight control on the software you could run on top of linux some had a very liberal view and in many cases you would install the GNU environment anyways because you want to tap completion and at least better shell and open systems was the big deal so you end up with controlled hardware vertical integration hardware operating system and often in the tool chains you had some choice of third party software vendors but then that also was like too controlled for many people and with linux we broke both of them we created a model where now you have free choice on the hardware from different vendors and you have free choice on the the ICV ecosystem software you are going to run on top and you have full transparency and insight into the operating system itself the operating system today linux today becomes an abstraction layer across different types of hardware both the different footprints of what we call the hybrid cloud so bare metal virtualization, private cloud, public cloud and it allows you to build a binary once and run it in all these footprints and it gives you that transparency it also gives different players market different ISVs a common way to run binaries, build binaries an implemented standard it also gives you the same interface across different hardware architectures right so if you if you are doing ARM or Intel or power today or what's really interesting from my point of view risk 5 you can use linux as the abstraction layer of looking at that they're not binary compatible but at least their source code and the kind of architecture is called compatible at that level so you don't have to write your software multiple times and a lot of the additional work of making things work in different environments is going away so if you so linux is the neutral run time across different types of infrastructure different hardware that abstracts from that and allows you to be efficient as an application developer to not have to deal with the underlying infrastructure or have dependencies or get locked into vertical dependencies so that's the role that new linux played and in our world that reted, reted enterprise linux played for the common industry right and you know the snarky common is that we allowed people who couldn't afford the sun server to run their IT like they could so that's what we did back in the day now early on the way we managed software stacks in linux we inherited that from Unix and you know when I went to university like we had we had a bunch of of Vax machines and a bunch of big sun servers in the university and a whole bunch of individual linux PCs and showed up but early on it was like what we used was primarily the sun services terminals and we had multiple admins per server and they would compile software locally on these machines right and use a local and then use stow to put it somewhere to find it and your binary path and you cross mount user local or even user depending on what you did from one machine to multiple machines so you shared your binary run time across multiple machines by cross mounting it over NFS it was a common practice like that we did a lot of that and even in the linux machines early on so linux distribution came up as like a way to easily install linux because like figuring out how to make 500, 600 different components work together in specific versions was really hard so linux distribution wasn't meant to make that easy the first linux distribution basically used tar files with pre-compiled binaries for the initials slackware was what we used early on but there were others sls or things like that but then we would get additional software and that usually was compiled in user local and we used stow like some links to put that in the path and then every time there was not that and all that good stuff the problem with that was that you had a lot of dependency on the state of the machine so you couldn't expect that the software would behave the same way the state of the machine at the time where you install software when you compile software because you don't install it you compile it really defines how your software is going to going to behave and that worked fairly well if you had like a few big machines and admins would occur of that and it became very clear very quickly that that wouldn't work when you used PC type machines many machines with linux because it just didn't scale it just didn't scale to compile everything on every machine so the solution was to create binary reproducible builds to package things in a different way not just TARS but manage dependencies and things like that which the original you couldn't and it was just painful so manage the problem by trying to capture more state of your software stack in the binary package you create so RPM and Debian came along you build ones, install it everywhere, it behaves the same way it manages dependencies as all this context you Debian added apps as kind of a transport model for that right here it's something called up to date and later replaced it with yum and what that gave you is really a good way of dealing with a stack complexity however it implemented like the original model is a very late binding you bind on the source code level you compile in place and you rely on that kind of leading to the same behavior here we still do late binding but we only bind dependencies late we use shared libraries extensively and we rely on ABI stability between the libraries to update parts of the stack and make it reproducible it worked fairly well and got us really far 20 years of this I'd say the problem is that dependencies get hard to manage over time a lot of people in the room are dreaded, are painfully aware of that you start back porting things to older versions because you need to keep binary compatibility because you locked in this ABI contract in this dependency model that's a downside and things break if you don't do that an interesting side effect that I didn't really realize for a long time was a problem like back in the day when you compiled and used a local and stored things you could have multiple versions of the same binary or the same stack you could have multiple instances of the same code on a machine in different versions so multi-incense, multi-version environment because it was just separated way and used a local in the move to binary packaging both in Debian and in the RPM world we implicitly turn Linux into single instance single version system you can only install a single instance and a single version of every binary package if you want multiple versions at the same time you have to rename the package so you have to do some naming things and relocation and if you want to experience it look at the Python stack some of the languages support that better than others but it can be really painful and we evolved around that with some nice tooling software collection but we have additional tooling to work around a single instance single version and I never really realized a bigger problem that was we kind of of course we want one version of everything because we moved away from these complex multi multi service machines but who never really got on board with that because they wanted their own version it runs everywhere as long as you use the same JVM right and they always wanted in their home directory their own version of the runtime they might be okay with a shared JDK JVM for the machine but then they want multiple instance of everything else and we had these arguments of like J-Bus should be in an RPM package and no one ever wanted to use that and we were like why wouldn't you use that like you get actually control over your software stack and it's also your dependencies scrap those zip files and we never understood why I never understood why they didn't want that until I realized oh they want multiple different versions on the same machine and that's why they hate this because RPM actually doesn't allow them to do that in a clean way or they have to do these additional steps there was an interesting side effect and I think we'll see how that's going to work in this discussion now with all these PCs and like scale-out architecture and more and more machines RPM and JVM weren't good enough so other tools were invented one of the big things for Reddit was Satellite Kickstart Kickstart allowing you to recreate a software stack on installation on deployment of the machine, automated deployment of Satellite we built some instrumentation around that and central service for content management Satellite server basically was kind of a content management for software streams or is still content management for software streams where you do your standardized approved build and then you deploy that through a Kickstart file and then you centrally manage what gets updated where so you have a large number of machines and you centrally manage the software updates for them, work like a charm and you have things like CF engine and then later obviously new implemented like Puppet which is CF engine re-implemented in Ruby why that makes sense different question or answerable like a newer model so centralized control for configuration management and orchestrating these machines that you are on and that got us to really big deployments that works really well under the constraints of single instance single version or some additional workarounds and the constraints of a binary ABI ABI contract that you have to maintain now the problems in that model like is still is that whenever you have a change in dependency you want something that's newer you can't install different versions of it at the same time to do additional work and with the complexity of software that gets harder and harder today just because of the amount of software and the amount of change I had that with I managed streaming for this conference and the presentation laptops and so I installed them on one day and then two days later I ran the same script which did a YAM update with 86 packages that got updated now this is Fedora not an enterprise linux but it shows you the amount of software that gets updated in a linux distribution in two days it was the kernel and some SSL thing and a bunch of other things so 86 things that happen so basically if you have a large cluster when you are done patching it you are going to start over because it is always moving and it gets into some problems we are late binding we expect that people do this in production so I have a production cluster and I have a running application and I am going to update a library like SSL which is kind of important in production in that cluster and I expect that all my applications keep running I might have to restart them so it loads a new version and I do that constantly because when I am done the next update rolls in if I have 5000 machines and then I want a new version and that breaks over time we do an update because one satellite wanted an update and then we broke open stack because they used the same shared Python library and ABI compatibility was not maintained I am making that up that never really happened but it is just a theoretical example so the problem is that the amount of software the amount of change the scale and managing the stability the ABI stability gets really hard and we call that dependency hell so what happened is that people moved to VMs VMs came around as a big problem one big thing we inherited it came out of windows primarily because in windows in Linux early on we ran multiple services per machine you had a cluster with a bunch of servers you ran your database your business logic in the same cluster and you had cluster managers that would schedule your services and by default you had three tiers with typical at least three machines the database on one the next server and then your front and on the third server and if one failed things moved around and consolidated and you could run multiple things in the same cluster it was really nice in windows they never could really do that you never ran multiple services on the same Windows server and that was a real problem for them because you had to buy a piece of hardware for every service they couldn't multiplex hardware like we could it took too long so VMs took off as a solution to that because with the VM you had virtual hardware so you multiplexed your hardware by running multiple VMs one instance of the operating system of the windows operating system per VM they worked like a charm and we inherited that in Linux because suddenly our customers started operating Linux like that that solved some of the dependency problems because the dependency problem in one service it's kind of manageable the problem is if you're running satellite on one machine we test satellite when we update rel we test rel and satellite and that usually works the problem comes in when you run your own application on top of that with some different version of a Ruby package and you're running another reted package there so you have a single namespace for dependencies for multiple applications and by different developers who want to use different versions or are not aware of the same level of changes in API and ABI you get dependency conflicts a VM source set because you're basically saying okay I'm using one instance of the operating system I'm binding it to my application and I just run a lot of VMs so it's a bit heavyweight but it gives you isolation and reduces the problem it has two downsides one is we call it VMsProl now you have a 1 to 10 number of servers that you need to update so your problem of like if you had 500 servers now you have 5000 servers and you have to update all of them and this whole problem you're done at one end you're starting over that gets worse and worse and people use VMs just like in theory it would have worked in practice they just screwed it up so the first thing is when VMs came along everyone said oh you're going to use image based deployment so I'm going to image my golden image and then I'm going to run use it like an appliance we call it virtual appliance and you're going to update the golden image and then roll out the golden image around your cluster and that's going to be super clean you don't actually have to update things in production anymore I'm updating an underlying library and suddenly my application logic breaks because the developer didn't know that behavior changed, ABI changed or API changed you won't have that, in reality that never worked so most people even if they're using virtual appliances they use them for deployment initially and then they go back and run YAM update in VMs so they are like I've not seen any, I shouldn't say that of course I've seen it once or twice but not at scale at large people are not running VMs efficiently with imaging they're treating them like pets so it's terrible and at the end it borders a couple of years on dependency help because you reduce the side effects of different stacks running in a shared namespace but it sprawled out too much and it just got too heavyweight and any cloud took over it abstracted us from the hardware we don't care about hardware anymore performance doesn't matter anymore it's all about scale and we're elastic it just made this even worse now there's no control at all anymore you don't have to request a VM and wait a week and justify it now everyone deploy things all the time you just press a button in a self service environment and you get a VM and you install something and you run it many companies have kind of like a workflow around that with golden images and some control and you're not supposed to, if you're the application owner the line of business you don't have route and production but in reality what most of them do is that they give you route when you install your software because it's too hard to like figure out how to not have route you're supposed to give it back but in the end that means that your operation people have no idea what this VM is doing no one knows what's in there so it's great it kind of works but you have things changing and you create a CI CD process around it you create DevOps around it works great but the problem is because you're treating it like a pet you're not using the image based deployment you're testing it you're developing it as a version of the stack then you're moving it to testing and someone runs YAM update so it tests a different version of the stack with SSL updated and some Ruby packages updated and then you're running in production and it changes again so when you run in production it's not the same stack as the one you tested and then in production you're updating it still with the YAM update and SSL changes again so it's still like it gets put together in production every time there's a security fix update and people are trying to control it but customers really are afraid of security updates and bug fixes because they introduce churn but there's no way around that because you can't separate like it's a single path forward basically and in RHEL we did some interesting things like extended update support which I profoundly apologize for to our engineers it's my fault but it was a workaround I was a PM for it because customers wouldn't want to keep on they wanted the new features for new hardware and new software so they wanted to keep us back porting things to versions of RHEL in a major release so RHEL 7 runs for a couple of years 12 years I think I'm not a PM for it otherwise you can see but then they wanted once a year a more stable release that gets only critical fixes because we didn't want to absorb the change anymore so we added like another full layer of life cycle to work around the problem and it works but it's very heavyweight now while all this was going on a couple of things changed in the world out there in the industry everyone became a software company because software was eating the world there is no business it's not a software business my landscape will disagree but if you're in larger businesses everything is defined by software your car is a freaking data center on wheels so this value is driving development we used to have a certain control over what happens in IT because there was an IT department that at the end could enforce standard could enforce a golden image but that went away with the line of business becoming a software business and writing their own software and getting the power because they're driving revenue so at the end you win the argument against IT over what to run in few exceptions and we also had a change in cultures I grew up with three TV stations and you had to follow a schedule and if it wasn't on it wasn't on it's kind of weird but it makes you a bit more tolerant about waiting for things but that completely changed newer generations and people and customers are not going to deal with that because we have an on-demand culture and that has trickled through so that has changed line of business behavior and corporate behavior and now bring your own device it's just an example of that who would today accept that the company tells you you can't bring your own phone thank you if you need a proof like me having my own laptop and my own phone it's just a proof point that those things don't work anymore you can't stun at us we have a much more so service oriented architecture it's kind of a buzzword but it's reality we are aggregating services it led to a preference to consume the most current version because developers are driven by getting the end customer value in the line of business so they're not going to standardize on old versions of anything they want the newest version of everything interestingly open source is the default but that also removed even more control because everyone can get everything everywhere and so customers now have the problem how do I control what they download and the biggest challenge it's very creative using a lot of value for a lot of people but it's hard to control if you had back in the day with proprietary software it was easy to control because if the vendor didn't publish it you couldn't get it in our world that's not the point if someone writes it it's on GitHub you can get it and someone is going to use it and we had like cloud taking off DevOps changing how we do things and then application centricity and I wanted to delete that point because I'm making that later so I'll ignore it so one of the effects of that is that the amount of software available and being used is exploding I got this on the internet so it must be true modulescount.com gives you a picture of how fast software is being created NPM is going to create a black hole if they keep going like this so this could all be forks of the same package someone is going to use every single version of that if you think that we had a problem with dependencies to put this into perspective I think Fedora has like between 20 and 30,000 packages in the ecosystem that we package in RPM and I think Debian has like 10,000 more or something so that would be somewhere down here so our ability to keep up with this we can't repackage all of this in RPMs so we have this nice solution that got us really far but it's not going to keep up with the software is that line of business software application developers want to use no chats you can't keep up with that and most of these languages have their own package managers so it's not that like this is going all the way back to compiling well it's kind of depending on what you're doing but if you're using pip install it actually is going back to compiling locally because you can't find the wheel for what you are doing and so it's actually going to compile on your local machine and you're going to depend on your behavior for that and I'm sure that other languages have similar problems with all of them but it's a huge problem so we're kind of like because of the complexity we are back and I would argue that even if you're in a complete like binary packaging complete RPM or Debian packaging world the fragility of the dependency chain right now is basically as bad as the fragility of compiling from source on every machine back in the day because it's too much change and the ABI contract doesn't hold end to end high in the application and I'm not arguing that we should give up on ABI contracts there are a lot of things where that is very valuable on the low level on the root of stability which is the kernel, glibc, gcc those things you want to control but above the interpreters you can't keep up so that puts us in a problem our traditional distro approach was application centricity is at an impasse we can't keep up with the amount of package, we can't even make the packages available and customers just don't care Solomon Hikes found of Docker in the event of the Docker container thing said that a couple of years ago in a talk at their conference in Europe DockerCon Europe or whatever that was called he said no developer using Docker uses the distribution package manager inside the container he's probably right that developers will use the language native packaging inside their container because the stuff they want in the versions they want is just not available from the distribution because it's too much content too much change, too many different versions and I would also argue that just repackaging things that already packaged in a binary format or in a format that deals with the intersection of binary and compiling local dependencies is of limited use so if you have a pipi package and wrapping it in RPM doesn't solve an additional problem it solves the same problem in a slightly different way so we see a change there there's also a problem that testing when you're up in the application testing actually isn't valid if it's not done with your application because most of our customers have more developers than we do and they write applications that use features in these libraries that we will never be able to test so it's not only the amount of software but also the depth of functionality in the software that we can't cover in testing for all of the ecosystem and so you actually need to validate that a certain Ruby library didn't change behavior to break your application you actually have to test it with your application and that further devalues the standardization that we have been driving in the Linux distribution again lower in the stack I don't think things have changed but when you go up in the stack it gets harder and harder because it's not a high value target validating the kernel is a high value target validating glibc is a high value target because everyone is using them and the benefit of like no one compiles their own kernel anymore like back in the day people did that and I don't no one does that anymore and the value of doing that versus the risk you incurring is just not worth it but you know if you're like some Ruby app that is different so containers came wrong and we thought that might solve the problem we had the ability now to run things in different namespaces LXC was there it gave us multi instance multi version environments we could run different namespaces different versions at the same time vserver was vserver was a project or virtuoso was a project that everyone was using like 2011 2012 2011 and that actually like solved a lot of problems because now I could run different things on the same machine without dependency interactions I could compile things locally without polluting my system I could install things from pip or Ruby gems without pushing them into the shared namespace that all of my applications share so this problem that oh I'm updating this Ruby library and I'm breaking satellite and OpenStack actually OpenStack doesn't use Ruby so I'm making this up for the sake of argument that's not happening anymore now that was like the first iteration of containers was looked at as a nice packaging or nice operational tool but then Docker came along and revolutionized how we do this and it's brilliant we had containers which is basically C groups, namespaces and as Elenox but then Docker combined that with a transport format and a copy and write layering model so they added the concept of aggregate packaging and the concept of distribution so the two things that we had so with RPM we had packaging of application and we had some concept of aggregate packaging when I was in in states engineering consulting big deal was installing Oracle databases on REL and they had this Java installer that was just terrible and you couldn't really run it headless or remote so what you did is you ran it once on a test machine and then did a binary RPM just for the whole Oracle install to make it redistributable so you used RPM for aggregate packaging of the whole software stack just to make it reproducible so that's kind of the same thing that Docker did so you build the ones you basically inherit the frozen binary reproducible builds and then aggregate for the whole software stack per application and you have the model of distribution that we had was yum they added push which totally makes sense in the modern world which didn't make sense in the broadcast world but on demand peer to peer world it totally makes sense that you can push and pull things and they had the management of it so in a way you can look at this as reinventing it's like a static binary it's an early binding so you bind the full stack at one point when you create a container and then you have a fully controlled binary distribution of exactly the same behavior and the only interface the ABI interface is moved down to your user space to kernel interface which is the one that we can absolutely keep stable even upstream is keeping it fairly stable it has a great track record of keeping it stable and it's a valuable target because it's the right interface to keep stable everyone buys into that it also lets you move the same by the same artifact through the development test and deployment process so we're building it early and then we're testing we're deploying the same thing and we build orchestration around that to make it fast right now we heard enough about Cryosol just skip this but you know it's containers done right Docker was great they innovated invented a lot of things but they also had a lot of overlap with the underlying system that wasn't that useful necessarily like you know it's not a natural experience for sysadmin when you use Docker it's probably good for developer but for sysadmin it's not ideal and they have too much demon overlap with Cryo we solved most of that and you know you want to know more dance and Nellyn's talks on that we're awesome today and we recorded all of them so you can find them on YouTube if you didn't listen to them another problem is like an application today is not a single container so we have multi-contain applications right that's where Kubernetes comes in it orchestrates multiple containers into a consistent application and of course we're moving from a single like traditionally a single server was your default deployment right that's what we cared about and then you might the cluster was the exception right you would aggregate multiple servers into cluster nowadays everything is a cluster and the cluster is the default the single server is the exception so cluster is a computer again Kubernetes is like how we solve that we call it the meta kernel but actually it's just a meta system d we call it a meta kernel because everyone likes a kernel and no one likes system d but Kubernetes gives you it's basically both the application orchestrator takes multiple containers puts them into a multi application or manages how many instances of what you're running and how they connect and it also is your cluster manager so it gives you a scale out cluster next thing is full service delivery so we are with operators service broker API helm things like that we are putting together the full transport capability right so the problem statement is today I can I can do YAM install IPA and then IPA IPA install dash dash config or whatever like it's a config script and I have a running orchestrated instance that next time I boot my machine we'll come back with just containers and Kubernetes I have like that's a bit hard to do I have to copy around the Git export files edit files run 15 commands to get to the same experience with the combination of things like the service broker open service broker in operators I get to similar experience where I can actually take a full application I see it in a service catalog in App Store click a button it gets deployed in my cluster so I get full application deployment and portability which is really important for scalability and reproducibility and I'm out of time so I'll quick talk for two more slides before Langdon kicks me out so yeah so I'll put a protest in here someone had this stupid idea sorry for that to put 35 minutes on a technical presentation like you can't say anything in 35 minutes clap if you agree so three use cases that matter fully orchestrated applications that are portal orchestrated Kubernetes you still have applications scenarios what I call loose orchestration so you're deploying one container somewhere using something like creating a unit file to start it up in a traditional server and you'll still have pet containers which is like oh I want to do what I did on a single server but I want to do multiple versions of it I want to run different versions of the user space on the newer version of the kernel that's where pet containers come in so you're basically operating inside the container it's an exception but it's valuable if you are on single servers and using things like that you want to pip install things and you want to install a different version of an RPM than you have on the shared namespace it's just namespacing the machine still valuable and with Builder it's awesome so what does it leave out it leaves us with a new architecture where we have an application platform that has containers plus Kubernetes as a new meta operating system extending the old model of Linux into a cluster application centric view it's awesome and if you think this isn't real unicorns are real thank you very much thanks for watching


OK, what do we got next here? Oh, yes, less pointed out something interesting to us, John. Less says, um, have you noticed that all Apple TV remotes have the volume controls on the right side as you look down at them where every other remote in the world has the volume control on the left side? Any idea why? I have no idea, but he's right. I checked all the remotes in my house. He's absolutely right. Like, what is it with Apple and this stuff? Like, why did Apple decide that remotes would have like need to be switched? I don't get it. Let us know feedback at MacGyb.com if you have the answer. You heard him feedback at MacGyb.com. I said feedback at MacGyb.com because I want to know why Apple decided to do it this way. Bizarre anyway.