Inside theCUBE, this is SiliconANGLE's exclusive coverage of SAP Sapphire now live in Orlando, Florida, theCUBE, our flagship program. We go out to the events, expect a signal from the noise, and we're here with Kristoff Struberg from EMC, 15-year veteran at Disney, SAP Architect, now working for EMC. And we have John Applebee, who with Bluefin, and I understand that you guys, EMC, Bluefin, and IBM and Optimal Solutions had a panna race. So John, tell me a little about how that went down and how you guys kind of met. Well, this came out of an idea with an SAP master, Amit Sinha, who runs the SAP HANA marketing program. He said, wouldn't it be cool to do something that was customer centric and showed how, in front of the customer, we could move a customer from a regular traditional RDBMS, I'm allowed to say traditional legacy, people don't like legacy, and move it on to HANA and show how easy it could be, right? Because everyone's like, oh, this is hard. So race from one side of the street to the other. Old stride of the street, broken down, slumming. To in memory, yeah. In memory, modern fast. Yeah, and I was there with, we're good for him, Vijay Vijashankar, who's now, who's worked for IBM at the time, and he's like, yeah, I'll race you to it. Now, IBM, they didn't want to go there. It turns out, yeah, Optimal were up for the challenge. So we said, let's do this, and from a race of time, we turned into a race of business value. So what are the, who can bring the biggest benefits? And just challenge of, you know, turns out you can do this thing in 90 minutes, right? You can move a system on to HANA in 90 minutes. What are we going to do for the rest of the conference? We've got 48 hours to do something new with this information that we've got. Well, before 90 minutes, you're like, we're done. Which is interesting that you bring this up because it happened TechEd last year, right? But people are still talking about the HANA race. In fact, I hear some SAP executives like to use that example to show customers how quick and easy it actually can be done. Well, the problem with a lot of the stuff out there, people love to see real stuff happen. And with social media, things can really be documented. The other problem though, is that a lot of these hackathons are kind of bogus. And people want to see the real, we call it meat on the bone. So, John, you run a consultancy with your brother. And you guys work with only the top, HANA, big disappointment. So, just share with the folks right now, what is the biggest misconception for HANA? In the, when I say HANA, in the real deployments, like the folks who really need it. You work with the top HANA deployments in the world. Vishal and the team does key notes with you guys. You're pretty high profile.  What are the big misconceptions as well? Yeah, what are people missing about HANA? Because people are trying to deposition it left and right.  So, the other stats, put that all aside, it's like the reality of HANA. Well, and there's a lot of misinformation put in by both sides of the fence, right? But the biggest misconception, is it probably that HANA isn't the database, is what most people say to me most often. Like, it's not really a database, it's just a fast thing that makes your things go faster. But, you know, the reality is, HANA is a complete platform of application services and database, everything. It's the end to end. It keeps your information safe, secure, and it makes it fast. So, within the SAP world, okay, if you put the dome of SAP around the deployments, and Christophe, I want you to comment on this, because you spent a lot of time at Disney doing SAP, and it's not a small company, right? So, these big deployments are very SAP centric. Now we live in a world where, you know, you need to do other things. You got cloud, you got mobile, a lot of different touch points. And when HANA was developed, when I talked to Hasu about it this last year, you know, it was developed years ago before Hadoop, before other things were happening in open source. So, what's your take on now, HANA? I buy that it's real, but dealing with other stuff, how well does it work? Well, what I like about HANA is that it's not a one fit all kind of solution, right? SAP understands that. First of all, SAP understands the whole value proposition discussion, that finding the right use case gets HANA into the door in organizations, that is true for Disney or other organizations as well, right? And that's where companies like Bluefin come in, that's what John does all day long, right? It's also a very interesting technology discussion, and that addresses your question, right? The way you can deploy HANA, you can start HANA outside of the critical path, basically. You can sort of use it as an agile data mart, an old word, but it's still applicable to also HANA. And you can start, once you ramp up the internal knowledge in the company, first you have some help from an outside SI, for example, and you ramp up internally in the company, feel more comfortable with the technology, understand how to maintain it, run it, operate it in a data center, then you can embark on more critical things that are within your running your business, relying on it. So, you're at EMC now, but you had that experience. What are the biggest challenges and now that you're on that side of the fence, you know, the kind of consultancy that Bluefin does is pretty high-end. I mean, this is not trivial deployments to migrate. You know, the HANA race fun thing is probably really cool and probably relevant. I didn't see the details, but let's take it to the real world, right? You know, some of the things you guys go into. What are some of the use cases? What are some of those high-profile examples where it's challenging and HANA fits the bill? Well, again, I like the use cases on the side that SAP calls the art of the possible, right? Outside of the norm. And there are actually some examples also here on the showroom floor, like McLaren, for example, uses HANA for something very creative. Same is true for Bigpoint, an online game company who use HANA and they're not typical SAP customers. They don't use ERP internally, but they are using HANA to do something that they weren't unable to do for their business before. So, I like to use these kind of use cases because in most organizations that have run HANA or they have run SAP for a while, they don't think of SAP as a hot company that can provide something that unlocks new business possibilities. Typically, it's running your financials, you're running your HR, closing your books, that kind of stuff that is boring but essential, right? So, bringing that kind of idea to businesses who run SAP already for a long time is internally often quite a challenge. I don't know, John, do you see that kind of dynamic alter in organizations? Yeah, I think so, and I'm going to weave in back a couple of misconceptions whilst I do that, if that's all right. Yeah, absolutely. And I was in a room a couple of weeks ago and I'm sat there with a bunch of people and I'm just drawing up the HANA vision and what's going on. And I stopped and I checked myself. I said, to the room, have any of you guys ever used SAP? No? Do any of you know what SAP is? No? Do any of you care, no? And I realized that in the world of big data, people are not interested in SAP in itself, right? They're not interested in ERP. They're interested in analytic problems. So, that's the second thing, is a lot of interest in HANA is from people that are not interested in SAP's original core business, the ERP side. And then that brings me to the kind of the third one and I'll come back to the use cases at this point, which is HANA is the center of the world for SAP's big data story, but it's not the only thing. And for the kind of use cases that we're talking about, and Steve Lucas was at the big data event last night talking about this use case. I'm trying to get him on for Thursday, by the way. Oh, yeah, well, yeah. Is he good for the cube or what? Oh, yeah. Steve is awesome. Oh, yeah. You'll like it. I'll love it. But he was talking about this same scenario that we're working on, which is, suppose that we want to consume market trading data in real time. We need something to consume that, and SAP has an offering called ESP. That's part of the story. And we need batch loading as well, so an SAP has a product called Data Services for that. And then we need a really fast store for what's happening with trades and sentiment. Today, we're going to pull data from Twitter. We're going to pull historical weather data and we're going to consume all of what's happening in the trading world in real time. That's HANA. But also, we've got five years of history. And that, of course, is SAP's Column of Database, SciBase IQ. And what about all of this sentiment information back to the beginning of time? Well, we integrate with Hadoop. We federate all of that now with HANA. So all of that is available in one model. So HANA is part of this story for a use case with a very high value around how we predict what trades are going to happen, where trades are going to move next and where markets are going to move. I mean, it's really kind of the co-mingulate in multiple use cases. And in a way, it's a consolidation. You mentioned a federation. In a way, it is a federation model where HANA is now a new center of gravity around multiple things. So, hey, I want real-time unstructured data that's coming off the web public sphere. No problem, bring it in. If I got legacy dark data as McDermott ripped off from Gardner, it's no problem. I mean, the dark night is returning. It's called the data, right? I was sort of thinking I thought of it, but it's like, you know. But again, this is an architectural issue. So a lot of folks that we talk to on theCUBE say, hey, you know, we've been saying the modern infrastructure's here and it's not converged infrastructure. It's a new, it's flash, it's new architecture. So how do you guys look at that? When you talk to customers, this modern era is definitely here. What is the key thing? What is the key thing around HANA and what else is in that package that you're seeing? I think the answer is you have to create reference for the architectures for the stuff that show them how it should be done. And it's as simple as that. We need to take, everything is by industry now, right? It's by industry, by use. We take a use, we create a reference for that. We understand which products, how they're going to integrate and then we help these guys on the hardware side build hardware solutions that meet that package and that has to be a package solution. And that's where we come in, exactly. Yeah, talk about that. You're now on the front end, you're on the EMC side, but you have 15 years at Disney, you were in the legacy side, you were the on-prem king for SAP, right? So now, you're on EMC, clearly it added an infinite... Well, yeah, it's true. And some of the things that I used to rely on from EMC are things along the lines that most architects and customer installations for HANA are concerned about today. That is high availability, that's data protection, typical backup restore and disaster recovery. So those are areas where EMC and VCE have shined in the past. There's a reason for the majority of SAP customers running on EMC storage today. I mean, it's not by chance. You know, it's funny, we had Brian Gallagher on theCUBE last week at EMC World, he runs the supposedly dying symmetric business. It grew 10%. You know, so it's... Dying? Well, no, that's what the NAC is. Oh, it's dying, it's being cannibalized by... But what's happening is just the overall growth, it's still growing as people need to store stuff. They need to store more and more data. Data is growing. It's just one federated component of the architecture. It is, and this is great. So we move to misconception number four, if you like. We can keep going. All day long, I tell you. Which is, HANA isn't enterprise ready. Especially with the SP6 release, which is coming. Allegedly, these features are coming, I think. Yeah, I hear that. Probably not allowed to say solidly, but for me, the remainder of the enterprise-y stuff is done now. High availability, disaster recovery, point in time. All of the things that you would expect of a database exist in HANA, and sometimes people say they don't exist, but now I think it's really a full suite. You can back up from your disaster recovery node now, which is a big thing for people. You can stick all your researches on that other system if you'd like to. Because they slow down this one with all their difficult questions. So, yeah, it's all there. So John, I want to get your perspective. I wanted to ask Kristoff first. We had Henrik on earlier from EMC, and we talked again about this idea of speed and performance. A lot of the benchmarking that's been kicked around, whether it's Pivotal, Green Plum on the EMC side or someone else, everyone always has that benchmark. And Green Plum put out a benchmark before Strahd is saying, all right, we are zillion times faster than Impala, which is Cloudera's version of Real Time, which has its own issues, but Green Plum isn't pristine cleaning themselves. So, they know that, there's no damage done there. But Pivotal's going to take that all in. The data issue, right? So, data matters. Where the data is affects the benchmark. So, how does HANA, like the benchmarks all in memory, you got to move data around, you push the processors where the data is, where does virtualization fit into? I think it depends at the end of the day on the use case and the priority of the data, how it relates to the use case for the customer. And that's different for every installation or it can be different for every installation. I like to use the analogy of the time value of data, although time may not be the only metric that is of interest to move data from one location, if you will, to another. Over time, traditionally, however, that's a pretty good analogy. So, if you will, data that's applicable to the running business right now has a shorter period of time to live, right? It's more aligned with the technology that HANA provides. Data that is much larger in nature and maybe in itself doesn't add the requirement of the time element in its own use case may be applicable to something like what Pivotal does or Hadoop, not everybody knows that HANA and Hadoop actually work quite nicely together, especially with SP6, right? So, I love that time value of data because that's truly relevant. Latency, people we had a big discussion two shows ago about IAPS versus latency, completely different metrics depending upon what you're looking at. So, latency matters. And I'd frame that almost slightly differently in HANA terms to how a green plum or on a teaser would explain this. Which is, with HANA, we always store data at the most granular level, we don't aggregate. And we know how fast we can aggregate. And the number is 12.5 to 13 million aggregations a second per core. So, based upon how many aggregations you need to do, and you just need to- Think about that, that's awesome. It's amazing when you see it. And just to put that in context, like on a small HANA appliance like this, that means that if you've got, let's say, bank transactions and you've got three billion of them in two seconds, you can calculate the total average spend for all transactions there. So, all you need to figure out is how many calculations do you need, how big a box do you need? It's almost un-plausible, people don't even understand it. It's like, you know, it's not really a lot of doubting Thomas's out there. No, when people see it the first time, that's what happens. They say, no, that's not true. Every time. So, that's why the sales guys use Amazon so they can get the POCs up and running so they can see as much as possible how fast. And the enterprise cloud now where you can have much bigger systems. In Amazon, you're limited to like 64. All right, so let's- We'll get back to your question originally about benchmarks, right? I think from a customer perspective, and I used to be a customer for a very long time, in the decision process and figuring out what is the right tool, who's the right partner, there are a bunch of different points of criteria if you will, that formalize a decision. Benchmarks is not the answer to everything. It is one criteria. It is an informative one, but it's- It's a sizzle, not the speed. Right, exactly. It's one point in your collection of data that helps you to make up your mind. Okay, you got my attention.  As you said, it's impressive. And the big thing with Hunter, and this is what we try and display every time, is you can get the information in near real time and you can aggregate on it at the same time. So, if you can get the information in, you can ask the questions at the speed of thought. And that's really the limiting factor is, can you get it in and can you formulate the right question? So the only bottleneck in that equation and based on the benchmarks, which I haven't seen, I haven't looked at, I'd love to come in and check it out at SAP labs so I can see it for myself. Who know, invite me up to Palo Alto, and I live in Palo Alto, I should be up there, but it's really architecture. So what is the bottleneck? What is the bottleneck for folks? I mean, it's like the miracle drug, right, for IT. So what is bottleneck to this? Why aren't people just trampling down SAP's door? Or are they? Well, they have a thousand customers in 18 months, right? But I think it's again the idea that something, that revolutionary, that it can be attractive to businesses running SAP or not running SAP comes from SAP. I think that is a hurdle that SAP themselves are about to overcome with this HANA solution. I think the messaging has got to be tight on how it fits. Architectural reference designers, still a little early, but I think that's what I'm finding, is people trying to figure, okay, where is it fit? Is it a square peg in the round hole, or is it, you know, they can shape it any way they want? HANA seems to be elusive that way. You could actually move it any way you want. You mentioned people who use it, but not even have SAP. Correct. Yeah, I think there is a couple of things that will make a big difference in the next year on HANA.  The questions have to be able to parallelize, right? So, you have to build a model that works. And that can be challenging. And the second thing is that from a hardware perspective, the current generation of hardware wasn't built for in-memory solutions. The next generation, which I think there's some of those kicking around that are coming, those are truly built for in-memory solutions. And they're smaller, they're cheaper. And all the hardware, we're going to have Don Bacille on Thursdays, the CEO of Violent Memory Systems, and Fusion IO, obviously they had a little management problems on top there. They've shifted over, now Violent and Fusion, they've proven that you put server-side flash architecturally opens up a lot of creative opportunities for technologists. Yeah, although actually in HANA is kind of the opposite with HANA because flash isn't overkill. You only need the disparate persistence. Yeah, yeah, exactly. So I'm going to add- I guess there's some data areas that benefit from it, but not everything. Yeah, you don't need flash. Correct. So, the answer again, like your questions, you asked a question about the bottleneck, right? In the past where you call them traditional database systems in the past, right? The bottleneck used to be between compute and storage, right? And it required a lot of performance tuning. Now, buffer pool hit rate ratios and things of that nature, laying out the data correctly across spindles, all that kind of stuff. Well, if that is out of the equation, the next bottleneck that we now see with HANA is basically between cache and memory. David transfer. Yeah. It's all about getting data closer to CPU circles. And it's all the traditional, think of L1, L2, L3, cache testing. You start a whole other economics class, Christof, the future value of data. Yeah. You know, we can do a little, you know- And actually, I wrote something together with Gardner on that very topic. I'm passionate about that too. Yeah. Okay, well then I'll go start with a question for both of you, start with John. What was the most amazing thing you've done with HANA? The most amazing thing. Ah, I'm curious. Yeah, can I talk about it? Yeah, that's the question. You can generalize. John Doe, and... I think this one that we're working on at the minute is amazing, which is around financial services. The speed of thought, the one that calculates the balances of all the transactions. Actually, I mean, I love that one because it's such a great example because they came to me and they said, it's not running that fast on HANA. And I said, boy, there's a reason for this. You need to do a design that's not quite right for it. And it'd been taken quite a while to run. And then, like I said, the balances... So what were they doing? Taking the data, coming in? Can you explain that? What are they doing wrong? Yeah, no, what were they doing with the product? What was the solution? The solution is around ad hoc analysis for transactional data in financial services. Real-time financial data? Or not near real-time data? Generally, it doesn't really matter. And for analysts, they're not... What's transactional data, like... Transactional data, but taken in overnight because actually they're doing kind of research questions rather than real-time questions. So it's not time critical. It's a bulk of data that's pretty recent. Yeah, yeah, exactly. Recent, yeah, up to the day. And you want to ask questions around average balances, time, demographics. So free-form questions? Free-forms questions. No modeling, no front-end stuff, just like a Google search engine. Yeah, like that, exactly. And... Not run the BI query from the medium we have. Not bad to pass it. Right, no. Not a dark data query. They don't know the questions they want to ask. And they want to join... They want the flexibility to join. So I suppose you want to join the weather data for the last five years. Let's pull that in. Most people get to the right question after they've asked a few questions. That's the wrong way, right? That's the wrong way. Right, yeah. To your data point. Okay, you fumble through. It's kind of like, okay, that didn't work. Give me another one. Then you kind of get the creative juices going. Yeah, exactly. And they want to be able to do some of the stuff themselves. So the most amazing thing was what? The most amazing thing, I think, was the realization that they really can ask a question across five years of transactional data. And they can get an answer in a couple of seconds. And they can just keep asking. I don't think they believe that when they saw that. It's like, well, I can see average spend by customer, by sex, by demographic, by area. And I can visualize that. Did you get to see the people's faces when they were doing the queries? Or did you do that when you were... I wasn't there, unfortunately. Yeah, you saw the positive feedback. That would have been interesting, yeah. I would have liked to see that. Okay, your most amazing thing that you've done with Hanna, or been involved in. Yeah, well, I think the most amazing thing that actually happens right now, I can't talk about. The cube, I know, it's pretty odd. I know, exactly. But it's also something innovative in the financial space that we're doing. And there's a huge interest, I would say, because the analysis of numbers has immediate value to these financial companies. So from a business perspective, that drives a lot of interest, right? So I can't really talk about the details. But aside from that, I think we're worked also together with Blufin on something with VCE, right? Yeah, and we've been having some fun with that. Yeah. We said, basically, let's take a really easy, what should be a really easy example for people to understand, which, you know, Blufin runs SAP. We run it wall-to-wall, blah, blah, blah. Yeah. We got that checked off, right. Checked the box. Yeah, and we have a few processes which are a challenge, doesn't everyone, right? Probably our big one's our planning process, which is all of our resource management's done in SAP BPC. What about orchestration? I mean, that's a hard problem when you're dealing, you know, at the VCE, part of VCE, you've got VCE and you've got all these deployments. How are you managing the orchestration across all the systems? Orchestration of the code. So if I'm a developer, I don't want to, I'm a developer, I don't want to actually, I want to write, I'm writing an app. I want to manage all the infrastructure and orchestrate all my services across the infrastructure. So I'm thinking, okay, I want to use a little hon, I want some speed, I want some nests. I'm actually a developer. Infrastructure as code is a DevOps kind of mindset. That's a good question. I'm in the space of SAP, I mean, traditional SAP, you're really bound to ABAP. Still, I mean, SAP tried Java, it really didn't take off that much. So ABAP is the answer there. And in the space of HANA, it opens up a lot of, you know, other opportunities. And ABAP. We see a lot of developer start-ups, actually.  Have you seen any hot start-ups come out of the HANA space? I'm not aware of any, but... John, have you seen any?  I'm going to get in trouble here because I should be paying attention to this stuff because I know those guys. No, those guys really well. Are you too busy running? I'm busy on other stuff. But I talked to one last night. I thought it was... Don't pick yourself. Too hard. I think that's a pretty high-end stuff. There was one I was talking to last night that was pretty cool and we had a short conversation and he told me how they were doing refuse disposal or recyclable. So you can tell it, there's an iPhone app, you can download and focus. Only I can remember the name of the start-up. Pretty big problem. Yeah, well, yeah, but I spoke to him briefly. And basically, you open up the app and you tell it, what do you want? So I've got a, you know, I want to recycle this MacBook Air. It's not very good. So, and they'll tell you where you can recycle that. And they're looking from what I understood to monetize that by taking that out to the suppliers like HP and so on. And then using that big data information that they're gaining from the customers to help the suppliers sort of refill. You know, you should really buy a new toner now. I see you just recycled one, for instance. Using the power of HANA to give them analytics on that. I thought that was interesting. And you guys are working together. Obviously, the HANA race we had mentioned is some of the cool things you guys do at events. And that was really going to demonstrate the value and kind of give kind of a face to HANA. Just in summary, our last question for both of you guys is talk about, just for the average person out there who's trying to figure out what this is all about, HANA and what it means. Not just the geeky stuff, but what it enables. You guys are exposed to some pretty high-end use cases and pretty intoxicating when you think about some of the new creative things that are happening. What would you share with them around, peek around the corner and give them a little view of what you think's happening with this trend and what's going to enable? So, I actually send customers all the time to SAP websites. Because I mentioned earlier this, I call it the time or the value dynamic between HANA as a technology and HANA as a business value and driving new revenue and things like that, increasing customer satisfaction and all that kind of stuff. So, SAP obviously realizes that and has made available a website that everybody who is in the field of HANA probably should know and it's saphANA.com, very simple. I like that website because of two reasons. One, there are specific success stories, little videos out there that show what customers have done with HANA, what is possible on the creative side. Secondly, there is a growing list of use cases that are organized by industry verticals. One of the SAP mentors, Rukshan Omar, she's managing basically, she's the gatekeeper of all these use cases. Organized by industry verticals, researched well, documented just high level ideas to spawn sort of the discussion what is possible with HANA. At the end of the day, sure, the technology may be interesting to people like us, but it doesn't matter, right? The business case solves a problem or generates a new business revenue stream, if you will. And that's where I think the meat of the conversation is to be drawn first. What are you personally like? I mean, from a personal perspective, you've had a storied career in IT and you've seen old school project management, big deployments, a lot of integrations. You know, a lot of people made a lot of dough, right? And time to value was months, now it's weeks. So there's a new market here, right? It's accelerated all around. What I like for IT organizations specifically about HANA is it gives IT organizations a tool or a message to be more relevant to the business, right? IT organizations throughout industries struggle with the fact that they try to get out of the Nike swoosh job description, just do it, right? And try to add more value to the business. Well, the business may not be available or aware of a tool called SAP HANA but what it can do, that's why I like these websites so much because they help IT organizations to educate themselves and they can engage the business, draw business people in, draw IT people in, draw data people in and have more creative workshops internally before they reach out, for example, to SAP or their favorite SI on value discovery workshops. Great, awesome. John, you kind of have a rope, you're pulling customers into the future, you had to go into the future a little bit in your mind, see around the corner and visualize these architectures. What would you share with the folks out there around, what sort of cool things you're in seeing that could be possible that yet have not been formed in terms of opportunities? I think the, if I think of the biggest challenge with HANA, right, it is making that journey to understand what you could do with it. So that's the big challenge and they talk about it as being a non-destructive technology and I'm not with SAP on this. HANA is a non-destructive technology but it's a technology which can disrupt markets. And that I think we haven't really seen yet and the things that you can do with it will blow your mind when you get there. People get afraid when they see something they don't understand and it's always the same question. What do I do with this? What can you do with it? A new weapon, a new opportunity, a new tool. And I think the real value, and I think McDermott got it really great today when he brought NBA in particular on stage, right, which is humanizing data. Because this NBA information, it's so easy to understand. Awesome. As an individual and I think- People can relate to fantasy football and stats and having stuff on the fingertips when they're watching the game or knowing, hey, pull the guy out, he tends to choke in the fourth quarter. Right, yeah. And I think that the real- You heard that up there from the Jed York. No, that was funny, wasn't it? But I think the real disruption in the markets is going to come from the humanization of that information and what change it brings. Humanization, guys, great conversation. I'd like to go more. We could go more, but we have to do a wrap-up towards the end of the day. It's already pushing 26 here on the East Coast. John Appleby and Christoph Struber, thanks for coming on theCUBE. Appreciate it. We'll be right back after this break from the wrap-up inside theCUBE. We've got EMC and we've got Hannah here having some great conversations. We'll be right back live from Sapphire. This is SiliconANGLES exclusive coverage. I'm John Furrier. We'll be right back.


Okay, we're going to get started with our next session. We've entered the longer session part of our day. So these sessions are now going to be 40 minutes until later this afternoon. So without any further ado, I'm pleased to present one more from Red Hat, who will be talking about the merger of VMs and containers. Thank you, Roman. Hi, everyone. So Mr. Michael, maybe you could open the door. Hi, everyone. So, yeah, that's great. So, hi, everyone. Glad to see you all here. As for another Z, I'm Roman Moore. I'm working for Red Hat as a software engineer. And today I want to talk to you a little bit about how we can arrange pet VMs and containers. And for all of those who don't know what pet VMs are, pet VMs are basically the opposite of global VMs. And that means whenever you have an application inside a VM and you think a lot about how you can migrate it or keep it up without shutting down the VM, then you're basically talking about pet VM. And on the other hand, you have containers or cloud VMs and you're obviously there. You don't care that much about the state of the VM. It's just more about spawning the VM again if it goes down. But now let's start with the actual topic. So first, I want to talk to you a bit about how we can arrange pet VMs and containers together in with different management solutions. Then I want to tell you a little bit on how we even care about it. After that, I want to show you how typical visualization and container stacks look like in the host. Then finally, how one could try to bring VMs into Kubernetes, follow a little bit more of a question than is how we can bring VMs into a container if you want to. And finally, I want to give a small overview about Keyboard, that's the project I'm working on where I hope is a good project which tries to combine all the insights I want to present to you first. So let's start with a little story about how we can manage VMs and containers to get us. Let's suppose you're an administrator in a medium-sized company and you're used to your O-word or VM-ware, virtualization stack, and there's an engineer coming to you and he says, yeah, I have this new containerized scalable application I want to host with Kubernetes. Could you give me, I don't know, maybe 10 VMs, make sure they're running in different hosts and I will do the rest for regarding to that. And the guy goes away and finally a few days later someone says, hey, the application isn't up, can you take a look? And you look inside all the containers and VMs and there you see, hey, the guy was really good at business doing, he created his Kubernetes deployment in there, maybe the HA, made sure that his application can scale up and down in there. But with this engineer, it looks pretty much like what you're doing in the host level already with your virtualization stacks. He's collecting metrics, he makes sure that specific components are always running, yeah, he's monitoring all that. And so start thinking, maybe you as an infrastructure guy should also care a little bit about what's going on there especially if something isn't right. So you start looking into that more closely how you could start managing that. And one thing which is obvious there is that you had different layers of services, image error, and they're all managed by different schedulers. For instance, Kubernetes has its own scheduler, over it has its own scheduler, remember there are schedulers and you have to, and so, and even the applications inside of Kubernetes do their own kind of scheduling regarding to scaling up or scaling down regarding to traffic and so on. And when we're doing virtualization, like we have it here, with containers inside or a container management framework inside, it's pretty hard for instance, the DevOps team to just care about the container infrastructure. You basically have to go back to the host level and the VM level to make sure that your containers are properly placed around the stack. So as an industry, you're not thinking about removing that layering to get more out of your VMs, more performance, make the placement of the services and containers easier and you might think about placing them next to each other on the same host. And there you can have, in the area you can run into a lot of problems. Most of the demands for different management solutions are completely taking over the host. So even just trying to place two different management solutions on the same host is very, very hard. Another thing you could do is you could try to split your data center. So yeah, let's keep the traditional data center where we're caring about all our VMs and only put them on one kind of host and put all the containers on another type of host where you're managing everything with, let's say, communities. But that kind of solves at least these problems we've been mentioning before, but you still have all the applications regarding to monitoring everything, collecting the metrics, making, you have two different systems which you need to keep HA also. For instance, Overt or Wendware needs to be HA, Kubernetes needs to be HA. And so, I mean, you can do that. It solves some problems, especially relating to host utilizations on the special part of your data center. But for instance, if you have a lot of spare resources on your VM part of the data center, you can't just move containers there. So it's still not optimal. And another approach would be, hey, let's start running VMs inside containers. The advantage with this approach would be that containers have much less assumptions than pet VMs about lifespan. If it goes down, you can just start it again. And so it is a kind of lower layer for all the services which you could use. And the VM, which for instance needs to migrate, can be migrated off of a container before the container is actually removed. So that could fit pretty nice into one management solution. Or you could go even a little bit farther, but that is more the cloud approach. You could start putting inside VMs again containers. For instance, Google Cloud Engine is doing that, if I remember correctly. I think they're using Borg, which creates containers where they're starting VMs. And these are the VMs you can rent when you're renting VMs. And inside there, whether people are doing that, they are deploying Kubernetes on top of it. And there it completely makes sense because there is a different assumption there. You're just talking about cloud VMs. And also about the separation of concern. Google as a cloud provider doesn't care what the application does.  So that is probably not the one here, but it's a nice thing you could do. So to sum it up a little bit, with some facts is why we really care about bringing all that together, is for one kind of applications which have specific needs. There are those VMs which need really high performance, those services, which really need high performance. And if you have a lot of them, virtualization overhead might matter. Another thing is that you have your, I don't like calling them, I guess applications that you have your monolithic applications, which in most cases are even until now your cash calls for your company. So you can't just leave them behind. And then you have the microservice-based applications. They don't care much about performance, but they are already cloud ready. And you have your DevOps teams which just want to scale up and down. However, it pleases them. They don't want to ask for new VMs to administrators which container is allowed to run on the same host or the same VM or so on. They just want to deploy everything when you run. Then you have management needs like hyperconversions. As you've seen before with the difference of approaches presented, when you have a lot of different frameworks, you have a lot of different components which are very hard to administer, especially when you run them on the same hosts. So you would end up with a lot of small clusters which are there for different things. One cluster would be there to manage your VM infrastructure, make it a hell of a little cluster would be there for just giving you the network provider for that manager. Then you have a network provider for your Kubernetes stuff and so on. And it's really hard to get all that together when you really try to manage two management layers in each other. Another thing is, yeah, if I've already mentioned it somehow, it's high availability of all the services. For instance, Kubernetes has a very nice concept of high availability built in. So if you're not trying to teach your applications which are ready for the cloud to work with your virtualization environment, you could instead rewrite your virtualization infrastructure that you can just deploy it as a container and make use of each eight concepts which you can use from Kubernetes out of the box. And when we can achieve a goal of combining those two, it costs a lot of money. Less components to manage means less things which can go wrong and makes everything easier. But that's not the only kind of benefits we can get. Another benefit, of course, is that when you look at the host stack of virtualization infrastructure and of container infrastructure, they kind of do similar things in slightly different ways. But for instance, in the Ovid stack, you have VDSM, that's the host demon which is doing monitoring of VMs, does storage mounting, disk provisioning. Then you have network providers inside VDSM. And then Kubernetes, you have the Qubelet which is the demon of Kubernetes. And it does pretty much the same thing. It does monitoring, it does storage. It somehow makes sure that network is there. Of course, especially for instance, network in the container world, you don't care that much about layer two networking, but it also doesn't hurt, for instance, if it would be layer two for them. If it still provides the appraisal, and so you could then, for instance, use your networking stack for both, for VMs and Kubernetes. And when you don't go further down a little bit more, there are also some differences, of course. On the virtualization stack, you can have something like Libre or Cuyuna. Which is very VM-centric, and there's a lot of knowledge in it which reflects how you keep VMs alive, how you can migrate them. And on the Kubernetes stack there, you have something completely different that's the container and the implementation. But if we can somehow manage to keep the Libre knowledge in the picture, or the Libre knowledge in the picture which Libre provides, and still can run everything in ports, it might be a nice solution which we end up with. So we actually tried a few different approaches, or I mean we didn't try all of them, but we at least played through a few approaches on how we could combine Kubernetes and Petlion's. And one very obvious approach would be to just try to extend Kubernetes, to support everything a VM would also need. So instead of parts it would then also be able to start the VM, you could say, okay, I want immigration control in there, and no hot plugging and so on, we could all at least try to add all of that to Kubernetes. But I think that we basically destroy Kubernetes. Kubernetes is so nice because it doesn't have to care about this. It's a very nice low-level sandbox which is really well suited to clouds and to provide basic rollout of services. So another approach would be we could try to, instead of teaching Kubernetes everything about VMs, we could try to kind of mutate parts to VMs by just replacing the host team of Kubernetes, that's the KubeNet. And there are two approaches out there which are trying to, or two projects out there which are trying to do that, one is Hyper and the other one is rootlet. And they're both trying to replace the KubeNet. And instead of starting a part, you're basically starting a VM, where in one case of Hyper, you're then running the part inside there, or in the case of rootlet, you're just having a playing VM then where you can do whatever you want with it. And that kind of works, but you really don't have any real way of expressing your virtualization needs on a cluster level. It's really, you kind of have to fit everything in the automated data and then extract it there and it kind of feels hacky and it gets even harder than when you think about Numer support and the migrations and stuff like that. You can't just easily pull that back to the scheduling layer, to the cluster layer regarding to scheduling and representing that nice. Another approach which you could try is to keep the KubeNet like it is and just have one container inside the part which spawns a QEMO process at the end. A VM is nothing more than a QEMO process. Of course, you have all migration possibilities and everything, but QEMO does all that for you. It's just a process at the end. But then you have finally left the problem. Yeah, I have the QEMO process in the container, but how do I manage it? I mean, I want to do a plug now, so I have to connect to it and so you have to start the infrastructure on it. The simplest thing, of course, would be to put also the update logic and everything inside a container next to the port or into the port. But that's kind of hard to manage and again, you don't get any benefits from Kubernetes regarding to managing all that. So what we finally came up with is trying to build VM management on top of Kubernetes. And Kubernetes helps us there already because they've seen, hey, there's so many people which are trying to do their stuff with Kubernetes and we can't manage to integrate all that in Kubernetes. So they're decomposing it and one thing they, for instance, introduced our third-party resources. That means, like a pod, you can register your own objects in the API server. You can post them. You can listen for changes on those objects as well. And we started doing that with a VM object. This also, and even better, Kubernetes extracted the core code for doing that as some kind of an SDK. So we can really use the core Kubernetes code for expressing that. And then we can also create our host demon which can work with the same principles as the keyword but just has to care about Wemston. I will tell you, I will have some slides later on where you can see how we're actually doing the placing. We can fire this, since we have our own demon now, we can fire this to let the root do the root stuff. And so we can basically build our own pipeline next to the Kubernetes pipeline after pod is scheduled in a node. And we can reuse the complete Kubernetes infrastructure there. And yeah, as I said, the big question is how do we find a place to VM inside the running pod because that's where you end up at the end. So you let Kubernetes do what it can best. You just let it start a pod. And when the pod is running, there's still no VM inside. Now, only when the pod is running, your host demon for the VM is notified that there is no pod and is trying to start the VM. So you kind of have to bring it inside the pod. And there, you can ask yourself a few interesting questions. First of all, does it even make sense to have a VM with a pod? And there are, that was a very hot topic when we started with Kubernetes, if it's even possible or if it makes sense. And to answer these questions, we have to look, we need to have a look on what containers actually are. And from my perspective, containers are two things. One, it's just a tool for resource isolation, for security reasons, for resource usage. And the other thing is you can group your application with different containers together. So it's not a logical thing on how you want to view on your applications. And when you look at the technical details, C groups are just, containers are just C groups and namespaces. And so the next question is, what does Kubernetes really care about? Does it care about this one? Or does it not care about the logical representation? And yeah, I mean, it's kind of both. It's, of course, at the end here you have containers and you have namespaces and C groups.  So you can, of course, use hostpids with your pods. You can, of course, use hostnetwork if you want to. It would still be containers and it would still be pods. So for who we need this, that's fine. That's very, I think that doesn't sound like very much, but for us that's a very, very important effect because it gives us some freedom on how we decide to bring VMs into containers and how much we want to bring them in. For instance, when I think about device hot-plugging, if we can create our own C groups just for devices where we allow specific stuff. It doesn't necessarily have to be the C group of a pod. But on the other hand, for instance, the pygmy space of a pod can be very important for our VM just to ensure that when the pod goes down that the VM also goes down, for instance, stuff like that. So we can choose here out of the best roles, I think. Whatever we want. And further, when we now go back to the VMs, Librit already puts VMs half into containers. So it's not only possible, it's what we kind of have already anyway. So that's nice in the first glance. It kind of is also a problem for us if we want to freely, freely decide where we want to bring the VMs so we are also very happy that Librit is an extremely flexible project and you can also disable it that it tries to manage the C groups. Firebase has even more hooks. Like if, for instance, you can say instead of directly calling QEMU, you can tell Librit, okay, when I want to start a VM, please don't call QEMU directly, call my wrapper binary before so that you can do additional stuff. And that's the final piece we need to bring our VMs finally into the pod. Because now we can freely decide where we want to start our QMU processes. In which C groups, which namespaces is completely up to us and Librit still provides us all the high-level virtualization functionality we need. And that finally combined, all that combined leads to Librit, or to QBrit, where we tried to final that. And before we look on how we started a VM in QBrit, it's probably good for everyone who doesn't know how to have a look on how plain Kubernetes starts a pod. And there is not so much going on, actually. You have a tool called QubeControl. With QubeControl, you can just post the pod definition to the API server. When that's posted, the scheduler sees, oh, there is a new VM definition, pod definition, so I have to react to it and schedule it here somewhere. After it has done that, the QBrit sees from the API server, oh, now there is a pod scheduled to my node, so I have to take care of it and start it. And that's basically all. That's some of the things I like very much about Kubernetes that although it's, when you look at it on the first glance, it looks so complex what it can do and how it works, but at the end, that's already one of the core things Kubernetes does for you, and it's really simple. In QBrit, since I said we're just trying to build on top of Kubernetes, we just added a few things there. Just two components mainly, the WordController and the WordTendler. The WordController is a cluster-wide daemon, which is watching for new VM specifications which are posted. And the WordTendler is our host daemon which sits next to the Kubelet. And when you now start, try to start a VM, you're not posting a pod, you are posting a VM definition, which we implemented based in third-party resources. That's what the WordController sees. The WordController is now with the VMWatch loop. The WordController is now creating a pod out of it and posting it to the API server. That's what the schedule sees. It's then scheduling the pod. The WordController sees that the pod is now scheduling and running, updates the VM definition that it was assigned to a specific pod, to a specific mode. And there, WordTendler now says, oh, this is a VM which is for me. So we're basically following the same principles Kubernetes is already using for the pods, we're just extending it that we can schedule something on top of the pods. And the beneficiary is that we let every component do what we think they can do best. Kubernetes just has to take care about pods. We have our WordController unit there, which is doing the translation between the two layers of Kubernetes and our VMs. The WordTendler itself doesn't even have to know anything about pods. It just has to look on the API server for our VMs, which appear and are meant to be assigned to it. And it can then move on, everything, pass on everything to the Word to start the VM. And it doesn't have to know anything about C-groups or namespaces or pods even. Really just plain VM management. And the Word itself just does what it always does. Just starts the VM or updates the VM. But since we are using this emulator tag in the Word, we can, before we directly call QEMU, we can take all the QEMU arguments, detect the namespaces and C-groups on the host, completely isolated from all other components, which we want to use and start the QEMU process there. I think I was way too fast, because I'm already at the end. So we have a demo up and running on the Internet on Qiverge. We have a demo up and running on the Internet on Qiverge. You can just try the one line of demo here. It will spawn a VM for you, and you can play around with our Kubernetes experiment. And you can also visit us on GitHub, and we'll be happy if you contribute, or try to understand what you're doing, and give us feedback if you like it or not. And thank you, and questions. So are there some questions? Yeah, please. So you're injecting the QEMU process. Sorry, I can't hear you. You're injecting the QEMU, even the port die, if the VM comes to that. How does the VM die? So the question was, since we're using the namespaces of the port for the QEMU process, what happens if the port dies or the VM dies? What's going on there then? So in the port where we're moving the VM, we have a shin process, which is basically just an infrastructure process which keeps the port open for us. And this one also detects when the QEMU process is started there. And when Kubernetes wants to stop the port, it first stands a sick term, and you can react to that signal, and you have a specific amount of time you can specify, and we're forwarding that sick term to the QEMU process. So the QEMU process gets it normal, shut down, signal it's used to react to.    In that case, nothing, because we disabled it from the configuration. It will just be ignored. The question was, what happens to the LibVirt XML? In the LibVirt XML, when you're managing C-groups, you can specify how much CPU memory the VM can get. And what happens if we disable that? So the answer is it just gets ignored. Kubernetes itself provides the same mechanism, you can add it to the pod, and all the VM definition will pop in the logo. So the pod gets it too. Okay, I think that's it. Thank you for your attention.