Hello everyone, this is Daniel Alvarez. I'm working for Red Hat For the past five years almost six years already. I've been focusing mostly on on Neutron and OVN and today together with my colleague Luis. We're going to introduce you a new project OVN BGP agent and Let's get it started. Okay, so we're gonna start with a Y Where we're starting a new project? So you probably seen in the in the keynote Today and yesterday there's been We have seen a trend in the data centers that typically We are seeing this very frequently like the shift is towards, you know pure L3 deployments where The layer 2 is normally Just ending at the top of rack level So beyond the top of rack Everything is routed and this is how the underlays are are trending to and We are seeing as well needs for scaling Typically, you know in we have seen this today as well like not not just increasing the number of nodes for for the for the sake of it, but also having better maintenance a better day to operations and You know just usually having multiple smarter clothes versus a large deployment with a lot of nodes this is something that is typically preferred as well and We explore what is available upstream and there's a couple of projects upstream that you're probably familiar with one is Newton and amic routing and Networking BGP VPN. Those are the two main projects available upstream and There were some gaps there. We wanted to to you know to explore how to close them and We will walk through it a little bit later on on the presentation So we decided to to start a new project, which is basically a building block. I could serve to Actually can be integrated in those two projects We of course we open source it so it's right now in the under the open Dev umbrella There's gonna be links at the end and and essentially we what we did is to start something a proof of concept display around to gauge the interest from the community and the users and Basically all of you can help, you know with a future direction of the of the project and we did it under one premise Which is essentially not to to be intrusive to any existing projects. We didn't want to do any changes either To Newton or Corvian. So we wanted to to have something just you know from a deployment perspective and This agent that we were going to talk about today Which is minimally invasive so Minimal changes of course it has some gaps and limitations We're gonna walk through it to and serve as a functional block for what is coming next Okay, so we'll start why we want to move away of of the layer to so Normally the the aggregation layer would be the demarcation point between the the L2 and the L3 domain and You know there there's some issues with with large broadcast domains like failure domains or You know typically broadcast traffic Convergence that you have a virtual IP and it can be moved to some of the nodes that there's gonna be gratuitous harps There's gonna be a lot of devices in the network Learning those the new location of the of the virtual IP Decom issues memory in the devices This is the typical layer to issues that you know probably most of you are familiar and have experience at some point And there's a lot of protocols involved right like You know for achieving bonding LACP you need to configure LACP in the fabric You know spanning tree board protocol those usually extend beyond you know villain and they spread across villain stance and They could cause a vast increase in the in the traffic level like broadcast multicast, right? Gonna walk really quick through this and This is where we are trying to to shift to Essentially bringing routing all the way to the to the lowest layer of the hierarchy all the way to the top of rock and down to the compute nodes there's more diagram here, but This is this is a you know kind of like a how an MP church could look you would have your rack And that is the L2 boundary and that you can host workloads from different clouds and Interconnect them somehow but essentially the the point here is that all the traffic will be routed down to the top of rock and the advantages are of course reducing the the size of the layer to domain reducing You know therefore the the failure domains faster convergence we can achieve load balancing and and You know instead of using traditional layer two bones We would be using ECMP bfd for failure detection and for this we need to run some sort of bgp like routing dynamic routing protocol in the fabric and What we are going to show is with bgp is the protocol that we selected and One of the advantages of using bgp is that most of the time what we are seeing today is that bgp is already being running there in the fabric, so it's kind of easy to Plug in there, right? Another advantage could be you know by doing this avoid static configuration in the fabric You don't need to go and configure static routes in the fabric and maintain them Things like prefix delegation for IPv6 a little bit too complex to implement. So this sounds ideal. Of course, there's There's also some pitfalls like with everything. You probably are familiar with it with the outage last year October I think you know when nobody of us could access Facebook or WhatsApp and that was the DNS service or Facebook They were they were disconnected. So the the bgp routes Were withdrawn so that the DNS service were actually up and running and they were functioning but the the routes were withdrawn right so There's these is not free of problems Okay, real quick. It's a little bit messy as like but The point of this is to show You know what up some of the benefits of of having bgp running in your in your open start cloud and You could have applications that live in different clouds So you achieve better resilience you would have active active Basically, you are announcing the location of a virtual IP from two separate places and the fabric will do its magic to To do that at the traffic either to the one or the other so that leads to Disaster avoidance and recovery you can achieve geo redundancy and Essentially you are avoiding normally L2 a stretching and all those problems that come with L2 when you want to spawn across different sites or geographies and So it's it's about not like there was some some reference this morning Well, not just growing your cluster, but growing your cluster and expanding it across, you know, or taking it to the edge Spanning multiple multiple physical locations again, like easy for simple day-to-operations Usually it's gonna be easier to upgrade a smaller cloud or drain, you know, just migrate your workflows from from a smaller cloud So it should simplify the day to operations And just so Luis will deep dive in in, you know to the actual implementation details and Floats, but we are not touching the east-west traffic. So OVN is still Into picture and and all the east-west routing keeps working on the overlay So we're focusing mostly on on workloads on provider networks and floating a piece Although we don't want to leave behind all the use cases like tenant network advertisement with EVPN and whatnot and We are relying on the on the BGP in the fabric Typically with the spinal lift topology to connect for north-south traffic How we do this? So we chose a far as the routing suite. So there's a lot of tools there We can do BFD able. Luis will dive into the details Now what we are doing is we are running a BGP speaker on each node So the PR would be typically would be the top of rock Of course, you can peer with the DC gateway and advertise directly connected routes. So For the control plane we need to have something so we did it pretty much just that the deployment layer Create a loop back IP configuration that's going to be used to peer with the with the top of rock and and just deploying it for our configurate and Setting up the agent as I said like no changes to new turn or OVN or any configuration at all with those components And for the airplane we're relying on this BGP agent, which is connected to the OVN database So this is actually it's not open stock specific. We have done it agnostic. Of course, like the focus was Open stock, but it just reads from OVN database and It configures things to trigger a far to advertise the routes in the case of EVPN It will configure the the VTEP endpoints for the for setting up the tunnels and it will be in charge of redirecting the traffic to and on from the OVN over like and off to you, Luis Thank you So I'm going to give a little bit more of the test about the VGP control plane and especially the The data plane for BGP. I guess you are well aware already But this dynamic routing protocol basically to change routes And it's basically based on the concept of autonomous system and depending on if you are changing those routes within or across Autonomous system is EVGP or IVGP We use a bunch of features from it like router reflectors, VGP and number, ECMP default routes and we are actually really leveraging the BRF one, the virtual routing and forwarding. This is similar to the Network name spaces, but just for layer 3. So in a sense, it's just kind of routing table insulation and the tool as Daniel mentioned To leverage or to use a BGP is FRR the one that we choose which is a Protocol suite. It's not just for BGP. It has all the protocols, but in our case we just focus on on BGP It's a Linux foundation project. It was a fork from Quagga and just as a It's a pointer in in real aid It was a bird was deprecated in favor of FRR and it came with a cell BTYSH that is what the the agent will use to reconfigure FRR when needed I have there just one example a random one for a configuration file for BGP It's nothing to take a deep look at it It just wanted to highlight the red distribute connected because that's kind of the feature we are going to Abuse or to leverage Basically that means that the local routes are being exposed So in the note everything we add to the devices is what it's going to be exposed outside And that's what the agent will do to ensure that the BMS IPs are exposed outside For the control plane we move to a way where it's a pure layer 3 control plane basically we This can be visible, but we have this kind of spine lift apology We're now the controllers of open stack can be moved in different racks instead of having to be on the same one Thanks to this layer 3 we make the triple O adaptation so that we kind of support this this deployment and do the FRR configuration with With our needs we have like a dedicated networks to connect the controllers on the computer to the to the lift So that we ensure complete the installation there and basically we expose the The BAPs of open stack name points just by adding them into the loopback interface in In the controllers and basically then this is thanks to BGP Share for the infrastructure and then a failover. It's as simple as adding these fit to a different Controller and automatically the it will converge and it will detect a new location I'm not going to give a lot of details about this because the talk is kind of focused on the data plane more than in the Cortone plane, but just to give a heads up here and of course any question. It's We will try to handle it For the data plane as Tani mentioned is the OVN BGP agent what we are been working on It's basically just a Python based diamond that runs in every open stack node It just it is just connected to the OVN cell phone database and just to detect the events So it's kind of reading only mode detects the events that are Needed to figure out when we need to advertise and do to To expose a VM and it basically leveraging a couple of things on the one hand as we explained the the far are BGP to announce the relevant IPs in our case the VMs and the load balancers VIPs And also the kernel networking to redirect the traffic once is in the node to the OVN overlay It has some requirements like of course we need a frr on the node so that we can use BGP it we need to be connected to some peer It's usually the leaf but it can also be the DC gateway at the center gateway if need be and we need to enable ERP and DP proxy so that the traffic can be redirected to to the OVN overlay and the Linus provider bridge the obvious provider bridge can answer to that It's just a simple diagram of how it works basically Haliting again that the idea here was to not need to modify core of EN or Neutron at all so just an addition and an extra agent there It has two main components the watcher and the driver and it's built in a sense that you can plug in different Watcher on different drivers depending on what you want to integrate with for our case We have couple of drivers as supported now like a BGP and EVP and and basically what the watcher does is to take a look at the audience of on database and Especially focusing on the poor binding table events and when certain things happen like usually like you put up a VN Or a load balancer VIP on the Provider network or you associate a float in a bit to them This event is detected and then the watcher is calling the driver and this driver is the one in charge of doing Beneath actions to ensure that the VM is reachable in this node from that point So the first thing to do is basically to ensure that we Kind of wake up for art to make the IP available from that point So basically we just need to add this IP to one dummy device dummy device created by the By the agent so usually you know separate VRF so that it doesn't interfere with the routine But it's just for the matter of exposing the piece of from that point All the infrastructure will know that this IP it's reachable through this note and then the second part of them of The of the work of this agent is to actually once the traffic can be Can reach this note to ensure that this traffic is actually redirected to the OB and overlay and to do that We basically use a set of kernel routine like IP rules and IP rules to redirect the traffic And then also need to to play with some with adding some extra obvious flow to the provider bridge from obvious So that we handle the proper Mac addresses It's simple to add new watchers or drivers and for instance another one can be as Daniel mentioned This is not relying on neutrons or something So it's just taking a look at the OVN so you can add one driver for OVN Kubernetes for instance Taking a look at the cell phone TV detecting the events in especially you will need to do it in a slightly different way Maybe detecting only the events that you are interested in and then maybe the driver need to trigger some extra actions We actually did a POC where we play a little bit around with EVP F and XDP And that's basically is like adding yet another box there of set of actions to be done to expose the VM from there I'm going to explain a little bit briefly the couple of drivers that we have so far We have the BGP one. This is basically in church or main target is to expose the VMs and the load balancers on the provider networks Or which has a floating IP attached to them one caveat here is that it doesn't have an API yet So everything is exposed by default everything that goes into the provider network But it allows us to have like different VMs in different open stack cloud Being exposed with the same IP and then rely on BGP to load balance between them or to read the traffic to one or the other The one that is available And it also allows us to do this without actually not needing to change the Physical network configuration after when you want to add a new provider network. It's as simple as that so It also came with an option to also expose the network but with some limitations like traffic need to go through The network are now there were no support for overlapping IP is no API So that's why we actually work also on the other driver the VPN mode to kind of address some of those limitations I'm going to spend that later on Actually already covered this in the previous one But basically is the watcher the one detecting this type of events either VMs or load balancer ports being created on this Provide a network of tips associated to them this called the driver the driver triggers the kernel OBS or FRR reconfigurations and Then the good thing is that traffic goes directly to wherever the VM or the load balancer is without having to go through the network I'm not again with one Limitation for OVM Octavia OVM load balancers as there is no VM associated to them the traffic need to go through them Through one of the network nodes or the controllers This is just an example of how this work kind of step by step like trying to zoom in a little bit in a couple of holes couple of leaves and basically We have a device there OVM VRF That's basically the device that the agent create to be able to later on Advertise the IPs for BGP and then there is a set of IP rules and a new routing table associated or kind of related to the VRX from from open stack which at the beginning are empty As soon as 1pm is boot up in in this case host one what we need to do is to the agent What it does is to add this IP to this OVM VRF device, which is kind of isolated This actually makes a for our to to advertise that IP so from that point on everyone in the Anyone on the spinelift apology knows that that IP is reachable through that note So the traffic knows how to reach that point and then it the agent is in search also of adding this IP rule Which basically says like all the traffic coming from everywhere that goes to this IP 172 24 115 Check how what to do with this with this by checking the VRX routing table and this VRX routing table It's by now only has one kind of default route that send it to the VRX bridge There is other routes that are added there when exposing tenant networks, but for for the floating a piece and IPs on the provider network. This is enough. So If more details on that Yeah, that's it's just a kind of the traffic flow like when you go from outside of another place It will just come from one of the leaves one of the nicks and then through the kernel routing It will be redelected there But of course if you have another VM on the same note, it doesn't need to go all that way It will just go like east-west kind of traffic and it will just hit the internal flows from from via int Well, and for the other driver and the main idea for this one was instead of focusing on the public IPs Or the provider network was to focus on the possibility to expose the tenant networks And also allow in the overlapping ciders overlapping IPs like meaning that two different tenants may want to expose The same kind of subnet IPs like ten zero zero zero that's 24 or something And you don't need to synchronize them so that they can do whatever they want They are advertising that at the end of the day in different kind of be excellent ideas. So that's how it's segregated or tunnel it and This actually has an API. We are leveraging here the networking VGP as the API for this I'm going to show you in the next slide and This one commit at the moment that it of course need to go through the network our note at the at the time being because it's This is a limitation of ML to OVM and how we it's deploying open stack that the SNAP traffic is centralized meaning the traffic to the tenant networks need to be injected through this OVM Router gateway port that is connected to the provider network But this driver basically allows us to interconnect different open stack clouds either in the same data center or across data centers for different tenants and I'm not sure we're going to have time that we have a demo at the end that we will It's already uploaded in case anyone want to take a look It's a little bit of the scheme house of how this work the API as I said before is based on the networking VGP VPN, but it didn't come with support for OVM So we add some some driver and the working progress patch is in there Basically networking VGP VPN is just an API or the plugins and API that allow us to expose a neutron ports through their networks and router and connected them through an EVPN or VPN and The agent is using this basically. We add just one. It's really simple Service plug-in driver, which the only thing it does is to consider or to take note of this information about the BXLan ID or VNI ID that is going to be used and information about the autonomous system For the VGP that we need to use and it just write that into the OVM northbound DV in the network Associated port there. This will make this automatically translate it and I'm promoted to the OVM southbound DV in the port binding table And this is actually again in the same way as it was in the other driver like trigger the OVM VGP agent So the watcher detects this event and then is in charge again of triggering or calling the drivers So that it triggers the needed steps which are absolutely different for this But this at the end of the day same models like FRR for advertisement and then kernel routing and OBS flows for traffic redirection One thing to highlight here is as you see this is running on the OVM network gateway As I said before it doesn't it doesn't go directly to the VM but need to go through the networking node or the Or whatever this CRLRP port is which is the router network Gateway and then from that point you inject it into the OVM overlay and it goes as in any East-West traffic through the GNIF channel to whatever the VM is This is an example of how this work basically the admin is the one creating the kind of BGP VPN like associating a VNI and then associating it to a tenant That's you may imply some tweaking on the infrastructure like if you have some BXN ideas that you want to use to interconnect things and Then the admin is the one in charge of assigning this to one tenor or another And then the user once it gets assigned this BGP VPN It can freely assign it to its networks or one router and automatically it will get assigned to all the networks connected to that And with that is what's basically triggers the VPN driver to do the needed kind of kernel networking tweaking It has to do several things it base as I said before in the VRF VRF concept So it creates one associated to this To this VNI ID which need to create also some other devices to do the VXLan ID tagging on or in-cap and de-cap So it creates a bridge and a VXLan device and connect it all together And then it use a best pair to actually connect this VRF to the to the OVN overlay through the VREX bridge Then it adds the IP rules in this in the routing table associated to this VRF so that when the traffic goes To this network it's redirected to through that best device to the to the OVN overlay And also in the other way in traffic leaving the the open stack We need to also have some obvious flow out at VRX So that if it's below it's coming from this subnet we redirected it to the VRF So that it came out to this kind of VXLan ID instead of going out the normal path without encapsulation And then just to advertise the IPs of the VMs. It's basically same behavior as before. We just add these IPs Into the also a dummy device. I put it there look back once 0 0 1 and those IPs are the ones that get automatically Advertised through EVP and by BGP in that channel Back to you. Okay, just real quick to rub up and Love some couple minutes to question for questions So yeah, this project and the code you can check it out. You have stream you can send reviews as you were Used to do it with any other open depth project. So we have a Gary you can submit your patches there check out the code whatever There's plans to add more coverage more open stack specific and and eventually move it under the neutron umbrella And there are a lot of information of the flows that we described with precise examples and you know detailed deep dives and even some videos That URL so that's really interesting blog from Luis and Just some limitations and future directions real quick It of course like you know, we've been saying it all along The idea was to keep everything as it is. So, you know, there's no integration You know with data path accelerations or we are falling back to the kernel that it obviously, you know, it's not really comfortable with Things like SRV or OBS DPDK We have things in in mind, but we're happy to hear ideas So far for the BGP mode, there's no API. You could be easily integrated and probably extended the API for user dynamic routing although the the approach that we we've taken is slightly different and You know traffic to tenant networks It's still traversing the gate. We know that that's some that's how we how neutral works today Forest nut traffic and future direction, you know, just coming up with an API trying to make Distributed north-south routing to avoid traversing this extra hub that I was talking about and maybe moving some of the pieces that we have To OVN to core OVN like the NDP or our proxy You know to make it or even even the The routing part move it to OVN so that it can be afloat at some point Some of the interesting things EPPF and XDP. Luisa has been Playing with that accelerated in the ingress part. So it doesn't need to go through the kernel and That this is pretty much it No time for the demo. We just have Few seconds left. So please feel free twice any questions and Thank you so much for Forcoming is we really appreciate that Right. Just let me repeat the question for the for the audience here So the question is about performance and scale. Have we taken any? measures about You know IP advertisement the time that it takes for it to be propagated and failover and how many routes We're able to take and whatnot. So we have done just very limited tests We haven't tested these at performance at scale So we have just tested the failover time. It's below a millisecond. It depends really how you configure BFD So BFD you can parameterize it to you know for different convergence. So there's certain tuning that you can still do But unfortunately, we don't we don't know like what is the actual scale? How many routes we can I guess that it depends on a lot of factors But yeah, I don't have a like a we don't have an answer now We have a colleague though that did some testing not with this type of deployment, but to check like BGP and this stuff It was more for the open stack for the open shift and component use case and it got really nice numbers Related to the scale and the amount of routes that you can expose that it was kind of really nice 100 it was more like the number of peers Or not affecting that much unless you go crazy big so but in this type of environment we didn't test it It was focused on open shift, but using a for our on the same type of approach. Yeah, there was no real bottleneck absurd That's a good point. Thanks for remaining Thanks Sorry, I cannot hear you. You have the mic there, but we can try. Oh Thank you How far is the integration of IPv3 version 6? Integration of how is how far is the integration of IP version 6? It's already working with IPv6 too Yeah Yes, it's I mean we give it a try and it's actually when doing their rules It's exposing both of them like IPv4 and IPv6 And that's why actually we need to enable NDP to make like similar to to to IPv4 For the proxy so we give it a try with IPv6 and it was kind of working We didn't measure the performance either, but it was working fine It is the same approach But instead of using our proxy we're using NDP proxy and obviously there's no floating IP because there's no such a thing in Neutron for IPv6 But otherwise, yeah, it should be working for it using BGP and number in this configuration to use IPv6 to Distribute the rules for IPv4. So Thanks There was other somewhere. Oh, there's a line there. Okay So you've got a similar setup. We don't use OVN. We use Calico The drawbacks are it doesn't support a lot of the Neutron semantics Some of the things we picked up You mentioned at some point you're putting the local IP address on the on the local host interface We found that putting it on a dummy interface is a little bit better because you can set You can turn the dummy interface. You can down it. You can change the MTU on it It's it's a little bit less intrusive. So we we made a fabric zero interface for that I Took a look at the Neutron semantics and I think the reason why there's this Reliance on the networker node is because the ml2 driver The type the networking types are all assuming L2 adjacency. So it's most of the the types there So I think I might be wrong here, but I think a new Neutron type is needed so that you can flag that the provider network is now L3 fabric and You can route directly to it. You don't need to go via the networker now node That's that's most of the stuff the other things is Yeah, a lot of the DHCP and network auto configuration is still not really mature in this space So it's still something we need to do On it. We need to put on all of our machines. Is there any Help in that respect that's that's already been done. I didn't catch the part with DHCP So so with DHCP and those kind of things you can push the normal L2 configuration, but there's no Auto config for for BGP routers yet Right. Yeah, actually with OVN the ATP would be served locally So it would be handled by OVN controller. I mean when the hosts get provisions Because all the provisioning time. Yes. Oh, okay. Yeah, so so that that's kind of a problem at our stage We need to hard cut the Or push the BGP stuff using custom configuration. There's no way to discover that Remotely through the network yet. Yeah. Yeah, so far we're relying on The top of rock being the next hop for all the traffic going out So we are not importing any routes and we're just exposing we're just advertising But that's a good point. So we are relying on a single a Pre-defined get way for all the traffic coming out the node or actually to you can do, you know What we're doing today is ECMP So typically you would have two nicks and you will you'd have an ECMP Route outgoing so you don't need to to know exactly because you're gonna reach the top of rock anyway So that you're moving the problem up to the top of rock and beyond I think our only thing that the top of rock exports is the default route in any case. Yeah. Yeah Thank you good points. Thanks Yeah, thank you at first for this topic and I've been waiting a lot a long time for this feature to come And thanks for working on that I've built open-stack installation with layer 3 rotting in the underlave for quite some years and that I was waiting for that What I'm wondering is The management of the switches to get the BGP session set up Are there any facilities in the project do you envision something like that coming or is this still something I have to do by hand? That at least for now it was out of the scope of this What we I mean for playing around with this we have like a set of Ansible playbooks to set it up, but it was more like a PoC kind of thing not like something for production to configure the switches and this kind of thing So that we then look at it. Yeah. Thank you. Good. Thanks. Thanks And I would just continue My question was the evpn features sounded a little bit like the ovn interconnect feature that it has on its own to Connect different ovn deployments Is there overlap that you intentionally build out or is it just two different solutions for the same problem space? to us to us ten is similar, but the things really different because For the interconnect is more like For the whole tenant right for the whole open-stack and this to kind of segregate the tenants for a different VXLan ID, right? So I think there's some limitation But we actually were taking a look at the interconnect to see if that was Good for us for for this kind of case and we actually wanted to use it But didn't found out how to to fit in in all the you mean the ovn interconnect? Yes, I Think you meant the ovn interconnect. Yeah, but that's using transit switches. Yeah, so you need to yeah But you need to agree on the on the site is an all because you need to share part of the of the networking So you will create a transit switch where the two glass table connect. Yeah, it's Actually, we were we've been looking into that to extend to scale a little bit better like when you go certain, you know after a Essentially you would have to ovn control planes, but you need some management to do between them Yeah, we haven't thought about That in the context of PGP, but it could be it could be interesting because you can you can advertise that transit switches to routes to them Yeah, thank you. Thanks very much. Thank you All right. Thanks. Thank you very much


All right. Good morning. Welcome to the first session of Automotive Linux Summit 2023. My name is Walt Minor. I'm the AGL community manager. And I'm going to give an update on what AGL has been doing this year and a little bit of what we have planned for next year. I know Dan Kaushy already gave some of the information during the keynote, but hopefully I've got some new information. So first off, who is this guy? So I've been the AGL community manager now since 2014, which is to me a really long time. I've worked in Linux mostly since 2005 or so when I started working at Motorola mobile devices, Linux Java phones. We were putting out the first Linux mobile phones in the world back then. And I worked at mentor graphics. I worked at some tier one suppliers and I did some defense electronics. And those are some pictures of me on some motorcycle trips, one to Yosemite and one with my new BMW motorcycle that I got this year. I had my motorcycles thing here on my lanyard, but I forgot it today. So if you want to talk about motorcycles, I'm here for you. So last year when we were in Japan, my colleague, John Simon and I, we took sushi classes. So now I'm an apprentice sushi master. The thing is, whenever I live in Asheville, North Carolina, and there's a great need for sushi masters there, but whenever I try to apply, they tell me I need advanced training. So I'm still looking for the advanced training. I was hoping to take another class this year, but I haven't had time yet. So this is kind of an overall timeline of what we accomplished this year. And we still have a few more milestones. So as you probably know, we name all of our releases after fish. And the beginning of the year, we made our optimistic octopus release. We did about five releases, five patch releases after that. We did our prickly pike release in the middle of the year. And then coming up at the beginning of next year in February, we'll have our prickly pike release, I'm sorry, quirky quillback release coming out. And we just are releasing our first milestone release, making our first milestone release on the way to making the final release in February. So prickly pike, which was again, it was the July release. I'm going to explain our relationship with Yachto, but basically the AGL is Yachto-based. All of our code is built on top of Yachto, Paki, and Open Embedded. And we've been using the Kirkstone, the long-term support release of Yachto now for a few years. And so prickly pike originally had the 4.0.10 release of Yachto, and we've updated it in later patch versions. We included Rust for the first time, because we needed that for the Cuxit.Val data broker. So that Cuxit.Val is an implementation of the CoVisa project's vehicle signal specification. Cuxit.Val was created and is maintained by Bosch. And so we use that now for our intra-vehicle communication and for our vehicle-to-cloud communication, as I'll talk about that in a few minutes. So we made a lot of general cleanup. We included some new VertIO features. We made some demo image changes. So it was a pretty busy release. You can get the full details on our release notes. And we added Chromium Embedded Framework. So we were using a WebOS open-source edition version of Chromium. And that was a very slow path of update from LG. So we decided to move on to Chromium Embedded Framework, which is a more modern implementation of using Chromium. And so that was now as a technology demo in the pike release. So this is just kind of a more detailed view of what we've done in pike. Every time we do a pike update, we have also updated the Yachto version, so that 4.0.10 has moved up as we've gone along here. And we have another 16.03 coming out in another couple weeks. That's pretty much ready to go. We just need to get it through QA. So Corky Quillback. So Corky Quillback is coming out in February. We are making the milestone one release. We had a little hang up with QA, and we found some issues when we wanted to get some extra features in. But milestone one is about to come out this week. As Dan mentioned, we're using Flutter. The Flutter Embedder has been greatly improved throughout this year. And the Flutter Embedder is now pretty up to date with baseline Flutter, upstream Flutter. It's now using version 3.13.9. We've made some, we made a lot of improvements to our Chromium Embedded Framework. Chromium implementation. There's a talk later today from, today or tomorrow, from Eagalia about talking about what they've done to the Chromium Embedded Framework. Again, we're much further up, so we're much further along on the upstream Chromium path. I think in the last six months, we've jumped from a version, a version in the 90s up to version 118. So we've made rapid improvement compared to where we used to be in terms of keeping up with upstream. So we're actually going to deprecate that old version of the WebOS Chromium and remove the support completely. In connectivity, we've updated to the latest version of Cuxa.Val, which is 0.42. We now use the Cuxa.Val data broker, and we're basically converting all of our APIs to use GRPC. And then a really cool thing that happened this year, something I've been asking for, for years, is for someone to create a demo, what we call a demo control panel. You can run it on a Raspberry Pi, you can run it on a PC, on a QMU emulator, but basically it allows you to control can signals or vehicle signals going into the vehicle from a remote device. And then you can either write a script or you can toggle switches on the can simulator. And that was all done by, we have a, we sponsor every year a few Google Summer of Code students. And one of our GSOC students, so Jimtin wrote that. Our other GSOC student wrote some new open source speech recognition plugins for us. So you'll be able to see that demo control panel. Tomorrow there'll be a GSOC update. So our GSOC students will be giving a talk. I really encourage you to take a look at what they've done. They've really did some really amazing work in a short amount of time this summer. And that control panel will be very useful for us for the next few years. I think everybody who's tried it out has really liked it. Audio, we've done a lot of audio work, basically updating to the new pipe wire and wire plumber 1.0. So wire plumber and pipe wire finally released their 1.0 version very, very recently. We've gotten that onto our main, our main branch. Our mixer API was now, is now being updated to use GRPC. And we added controls, specific controls for base, treble, balance, and fader. Previously, we only had very detailed controls for more like a graphic equalizer. But this allows people to think more in terms of, you know, the old school knobs you would see on our radio. We didn't really have that available in pipe wire and wire plumber before. For quillback, we're adding some new BSPs. We've got risk five supported in HGL now. Sci five has an unmatched board. And they're actually just built another 1000 of the unmatched boards. It's called unmatched RevB. Those are now available. You can go purchase those. We'll be supporting the Unrav, unmatched RevB as well as the RevA. I've got about eight of those coming in for HGL to use. The new Beagle Bone AI64 and the Beagle Play are also going to be available in quillback. The AI64 is pretty much there already and the play is also in progress. Raspberry Pi five was just released. You can pre-order them. You can try to buy them. They're really the stocks running short, but they'll I think become more widely available as we get into 2024. So we've got the BSP ready for that. And then the new Renaissance S4 starter kit is also available as the BSP is available there. I've already, John Simon and I already took a whack at creating the release notes for quillback so you can get a little more details there. We'll continue to fill that in the next few weeks as we get to the quillback release. So here's our schedule for quillback. We'll do the M1. Like I said, this month we're a little delayed. We originally planned for the 30th of November. We're now going to be this week, late this week. M2 will be before Christmas. Milestone 3 will be a few weeks after CES. What we try to do is anything that we do, we show at CES. We try to get into quillback so you can then download that and play with that. Do whatever you want with it. And the final release will be in the middle of February. And the slides are all uploaded already. I uploaded them before the talk so you can take a look at these at your leisure. So like I mentioned, AGL, we're Yachto based. We're trying to stick with Yachto LTS versions. Earlier this year, Yachto announced that they would be going to a standard four-year support model for their long-term support versions. We've been supporting Dunfell then for the last almost four years. It'll go end of life in April of next year. And our lucky lamprey release, which is our 12.x release, has been updated basically for every Dunfell release for the last few years. You can see as we go end of life here, I just kind of showed we'll get out to 12.1.19. So we'll actually have made, I think, 21 releases of lamprey by the time it's done, which is pretty amazing for what we've done the last few years. Right now the Yachto schedule shows that the last release of Dunfell is in mid-March. So what that means for us is our more later versions of AGL have been supporting Kirkstone, which is 4.0.x. Kirkstone support will continue through April of 2026. They will have a new Yachto LTS version 5.0, which is named Scarthcap. I'm sure I'm pronouncing that wrong. So Scarthcap will be released in April of 2024. AGL's Royal Ricefish release will basically move on to Scarthcap. And similar to what we did with Lucky Lamprey, Quirky Quilback will become our long-term release for the next few years as we continue to track Kirkstone. So what that looks like is, like I just showed you a minute ago, we finished the release in February, and then I tried to map the Kirkstone schedule for next year into releases for us. We tried to do them about two to three weeks after the Yachto release is done. So you see we have seven patch releases planned for Quilback next year. So this is the overall 2024 schedule. I think it's the first time I'm showing this. So pleased to announce that our late second half of the year release will be called Super Salmon, an easy one, most easy. Super Salmon. Every year our steering committee goes through a process where they vote on the most relevant features to work on for the year. And what this does is it doesn't necessarily determine all of the work that we'll do on AGL, but it does help focus our our paid development on particular efforts. And Jerry's probably laughing right now in the front here. He's going to have an SDV expert group update right after me. And he's going to tell me I'm all wrong. But what I tried to do was, what I tried to do was summarize where we are with each of the with each of the steering committee with each of the steering committee priority items, which were the top 10 or 11. So SDV, the top item, top three were all software defined vehicle related. Extending vert IO support. So virtual open systems is doing some work for the SDV expert group. The effort's about 80% complete. It should wrap up by CES. Now what Jerry needs to tell us in the next, you're going to, you're going to ask him this question. Okay. Cause I don't know what the answer is, but I've been able to talk to Jerry for a few weeks because he's been traveling. He's right there. He's been traveling. So what I don't know is how that 80% complete or how completing this actually finishes the vert IO support to decouple the UCB. I think there's still a few more things to be done to finish the decoupling, but I don't know how much Jerry's going to tell us in the next session. Okay. You ask him that on the Q and A. The container orchestration, to be honest, that didn't really get started. We had a guy working on it from another company and he kind of fell off the map. And so we're going to get that started again. Panasonic is making great upgrades strides on the unified HMI. Some initial commits were merged to Garrett last month. And we have some more commits that were actually made into Garrett this week that are in review. We were supposed to have a discussion about the unified HMI, but that one got canceled, right? Oh, that's not the one that got canceled. The vert IO one got canceled. Yes. See, that was the one that was canceled. He would have answered the question, but now Jerry has to answer that question. So there'll be a good talk on the unified HMI work that Panasonic has been doing and they are making great strides on this. Vehicle to cloud. So if some of the features were vehicle to cloud related, and again, vehicle to cloud is kind of part of adjacent to software defined vehicle. A lot of people will argue without vehicle to cloud, without OTA updates, without the cloud data component, you don't really have a software defined vehicle. So that's a work in progress to convert from those standard vehicle signal specification object names to something that talks that the cloud can understand. That's a work in progress. We expect to complete that in time for embedded world, which is in Nuremberg in April. So we'll be showing some new demos in embedded world as well as the new some new demos at CES this year. Additional reference hardware. From my point of view, I know Panasonic's been hard at work with this with Qualcomm. From my point of view, there's no progress. Flutter. So like I already said, we've made a lot of improvements in the Flutter workflow. One of and the Flutter and better continued evolution. So we've got those new reference apps that Dan talked about from ICS. One of the things that we need to work through, we're going to have a kind of a wrap up session next week with ICS and our Toyota Flutter guys. We're going to talk about kind of that workflow. So we had to bring those flutter. We had to bring the ICS guys on board, get them up and running on an AGL, just get the UCB up and running for them. So what lessons learned do we have from that and how easy or hard was it? And how can we improve that? How can we get them, how can we onboard new developers better and faster? So we should get some lessons learned and turn that into some activities for 2024. Numbers eight and 10 are more vehicle to cloud topics. So there will be, I'll have a couple slides on vehicle to cloud later, but there will be a vehicle to cloud expert group update by Megan Knight tomorrow. Our top support for AGL UCB. So again, in these mixed criticality use cases where you have the AGL with Linux running alongside an RTOS, we have, we've done a lot of requirements and design work or our requirements and architecture work within our system architecture team. And that work continues to evolve. So this is the list of the current, I broke it. This is the list of active expert groups that we have. Our system architecture team, we have an IVI team that's IVI production writing this team that's devoted mostly to IVI type topics, instrument cluster. So I'm just going to give you a quick update on what each of these groups has been doing. In the last year, last six months or so, we went through Confluence and created a much better, I think a much better landing page for AGL overall with very clearly marked links to each of the expert groups and the system architecture team. We're still, it's still a work in progress. We're still improving those, but we now have Confluence pages for each of our expert expert groups as well as that landing page. And the system architecture team has really been focused on that RTOS support for AGL. And they've been working with the SDV expert group on defining some of those use cases as well. The IVI expert group, again, I have these slides all uploaded to the schedule page. So you can click on all these hyperlinks. The flutter and better is now updated to 3.13.9. We've got the new IVI reference design ready for CES. And I had to tell my boss just now that we're not unveiling that this week. We're unveiling it at CES. But it looks really cool. And if you come to our old member meeting in January or February, rather, which he announced, we will definitely be able to show it to you there. So it looks really sharp. I think you'll really like it. It's a big improvement over our legacy UI. App framework and connectivity, Kooxa.doll data broker, the flutter and the web apps, as well as the Qt apps, the legacy Qt apps, are all adapted to use the Kooxa.doll data broker, so they're all using GRPC now. We did a major documentation update on our documentation site, created a Rust Mix-in layer. So now you can use Rust on AGL. And the reason we had to do that, that Rust Mix-in layer is a Yachto thing. Yachto Kirkstone had an older version of Rust, and we needed a newer version of Rust to be supported for the data broker. So now we can more rapidly update our Rust versions. That Yachto Mix-in layer for Rust was created by Scott Murray and in our connectivity group. And you can see there's the Confluence page for the app framework group. There's also a separate one for the connectivity group. So SDV, you'll learn all about it in the next talk. I'm not even going to talk about this, this is all great stuff they've been doing. We have three talks planned for this week, and then a boff session to close tomorrow out. So we can all talk about SDV. And I think one of the questions most of you are going to ask is, what is an SDV? And hopefully we can answer that. Vehicle to Cloud. So we're working on this reference implementation for messaging between connected vehicles and cloud providers. The work is being done by largely being done by AWS as well as our connectivity group for the VSS part. But intentionally the interface is not going to be, it's going to be cloud provider agnostic, but we will provide a AWS reference implementation. So the idea is the APIs and the specs, so to speak, will all be agnostic to the cloud provider. You can use Microsoft, you can use whoever you want. And again, we're not trying to reinvent the wheel here. We're not trying to create an AGL specific version of a vehicle to cloud service. What we want is to work with other groups like Covisa and other groups that are working on these types of activities, make sure that we're all focusing on the same specifications and that AGL is implementing those specifications and providing the reference architecture for that. So we don't want to write specs. There's nobody on AGL that I know of who likes writing specs. Getting the right documentation is hard enough. Writing specs is even harder. So Megan tomorrow has a session with a full update of the vehicle to cloud activities. Instrument cluster expert group. This afternoon, Kurokawasan will provide an update on the expert group activities. Basically our prickly pike release had our full CES 2023 demo equivalent with the containers solution that they have. And I think they're showing some of that upstairs in the AGL booth. And they're working on basically what they're going to provide in Quillback and for CES this year. Our continuous integration and test group expert group. So they're the guys who are focusing, the folks who are focusing on Jenkins and Lava and making sure that our Garrett CI works whenever you make a commit to Garrett. It goes through automated CI, some automated testing. We added risk five, are that risk five board to the Lava lab. So now everything that we're doing, all of our, any commit that you do now gets run through CI for risk five as well as ARM as well as Intel. We're still trying to acquire that minus us as four starter kit. We're going to get it soon. We were looking at, but I should have updated this. We're not looking at Beagle bone AI and Raspberry Pi five support. We've got to been there pretty much already now for Quillback. So you say, well, I want to get involved. How do I get involved? I'd love AGL. It's really wonderful. How can you help? How can you help me? Or how can I help you? So if you're interested in any of these topics, all these expert groups have a zoom call every other week in addition to the Confluence pages. So you can talk to the people who are actively working on it, learn what's going on. So there's a call, like I said, every other Monday for vehicle to cloud, every other Tuesday for the system architecture team and the SDV expert group. We have a weekly developer call. So every Thursday, if you have a question about AGL or how to do something or you're having trouble getting something to work, or you just want to talk about AGL or listen to me blabber on for an hour, you can call into the developer call. We've been really working on our documentation improvements there as well. So we've had some folks who very kindly volunteered their time to just do documentation updates. It's really helped our documentation. The gentleman who was working on that, he's moved on to some other things now. But that's a continual effort to get our documentation improved. And I think helping improve documentation, you will learn AGL. Just kind of the way it is. So here's some more information. If you want, we have a mailing list for our community, our developer community. We have IRC chat available on IRC available on hash automotive on Libra chat. And our weekly developer call. So that's it. Thank you. And if you have any questions, I'll be happy to make up an answer.


Wacom to nursing school explained in this video on how to inject an intramuscular Medication how to give an intramuscular injection? So first of all, I've already prepared my medication and make sure that you check all the patient rights as you go through the medication Preparation once you arrive to the patient's room. Of course, you're going to introduce yourself Do your medication check and then get ready for the administration by exposing the appropriate site for the intramuscular injection and performing your hand hygiene as well as putting on your gloves Remember to always have a an alcohol wipe or two available as well as a band-aid And I'll show you a little trick here on how to handle the band-aid so that you can Get a little bit more familiar with how easiest to do this because you're going to need both hands to administer the IM injection So what I do with the band-aid here is I open it up And I peel one end of the adhesive and I stick the band-aid on my clean glove So that way I have it handy and I don't have to fuss around With opening up the band-aid wrapper as the maybe there might be a little bit of blood coming from the injection site So here I have used this 3 ml syringe with a 25 gauge by one inch needle and Typically that would be used maybe for a vaccination administration in the deltoid It could be used on smaller children for vastus lateralis or even gluteus medius muscles if the patient is old enough to do that So before I select my injection site, of course I'm going to always tell the patient what I'm going to do and then Once I've picked the injection site I clean in a vigorous motion from the center Outward to carry any germs from the inside away from the injection site then I want to remove the cap and Stabilize the muscle and the skin in whatever way is Easiest and I move my hand aside so you can see and this I am we're going to give a 90 degrees So we're going to pretend to hold a dart and give the patient a warning here. It comes one two three poke in a dart-like motion and then with your non-dominant hand you can push the plunger Stabilize with draw the needle right away cap it for safety and Then use the band-aid that you have handy for Securing the site get rid of your Needle and your sharps in the appropriate container and that is the administration of an intramuscular Injection at 90 degrees. Thanks for watching. Also check out the other videos about Subcutaneous medication injection as well as insulin administration. Thanks for watching. See you soon


Live from Las Vegas, Nevada. It's theCUBE, covering EMC World 2015. Brought to you by EMC, Brocade, and VCE. Okay, welcome back everyone. This is theCUBE live at EMC World in Las Vegas, our flagship program. We go out to the events and extract the civil noise. I'm John Furrier, the founder of Silicon Angle. Our next guest is Mike Kohler, the president of EMC Professional Services and guest Mike Von Slomsky, senior director of IT at Marko. Welcome to theCUBE. Thank you for having us. Mike, great to see you again. So give us the update. Last year on theCUBE, we talked about the cloud and the opportunities. Where are we? Give us the update. A lot of work. Is it a heavy lift? It has been, we've talked on this show many, many times as we've talked about cloud and getting to the cloud. Last year we introduced the concept of Enterprise Hybrid Cloud here at EMC World. And that was a proof of concept. It showed very well. And what came out of that was the customer interests continue to grow from just getting to virtualization to really IT transformation and Enterprise Hybrid Cloud. So the last year for us has been a lot of heavy lifting of taking those concepts and ideas we showed here around an Enterprise Hybrid Cloud and turning that into a solution, both products, but also services. So obviously my role at services been a lot of heavy lifting because the services that sit around an Enterprise Hybrid Cloud are much, much different than a product implementation or just a transference of storage. We'll get some of the use cases and some of the educational things and kind of where the proof points are these days. But we have a customer on Mike Bonsomsky with IT and the insurance market company. So tell us. I mean, you're the guy who's got the problems that you're trying to solve and it's not an easy thing but there's a real advantages of doing it right. So take us through quick overview of the Hybrid Cloud solution that you guys are working on, what you've done, kind of where you were and then what iteration points did you go through to make that happen? Sure, Markel being a company that's largely grown through acquisition, we found ourselves being in a place where we really wanted to take a look at some data center consolidation, bring the infrastructure together so that we could bring the platforms together and the applications. Last year at EMC World we saw the EHE or the Hybrid Cloud being offered and we thought that was kind of a right time, a right fit and it fit our long-term strategy for where we wanted to go from hosting that private cloud and really bringing the standardization in so that as public cloud starts maturing, hybrid cloud, et cetera, we can really start taking advantage of it. So it was kind of a seamless fit of a good timing for us to get involved. So you had a lot of disparate kind of architectures. A lot of disparate architecture, right? A lot of different architectures and again we chose the Converged Data Center through VCE to kind of bring that together and in the past what was really different or hard for us in the cloud space was bringing the APIs together and trying to make all the components fit. Everybody had kind of an open API stack back then but nobody had taken the chance to integrate it and that was kind of the innovation we saw last year with the EHC presentation that made us make this public. So EMC World last year you saw the presentation so take me through just from my own kind of personal curiosity. You go back to the ranch, go back to the company. What do you say? Hey, I just saw this epic thing. Look at this really cool stuff. Let's do this. What happened? I mean, did you have to sell it through? Was it pretty much a no-brainer? Was there challenges going on? I think it married up with when we presented kind of what our five-year roadmap for IT was and how we wanted to bring the Converged Infrastructure together and start making a play on leveraging applications across divisions. It made sense to us because the standardization that the cloud brings, the automation that it brings fit exactly what we was into our plans. When you think about today where the old story, it takes five weeks to build a server and go through the process, et cetera. Our IT teams are spending so much time just trying to meet the demands of the business as opposed to looking at things like digital technology and hybrid cloud and big data, et cetera, where they can really bring that innovation and focus. Yeah, I want to drill down on some of the use cases around big data and a little drill down, but Mike, I want to go to your take on this because we're at Joe Tucci up on stage this morning and Joe, I've gotten to know Joe over the years and he's kind of afraid to come on theCUBE these days because it's the first CUBE experience, but Joe, you know it's coming back on any time. We'd love to have you. Storage is sexy, we know that. Gone beyond that. The education involved, so the use cases. So what we have here is classic pattern we're seeing in cloud which is, okay, there's some serious issues to resolve. Now there's kind of a road mapping and educational process going on in real time, kind of parallel paths of tool up, transformation and education. What do you guys see for use cases and what are some other customer situations like this that you've seen? Yeah, so it's funny because over the last year we really hone these conversations with our customers and we built and we engage with our customers. Not anyone customer starts at the same place. So everybody's either on a road to IT transformation and to a hybrid cloud scenario, but there's other customers that have never not even started. So we kind of go to where our customers are. We start with some of our customers are just completely undecided, meaning they don't really have a roadmap yet. They don't know how to get started. So we engage with them not on a, let's talk about implementation. Let's talk about building an IT transformation road map with you. There's another set of customers that may have that roadmap, but don't have a way to think about the legacy application environment because it's one thing to stand up the infrastructure doesn't produce value. So you actually put applications, either legacy transitioning over or building up new. So we do a lot of work with customers on the application transformation that goes on. What's the success for them? Because you have a couple of things. You guys, you work for EMC, but you've got EMC as those channel partners that will go talk to end user customers really trying to solve these hard problems and change the airplane engine out of 35,000 feet as they say. So it's challenging. So what have you seen has been successful? Do you guys come in and do a lot of the legwork? Do you guys, what's the partnership with the customer look like? And Mike, could you comment on your take after? Yeah, absolutely. I think that's one of the values where we brought in EMC to help us just do the discovery piece. We knew we had a lot of disparate technologies out there, but at the same time, our guys are taking care of the day to day business. So EMC was able to come in and really augment us and help us figure out what do we have? Why do we have it? How's it connected? What's the conversation it's having? And more importantly, as we went through that data with them, do the analysis to come back and say, these are the things that are kind of cloud ready. You may need to make some changes. And these are some legacy platform things that may not be ready to fit. And that was really a big win for us to be able to get that decision analytics up front to decide how do we move forward from here? So it was a partnership you'd say? Absolutely. Was it like a menu of services? Or was it real time? It was more consultative, I think, yeah, yeah. Definitely. And that's what we're definitely seeing. You'd like to be as scripted and as menu-driven, but it really is, everybody's different. And so it is consultative-less engagements that have high customization around the customer model. But you must get some good data points around best practices. We didn't lock them in as recipes, if you will. Absolutely. And to the point of the discovery tools, the way we actually approach that, a lot of times we help customers build a business case. And so we've really gotten the nice templates down around what does legacy cost look like? What does it look like? And when you can get to the new model. And that's helpful. So you're learning with your customers? I mean, learning, yeah, we're refining the, yeah, we're evolving. I mean, we're evolved because this is, it's not just a technology conversation. It's about what do I do with my organization? What do I do with my applications? And what do I do with my infrastructure? So that's a much broader conversation consultative-led than we at EMC has just workly done. So Mike, talk about this transformation. So hybrid cloud is just a component. What's beyond hybrid cloud? Can you share more? And what did EMC bring to the table? Because digital transformation is like, now the buzzword, but that's essentially business transformation. Right. Well, again, I think what they bring to the table is, it's not just a technology change out. It really starts changing the business process, not only with an IT, but how that workflow and how the business operates. You know, and being able to bring that in and help us get a leg start of, you know, knowing the right questions to ask and how to ask them. And more importantly, how to start framing that workflow that needs to be put in place to start leveraging the technology that's underneath it is really one of the things that's made a big difference. How about operations? Because DevOps kind of comes into the conversation.  What about your ops? What has changed? And what's new and or different about how you guys run your ops? Because you got a little bit of multiple disparate stuff coming together. You get the hybrid cloud. What's kind of going on for your operating model? So from a software-defined hybrid cloud model, it really brought the platform teams together. You know, traditionally where you had the storage teams and the server teams and the network teams, now all of a sudden those guys are really performing the same type of functions underneath the software-defined type data center. And the same thing on the application development side, you know, by being able to stream in, you know, a small, medium, large service kind of on demand for them, but keeping them in a standard platform window is starting to change the way they approach their development. And the whole idea that, or the term that we've kind of coined is ruthless standardization. And that's been the big driver and part of what I think EMC's helped us get over the hurdle with so that when we get to the hybrid cloud question and we do want to look outside at a public offering, you know, our standardization is done or applications have conformed to something that's specific and it'll be much easier to make differences. Do you guys have a lot of developers on site? We do, we run almost three to 400 developers all in-house. What's the cloud development model for you guys? Is it, is there a DevOps ethos or is it more of? It's definitely evolving. I mean, we're brand new, we're building it out. So it's definitely an evolving model as we go forward. And that's part of what the teams are looking at today is exactly that. How do I change what I used to do in the past where maybe I had some servers stashed off the side and I'd go play with something versus I can rapidly stand it up and get something published and bring that into play. Mike, EMC, talk about the Federation's role in services because basically today's Federation Day here at EMC world, you have all the big guns up there, Tucci, Moritz, Gelsinger, they're kind of all hanging together. There's a unified front there, but they're all separate companies. You get to play across the Federation and has that relate to IT as a service and hybrid cloud. So we do, the Federation, it's a federation enterprise hybrid cloud is just that it's a federated offering. And so what we've done across the services organizations is sat in a room and said, what are the outcomes we want to drive from a service standpoint? But what does a customer want to buy? Forget the organizational structure of the Federation, but what are those outcomes? And then what we've done is lined our portfolios around that. So when we approach a customer, we talk about the outcome with customer buys that takes us up on that. We then have organized ourselves to a strong roles in program project management. VMware has very strong roles in the VMware technology. So we've already done pre-packaging of who does what and how that actually shows up in front of the customer. And you make that transparent to the customer that you bring them in when you want. No, I mean, transparent is in terms of touch points. Yeah, we want to make it look like one organization. And that is the goal and the mission. And customers should expect we bring the best to bear whenever it happens. And that's what we've been committed to do. So, Mike and Mark, I'll talk about the last minute we have here, kind of your challenges going forward, opportunities that will presented from these challenges as you solve them. And what's on the roadmap around things like big data?  I don't have that kind of dough yet, but it knows everywhere. He knows he rolls through the stop signs. He knows it. I mean, that's gold for insurance company. Well, absolutely. And it's just those types of things that the teams were hoping will be able to leverage their time now to do as opposed to today, where they're spending the same time just running the business, trying to keep the lights on, et cetera. It's really being able to shift that focus into the innovation side. And do a forward look with the business and understand what do they need? What do they think they want? And how can we leverage IT to provide that? So you'd say that you're level setting and then investing on top of that? Absolutely. Okay, so what's your advice to folks out there who are not here at EMC World or and are in the same situation we had earlier customer on. It's like an email service could go down at any minute.  In your case, disparate architectures to getting level set and then transforming. What advice would you give those folks out there that you've learned that have been magnified through your experience? I think it's got to be, it's not just an IT play. It's really a business play and a change. And I think one of the things that we tried to do early on was engage our business and make them understand what was going to be different and how was it going to be different. And what IT could bring as a lever for them. I think that engagement is driven back down through the development community and our side Markel as well as the IT platform teams to start initiating those types of change. It always sounds easy to me, but I know it's not. What's the hardest thing about doing that? Getting business focus. Is it just the blocking and tackling the meetings? Is it just mindsets? It's the blocking and tackling and the learning that you got to remember. Not all of us are in the IT business, right? And so we can't come in and tell them about the greatest new widget we got. But what we can come in is talk about how their process works, how the workflow works and what kind of, you know, how can we leverage what we're bringing to the table. Business guy says data lake in the IT guys. No, no, we don't have data lakes, we have data. We have data. Lots of good marketing data lakes, good way to visualize it and certainly great opportunity once you get level set conversion infrastructure. You see big growth in that area. And that's where, you know, so for us in the services, the engagement we're having with customers is all about a big data visioning. It's how, what could it do for me? So to get everybody, the business minds open, IT minds open and what is the concept mean? And then we kind of go into a proof of value conversation because everyone wants to look at it and say, if I spend the money, is there anything real here? And so we're having a lot of proof of concept. I mean it's cliche, but outcome driven business now with internet of things, everything's connected. Everything's connected, but you know, a lot of people I think look at it and say, yeah, but I've got all these data warehouses and I've got all that, so what's different? And so that's where we spend a lot of time talking about the visioning of it. Well, I think, you know, the future EMC worlds we're going to have the new benchmark, which is if you don't have real-time data, you're going to fire it because it's all there. There's no excuse for it, that's exactly right. Everything's connected, you can measure everything. All right, a hybrid cloud, again, John and Martin since early days and it's super exciting. Thanks for coming on theCUBE, sharing your story and your experience in journey of hybrid cloud federated solutions here. This is theCUBE, sharing everything here from EMC World 2015. We'll be right back after this short break.