Yeah, let's get on. So, whatever my computer to finish restarting. I did an update. Okay. Okay, here it is. And meeting minutes. There we go. Okay. So, if you have not added yourself to the agenda yet, please do so. Can you add me as well? Okay. And so events, so we now have. Three, well, we have three recurring talks or weekly meetings. We have this one. Every Tuesday we have the NSM docs every Wednesdays and we now have the NSM use cases, which has been moved from Friday to Mondays. Every other Monday. And so, if, if you're able to. Romkey or Prem are you online? Yes. Yeah, if you're able to fix and have the calendar onto this for your. Yes, yes. And it's, but it's every other week. Yes. So basically it will overlap with the. Yeah. The other thing is if folks could. If you could update, push a patch to update the website. Currently it shows on the community page on the website. Yes. That will fix it. Definitely. Cool. Awesome. So. We have a talk coming up. So this Friday will be. Service mesh days. And. See. I have the schedule right here. So I will. I will post it. The service mesh days is March 28th and 29th. So the actual talk itself will be. On the 29th, the 28th is comprised primarily of workshops. So if you are in town. That is definitely. Definitely consider. Joining in. We also have the Intel out of the box and network developer made up, and that is during the afternoon on April 2nd. And so we will have a 90 minute. Talk and. Hands on. Workshop that people will be able to try. There is the day right before. Right before. Starts. And I'm grabbing the link to that right now. April. Okay. And the link. There. Great. So. So. We will have a 90 minutes. Talk and hands on. Workshop that people will be able to try. There is the day right before. Right before. So we also have ONS coming up of where we have. Three, three talks. One of them is the intro to a network service mesh. We have a panel discussion. About using Kubernetes as a network service orchestrator. And we have a NSM and open stack integration. Exactly. We are doing in conjunction with the Sila at Ericsson. I, Sila has told me that she may now be, which is probably not going to be able to show up. So I may end up just giving me the talk. MPLS, SDM and NFB event in Paris, April 9th, 2012. Good question before we move on to that. So, are we still on for the LFN demo booth stuff? Yes, I mean, yes it. So in fact, I was just chatting with Nikolai about certain issues, so we have, I mean, we are fully into it. Yeah. Awesome, super cool. Cool. So yeah, so by the LFN booth and you can see ODL and members who are as much in action. We also have a, so I already mentioned about the Paris event. So we have a container world coming up in April, 2019. Sorry, Fredrik, just a quick interruption. I got to know that there is going to be an unconferencing track for ONS. So just wondering, should we submit a talk and just pasting the link on the chat window? So this is a good opportunity wherein this is more like a free flow discussion if we want to properly leverage this. Yeah, that's a good idea. So do we want to do something with us towards like just giving people to meet the community and use it as a time to as a set time for people to come and talk with us? All right, exactly. And also we can talk about, it's a free flow. So if you click on the link, you'll see the schedule there are the on-app discussion, OPNSE. So this is LFN. So what we can do is we can probably share it and at least post it along with open delict. And then we can probably open up to the audience and ask, I mean, we can have a free flow discussion. Yeah, that sounds good. Ed, are you up for that as well? Yeah, it also occurs to me by the way, I stuck in the link for the events page. So if we're gonna do that, we want to make sure we listed on the events page on the website. The other thing is we also need to figure out where and when we want to do the traditional NSM happy hour as well and get that updated on the website. But it looks like they've got, yeah, let's put something for NSM in there. But I don't necessarily want to do a talk so much as I think you said just something free flowing, like an NSM boff perhaps. Only caveat is I'll just check since NSM is not technical under LFN, I'll check with Phil Robb to see if we can include it. We can probably add ODL and then we can do that. Yeah. I'll check. In Europe, handle with us doing some on some of this time. One other quick question, the MPLS SDN NFE stuff, do we actually have anything there at all for NSM or should I remove it from our list of events at this time point since the CFP is fast? I think we, well, is anyone going who is interested? I mean, I'm okay with removing it because we don't have anything that's NSM related that's in the event itself. Yep. Okay, cool. I have some colleague who will attend and I know that Daniel Bernier from Bell is going to attend the MPLS Paul Congress, but it could be open discussion about NSM, but nothing official. Okay, cool. Cool. So mid-April, April 17th to 19th, we have Latino World with Prem giving a talk and he's going to be talking about networks for as much. We have Hugh Connie, you coming up with Barcelona. And so with that, yeah, let's go and remove those bigger ones. If you are heading over, make sure you book your hotel sooner than later because the hotels have a tendency to talk to you. The hotel situation is rapidly deteriorating. So if you haven't booked your hotels, absolutely do. I think we probably will have a couple of talks in a happy hour and some other things going on there. So you'll give me a little bit of time. I'll get that information up here. Cool. Yeah, and if you end up missing it, there is a really good transit system in Barcelona. The train leaves every five minutes or so. And so if you don't find somewhere near there, you get one that's near one of the trains, but just make sure that the line you're getting on is close to where your hotel is near the line that it's on because otherwise it didn't do a few transfers. And we also have a couple of located events. We have the FIDO mini summit and we have the, let's see, that one, the call for paper closes on April 5th. So if you would like to talk about, sorry, did I get that wrong? No, you're right, it's April 5th. Yeah, so if you like to talk about MSM in the FIDO mini summit, then definitely submit a talk. But is there a list of accepted talk for the CubeCon? Very easy to answer. So we have a couple of slots that are available to us on the anticipation of us being a CNCF project, but the talks that we submitted, none of them were accepted. This is actually not super surprising. There's been a lot of stuff floating around Twitter where the kinds of talks that accepted tended to be fairly within a particular range of things that were well understood by the program committee, which is a part of the course for program committees. So I think effectively what it comes down to is, as we become more well-known and understood beyond simply networking part, I think we'll do better there, but we will still have some things that we can do. Yeah, the CNCF appears to actually reserve some slots for you in order to help mitigate this. So they hand out two things that they think are useful, but the community are not very aware of just yet. So I mean, we should be good. I just need to get that sorted out. Cool. So we have CubeCon in Shanghai. I don't think anyone submitted a talk for that though. I'm inclined to declip it then just because our events list is becoming unwieldy. Yeah, I agree. Let's move it. And ONS Europe talks are, the call for papers is already open. We have a little bit of time before submitting in. So if you intend to talk there, feel free to do so or feel free to engage us and we'll help you put together a compelling talk. So this part is unfortunate. We have MEF 2019 and CubeCon North America on the same days. So there is a talk that has been, or there was something similar to MEF 2019. The call for paper for CubeCon has not been open yet. And with that, are there any other events that we need to talk about or should we move on? So, Ed, NSM CMTF proposal. So it is now in, do you wanna talk about it? Yeah, so the CMTF proposal's gone in. We have our two sponsors. We're anticipating review for the proposal sometime in April. Now, the way I think this is currently going in the talk is they have normally the technical oversight committee meets at 8 a.m. Pacific time on Tuesday mornings every other week or actually the first and third, the first and third Tuesdays of the month. I believe what they have done because they have a little bit of a backlog of projects is they have taken the Tuesdays at 8 a.m. that they normally don't meet and they're now meeting at those times just to process project proposals. And so I think that would mean that if we were to be scheduled in April, it would be either April 9th or April 23rd. And because that is at the same time as the NSM community meeting, one of the things that we can decide what we wanna do when we actually get a firmly scheduled time slot is do we want to simply cancel the NSM community call on that day and redirect folks towards the talk call for the project review? So that makes sense, Ned. And also one request, can you share the final submitted proposal? I know the link, you have the link, but it sort of has all the form and the real grand finale. That link is actually to the final proposal right there. Oh really? Yep, yep, that's the one that actually got pushed as a full request. Oh, okay, okay. Yep, yep. Yeah, so is there a value in showing up with people and numbers to the meeting? I don't know necessarily that there's all kinds of different kinds of value that could come from that. I don't think there's value in the sense that you will influence one way or the other how the review goes, but some of us will at least have to go and present it in that meeting, so we'll not be able to be here. And it might be the kind of thing that would be nice for the community to be there to witness. And for some people, I think there may be value there. Yeah, so I'm up for having it canceled and for us to continue at the meeting itself. Do we have an exact date yet? We don't. I'm currently working on the scheduling. The sort of rough cut estimate that I got was probably sometime in April, but the scheduling is to a certain degree up to the talk and I'm trying to get a clear picture of exactly all the ins and outs that go into that. Okay, that makes sense. So, it turns out that some of the more helpful people at CNCF have been in India the last week and between going, being there and coming back, it's been a bit of a slog for them. Yeah, I can see that because they have their first Kubernetes days event in Bangalore, I think. Yep, we're excited. Well, I think it's a good idea to redirect the community call to the talk then. And so if you'd like to come and watch and so on, then definitely feel free to do so. So I believe that those talks are all open. So there should be no issue with getting people in. Yep, I'm... Anyone else have other opinions or thoughts? Well, I think that the most, I'll say, I mean, the core people should be there. I mean, in case there are some questions and things that we can help. So effectively the call will be more or less obsolete. I mean, this will be a group call if they're all open. So I'm all about just moving there. Cool, that makes sense. Cool. All right, so I think our next thing is just to get the time slot organized and then we'll put an announcement on here. The worst-case scenario is the announcement is done less than a week before. In which case we will put a big banner on the top of this document saying please, please go to the other meeting. Okay, so we have a CNCF testbed that we're starting to do some work with. So there's some stuff that needs to be done towards us. Actually, if anyone wants to help out with this as well, it's a relatively easy task. Let me pull up the URL for it. There's gonna be an interesting meet in the middle here, I think, because we're going to want to stand up the chains of things in the CNF testbed as a network service in points. And then I think there's some potentially interesting stuff around the NUMA issues, the CPU pinning, those kinds of things that may be quite interesting work. Does that match your understanding from the folks from the CNF testbed? You guys know that environment better than I do. That's, yeah, that sounds about right. I think we're trying to do this in multiple steps, help with the use case of using NSM with OpenSAC is one of the items, and then being able to use NSM as an option with Kubernetes for use cases in general on the CNF testbed. And so we're going at that in several ways. There's quite a few tickets that are running right now related to this on that one. One of them is getting the V-switch in the pod, which is actually completed and several other things that are related. That's just an NSM related tie-in in, but is there anything else, Fred, about the OpenSAC and NSM? Yeah, I think what you just linked, the 213. Quick question. So you do now have the V-switch running in a pod, correct? Yes. Have you guys sorted out how you're handling the CPU pinning stuff? Are you still running everything in privileged containers, for example? Yeah, we're doing that in steps. No, no, it has to be done in steps. It can't be done as a big bang. I completely agree. Right now it's, the testing is more on pure functional and then as we add in the pinning, we'll deal with other stuff like performance whenever we get to that. But there's some items with Mellanox and the driver issues when we moved the V-switch itself, so not the CNFs, but when we moved the V-switch into a pod, we have that working on Intel, but on Mellanox systems, the packet systems have Mellanox NICs. And we saw some issues with the driver, so the current working code is with the Intel packet servers. Oh, that's, okay, that's very good news. Okay, so if you've got it in a pod, if the V-switch is in a pod and you've got the Mellanox NICs, I mean, you're using Intel NICs, then it should just be a relatively simple matter of getting everything turned into a network service input or a network service client to get NSM working in those testbeds. And then everything else around CPU putting stuff is things that have to be figured out in their own time anyway. So, yeah, okay, no, there's big total silence. So we may wanna break this part down at the item that's highlighted right now, that CNF testbed use of NSM, that wouldn't be 213, we probably need an Epic or I'll create a project or something that contains all the tickets. That ticket 213 there is for using in a Kubernetes cluster that has NSM enabled talking with the OpenStack cluster that's deployed using the CNF testbed code. So we should, we have, I guess two efforts happening at the same time, adding NSM to the CNF testbed so that it can be used anytime with Kubernetes clusters. And then the other item is that ticket 213, which is a Kubernetes cluster using, I believe it's all the make files and stuff that's currently used in NSM for deploying or setting up a cluster and then adding NSM to the cluster. Okay, let's not do it that way. So we now have Helm charts for NSM. And so I think probably our best bet is going to be using the Helm charts rather than the big file machinery. The big file machinery is delightful if you're a developer trying to work on NSM itself, but it's kind of vicious awful if you're just trying to deploy them, which is why- Yeah, I was gonna bring up the same thing after looking through the code. So the one area with the Helm charts is I don't think that they're not CI'd at the moment. So we need to start working on getting that stuff into the CI. Because we don't wanna be breaking the CNCF testbed or others. One thing that we could possibly do is, do we wanna make sure that the Helm charts work on a, we don't have to run them with every call. We could run them on a nightly basis as well. So that way they don't take too long. Yeah, I have been thinking about nightly builds. And yeah, that was definitely one of the things that I was considering. Like we just deploy a packet cluster. I mean, if we're talking about the currency, I just deploy a packet cluster and do Helm chart deploy just to verify that something is there and then maybe do some pinks. I don't know, invoke the checkscripts, something like that. So the one thing I want us to think about when we do that though is, there are absolutely things where they just take too long and we really get things like nightly builds. And that's fine, that happens. But I wanna try and see if we can keep as much in the line of the incoming testing that's done on a per patch basis as we can get away with without bloating, verify times to insane levels. Because that way we actually do know the role is in a good state at all times. So, that said, I don't think, I'm just gonna go back about 20 minutes, end up being helpful.  Well, we are a little bit over 20 minutes, I believe today and with some pending changes in the testing framework, we can very well go to 45, 50, something like that, which is not really... Can we potentially parallelize some of the testing? Yep, maybe. I mean, we have some patches that are being prepared for having a support for namespaces, which could allow us to run different DNS managers in different namespaces, which could eventually help. Like if we have unique namespaces for each of the tests, then they probably can be parallelized. We should also talk about that, because I would expect trying to run multiple network service managers in different namespaces. So essentially multiple network service managers per node. There are a bunch of places that we can opt to trend carefully for that to work properly. Yeah, of course, of course. So, pardon me. So no, okay, that's good, that's good.  And I think that's mostly it. All right, so one option that we potentially have as well is to throw more hardware at it over time. And I think the namespacing stuff will definitely help in another aspect. One option that we have is to spin up, is when we spin up a cluster, we could actually spin up a persistent cluster. And that avoids, so we could add and delete namespaces on the fly if we managed to get that to work. Yeah, there's a lot of work that we need to do in order to get us there. And I'm not entirely sure whether that would meet the standard MSM practice in production. There's a question on that as well. I think that we are keeping this constantly deploying and destroying the cluster for pricing purposes, like for just having kind of price control on what we use. I don't know, is that all? Yeah, so basically what it comes down to is being a good citizen, which is we should try and strive to make sure that we're actually not consuming insane amounts of resources, and that what we're doing, we're actually consuming it in an efficient way. Right, so right now I think if memory serves, when a CI run occurs, we start up two of the smallest instances that Pacin offers, and I think that runs a seven cents per instance. So we've got a total of 14 cents cost to run a CI run, which is not bad. If we were to double that to 28 cents, presuming that we were actually doing something useful with it, like parallelizing our testing, I wouldn't think that's an egregious use of resources. I think that the next actually available instance is about 40 cents, so it's more than double. Oh, no, but if we're going to parallelize tests, you really want to spin up two new instances, you spin up another one. Ah, I see, I see. Running on bigger instances doesn't really help us parallelize, but running on, instead of running one cluster, if we were to run two clusters, for example, which meant we could parallelize the test running. That's a fairly marginal cost shift. Quite frankly, I'm much more concerned about figuring out why we occasionally have zombie instances. Yeah, we do. Rememberly, that's something I'm much more concerned with than the notion of starting up a second cluster. We do want to be able to listen. The CNCF and Pacin have been super nice to us about all of this. But more than anything else, I tend to be someone who thinks in the world in terms of value. Back in the good old days when I first joined Cisco, one of the core values the company used to espouse was frugality. And they were super, super, super clear about the fact that frugality had nothing to do with how much money you were spending. It had to do with taking care to maximize the value for the money you were spending. Okay, so I think that's a good thing to keep in mind here. A quick question, you would mark this in progress, the VVVVV switch in a pod thing, Taylor. Does that mean that we haven't yet quite got the VVVVV switch in a pod, Taylor? So the testing with, I think the last thing was enabling hyperthreading on one of the Intel quad Intel machines. That was yesterday so that we could increase basically for all the test cases that we're trying to validate. We couldn't deploy as many CNFs. It's working, we wanna do more of the testing. Okay, that's super useful to know that detail. Cause again, that makes the next step that we would do for getting CNF, NSM working in force, the test bed, once that VVVVV switch is running in a pod, that's next step gets much simpler, much simpler. So that's super good to know. Yeah, one quick comment here on the CPU pinning of the CNFs or even the V switch. As you guys understand, the NUMA affinity will be associated to the physical NIC as well. For example, the interNIC that's been discussed here. So on a test bed point of view, I was thinking, we should have a NIC per NUMA node. You don't need to have it on day one itself, but the final test bed probably should have a NIC per NUMA socket. And that'll be the right testing for the various CNFs running on each of the socket. Yeah, you're absolutely correct about like what makes for good results. There's an ongoing set of interesting questions in Kubernetes around how to handle the NUMA affinity of things and it's the very short version of this is it is not going to ever work the way that it worked in something like OpenStack where you just do very fine granular mapping of stuff. That's never going to be acceptable in the Kubernetes community. That's the bad news. The good news is that there are things in progress in SIG node and resource management working group for actually allowing you to get what you need without doing that fine level of granularity of NUMA mapping. And those are hopefully going to land in Kubernetes 1.15. And I would expect the CNF test bed would want to take advantage of that. Did that answer at all some of what your comment was? Yes, my experience is surely coming from the OpenStack side. So yeah, that granular pinning was required and was done, but here if the flexibility or if it's not going to be so granular or strict in pinning, I understand, you know, but ability to exit out of the right Nick as a policy might also be a good thing to think about. One thing to keep in mind too is in June, AMD's drop in the Rome architecture. So the actual silicon underneath all of this is about to get some drastic updates from AMD and Intel are going in drastically different directions, but Intel is going to give you the ability at the hardware level to kind of customize what your NUMA zones look like. And then AMD is coming up with this like, I forget what they call it, but it's basically distributed across all the different dies spanning the different sockets.  And so like, I think when we get to maybe around this fall, even how DBDK consumes these architectures and stuff will be different and it'll give us the ability even at the hardware level to kind of tune, therefore not forcing Kubernetes into awkward positions because it can be basically ignorant of all this stuff that's going on in the actual hardware itself. That would be super nice. By the way, if you have friends in AMD landia, you need to encourage them towards the Kubernetes community and make sure that things work out in a way that's good for them if they have good architecture. Yeah, I think I've got some slides. I'll try to track them down, but the Rome architecture is very unique in the sense that they're getting away from like the NUMA madness.  And we think that the ease of use overcomes the small bit of like, we're talking nanoseconds of latency here. Yeah, no, I mean, that actually works out, which I think it does. Then that would greatly simplify life because the NUMA madness is kind of awfully painful, particularly in Kubernetes. No, so the comment earlier about the open stack way of doing things, the reason I mentioned the difference with Kubernetes is, absolutely, if you're coming from the open stack side, you're used to granularly specifying on what this thing running on this core, which is in that NUMA zone because that's where the NIC is. In Kubernetes, it's a little bit more I have this thing that wants this NIC and Kubernetes figures out that that NIC would really rather that things who were using it were running in this NUMA zone. And so does it's level best to schedule any cores for that pod into the NUMA zone of the device that it's trying to consume? Does that make sense? Yeah, it makes sense. Yeah, the placement of the CNF, yeah, that makes sense, Ed. And that happens in open stack as well. It's just that there are a few detailed scenarios. Maybe it'll be deviating the meeting today, but there are detailed scenarios where you wanna make sure that the traffic enters to the right physical NIC so that the received path is optimized for the CNF. The placement of the CNF is one issue and how to steer the traffic towards the right NUMA in a server which has multiple NICs. That is another problem. Yeah, I know the unfortunate problem with the current solution as I understand it for NUMA zones and Kubernetes is that it has a couple of sort of presumptions that aren't stated. And one of those presumptions is that a single pod is only gonna really be understood in a single NIC because it doesn't really have a good solution for the, I have NIC zero on socket zero and NIC one on socket one and I have a CNF that wants to use both. It really has no meaningful solution for that problem. Sure, and I think as Jeff was pointing, the CP architectures are evolving and changing, right? I think it might be a good idea to put the boundary with the V-switch and the CNF doesn't care. And if we have to add any NUMA awareness or some intelligence, that can be in the V-switch layer rather than the CNF's one. But we do want a mode that, we want to avoid this inter-socket communication, right? That buses the bottleneck, correct? I mean, we do want a mode that, like... Yes, Soranki, they're the... Yes, I mean, the full vertical alignment, right? Yes, yes. The way that you picture that in AMD, at least in Rome and potentially in Intel, that's gonna change, like, instead of a QPI, you're now gonna have, like, just this distributed bus that spans both sockets and all the different dyes inside of each socket. So, I mean, at least on the AMD side, now that they're moving back into the enterprise-class server market, like, they are not gonna do NUMA from the standard of, you know, this memory lane with this PCIe lane, with this socket all go like this. And I've now had to cross the QPI two times and add an X, Y, and Z latency. Like, some of that is gonna be abstracted. Some of it's gonna become more complicated, depending on how you decide to carve these up in the bios. So, it's gonna be some exploration and it's not gonna be the, you know, X86 we've known for the last several years of I wanna get everything into one vertical so that this memory lane comes out of the same socket as the PCIe lane that I've pinned this VNF to. So, just that spot answer, the only... So, one thing we have to keep in mind is that transition to newer processes is, I mean, it's slow transition all day. So, basically, the existing architecture, right, will there be that for a while? I mean, at least from what we have seen, right? So, we need to sort of be able to support both. Yeah, so, Ramki, the execution of the CNF on a core, which is associated in NUMA, you're right. We need to have an opinion, the placement part, right? What I was trying to say was the CNF, it exiting out of the server via a particular NIC, that can be hidden behind the V-switch. We've got to divide the problem into two pieces. Networking, the exit and entry point to reach the container is one aspect and placement of the container is another aspect, right? So, there's another part to this too, right? Because keep in mind too, like, and this goes to Ed's point, we've got to decide how convoluted and complicated we'll ultimately make this. Because if you build a CNF as multiple services in separate pods and use something like MIMF, are you gonna try to force that all of the pods that are co-located are all within the same like memory address space?  I mean, we're gonna have to like, tease this out and not want to get too bogged down because we'll just recreate all of the headache that OpenStack gave us. Yeah, so I think it's nothing to do with OpenStack, Jeff.  So that, but you're right, you know, that kind of low level details the NSM shouldn't worry about. That's why I was also kind of indicating that maybe we can solve it under the V switch layer. If a complexity is needed there, you know, not that I'm saying that we should go and solve all these issues. This is actually, so this has been super interesting and useful. I do want to move on because we've got some other things on the agenda. The one comment I'll make in closing is that what we've mostly been talking about here is essentially a pod placement problem at the end of the day. And the good news is network service mesh, that's not what it actually does. So we have an interest as a community in how this gets solved and signowed in the resource management working group. And I would highly encourage folks to participate in those spaces. And we certainly care, but that's not specifically our problem to solve. It's a problem we very much need to have solved, but it's not going to get solved in NSM. So one request yet. So the ENSN is a very common hot topic for the upcoming ODL demo, like our next talks and, you know, the ONS around panels, because there's always a, I mean, ENSN means one app interaction. I also see sort of the open stack VM interaction, right? Can you talk about that? So I think what you're saying is, could we trigger the agenda priorities so that we talk about the ENSN now because we're running out of time? Yeah, much appreciate. We have some slides too. We had some good discussion on the use case call yesterday and we have some updated slides actually. Yeah, if everybody else is okay, I'm happy to do that. Is it, does anyone have something else in agenda they'd like to see at higher priority than that? I think this is good. I would just like to say one sentence. So for the upcoming release date, they stayed the same. So end of April 23rd, as we planned, I am going to do on Center PR to update the sites with the proposal for a table to reflect that. Yeah, if you could do that. And if you could just insert in the meeting notes the salient dates, because I remember having them. Yeah, yeah, I will, I will. I don't remember what they were, which is part of why it's done that. Yeah, I am, I am, I am. Okay, let's go to the next one. Many thanks. Cool, would you like to share, Romki? Yes, thank you, Ed. Yeah, I'm good. My pleasure. Can you see my screen? Yes. Excellent. So in yesterday's use case call, we had a very good discussion. What Nikolay point of blue was like Ed, you several folks had kicked off a very nice document on E&SN. So what Prem and I did was sort of took it and then summarize some of our key discussion points just around E&SN. That's fabulous. Steal from me, steal everything from me. That's exactly what they're doing. Oh, thanks. Yeah, yeah. Thank you. So I'll get straight to E&SN's here.  So just to add to, just to add to Romki, it is the need of the hover Ed. We are struggling a bit with OD also. One of the reasons here. Okay. Yeah, sorry, Romki. So the key message here is E&SN. It's a gateway or external controller function as depicted very nicely here. So basically on the northbound side, it's the E&SN protocol here. On the southbound, it's these equipment-specific APIs. That's what E&SN are about. Exactly. It speaks whatever the hell. It's not our business. Yeah. So now, so what we did was essentially took this and then mapped out certain specific scenarios. We'll walk through them. So one case is essentially, I mean, we wanted to start small, not blow it up, drive through examples. What he said was, let's take one case, SRIOV unique VLAN per VF, so complete hardware slicing, right? Complete. Where essentially what we're really saying is, hardware in this topology, hardware port is already nailed on a specific node and PNF. I mean, basically the port is known, where traffic is coming from, everything is known, predetermined. And what happens is E&SN, in this case, the key to note is E&SN exposes one and only one endpoint, right? Thanks. And if you look at the control flow, E&SN requests E&SN, E&SN assigns the VLAN ID. In this case, you're talking still VLAN. Very, very simple. Yep. And then in this example, we said, let's let it assign VLAN ID 100 and VLAN ID 100 gets programmed with a data plane on both sides, right? On this side, on the NSN side, on the port and also on the PNF, right? Yep. So far so good. Very simple. So now enters a little more complex scenario. Uh-huh. So here, what we're saying is, there is no SRIOV, right? Correct. I mean, more, yeah. Interesting one, no SRIOV. Of course, hardware port is already nailed on the node and PNF. And the same thing, the key is E&SN exposes only endpoint still. So basically, as you can see, you have two functions to deal with from E&SN side, the gateway and the PNF, right? Okay. But E&SN, as far as the E&SN goes, it still exposes one and only one endpoint. So the first step here is essentially establishing a tunnel between the port and the gateway, that's the first step, right? Where E&SN finds that VXLAN ID, right? For the tunnels, right? Another next segment. So what you do is then, at the end of it, you create this tunnel between the port and the gateway, right? Correct. And as far as the port goes, you have no idea that you're connecting to the gateway, by the way, right? It's all completely accepted. You're connecting to something. Exactly. It has no idea what's happening. He doesn't even know there's a PNF on the other end. That is correct. Yes, yes. In fact, yeah, we can take this picture and maybe putting a whole box around this. I think that will send a better message. I'll do that. I think that may be a good idea. And there are some boxes in some of the more recent slides that are sort of like nice color coded. This is a network service stuff. And it may even be the case that your gateway box here lives inside the physical network function in some cases, right? All it's really doing is terminating the tunnel, correct? That is correct, exactly. But yeah, but I thought it would be good to depict a disaggregated scenario and put a box around this, right? Yeah, we'll do it. Yep, yep, yep, yep. Communicating is hard. So, so next. So once you have done the VXLan creation, so now goes the second part, right? So it's not over it. So now you are using that VXLan tunnel to signal the VLAN ID into it, right? More interesting. So basically what you're doing is you're still talking to ENSM all this. So now, ENSM assigns VLAN ID. So basically you fix the VNI. Remember you fix the VNI. You know which tunnel you're going, right? Now we're generating a VLAN ID, 100 for that VNI, right? That's what ENSM gives out, right? And interestingly enough, you have to remember that, hey, from here you're sending VLAN ID 100, VNI 1000, goes here, right? And then the gateway could do any translation. You have no idea what it could translate to. It could translate basically. None of our business at all, agreed. Correct. But I thought it was very cool to show the end to end, to show, I mean, a real deployment scenario, right? Yep, yep, yep, yep. No, I get that. One thing to keep in mind as we look at this, and this is something that's super counterintuitive because we're used to thinking about things like VNIs particularly as point-to-multi-point concepts. And so they end up being quite a bit scarcer. The truth of the matter is that when you look at that original tunnel, if we let's bounce back to slide five really quickly, because it's easier to explain there, it's a simpler slide. If you bounce back to slide five, that tunnel is actually not parameterized by a VNI. That tunnel is parameterized by a source IP, a desktop IP and a VNI. Those are the three parameters that uniquely specify that tunnel. Yep, yep, yep. And the reason this is super interesting is that the space of VNIs, which you must select a VNI, is not the global space of VNIs. It's the space of VNIs between that source and desktop IP. That's it. And that ends up making the problem much simpler because the multiplicity of things you have available is enormously larger. And so the likelihood that you would have to do something like add additional layers, like VLN tags along that tunnel is much smaller. Not saying it doesn't happen. I'm sure there will be cases where it happens, but the likelihood is much smaller. In fact, Ed, thinking through this and another question came to my mind. So that was one of the related topics. So basically, so the key is essentially, this is very akin to sort of the MPLS, a label allocation strategy. If you go back to MPLS, so basically the downstream router will assign the label, very similar, right? So now in that scenario, of course, if you, the MPLS world, so what happens with each router is the full-blown headchair, right? So basically, you can be assured that it's highly available because each, I mean, imagine very simple, you're talking to the gateway, which is a router as an example, right? And that implemented in SM, it's got headchair functionality, right? So for example, if one control plane unit goes down, then you have the backup control plane to make sure that things are stable, right? From sort of a control plane label exchange, or in this case, ID exchange perspective.  But the question is, what if that node goes down? I mean, we have headchair, but of course at a global Kubernetes cluster level, not at a node level, right? So how do we handle such scenarios, right? Well, so think of it this way. Effectively, the way we've handled it today, the current resilience and story is, if I'm a network service client and I have a connection that takes me somewhere, right? And I don't really know what happens inside that connection, I just show packets in and they come out at the other end and vice versa. If the network service endpoint that I am talking to goes away, right? The particular instance goes away. What network service mesh will do today is it will attempt to auto heal. And what auto healing means is that it attempts to go and establish a connection to a new network service endpoint that provides the same network service and has the same network service selection criteria. In the hopes that things will be more or less okay. Now, when I say more or less okay, let's be super clear. If the network service endpoint is stateful, something like a stateful firewall, unless the guy who wrote the stateful firewall wrote it so that state could be shared among replicas, you're going to have some state loss there. But network service mesh will auto heal those connections to whatever the network service endpoint is. So there is resiliency built into the system there, definitely. So the point to note here, it is like the ID itself, right? So basically, let's go to this very simple example, right? This simple example, even simpler. So basically, the IDs, like you have some running ID generation here, 100, 101, et cetera, right? And then this function goes down, correct? So basically, this is the one doling out IDs. So you, and this was the one managing these IDs, right? Correct, this will an ID space. So now, and then now this node connects to some other node and moves ahead, right? So basically, then that will start giving out IDs from a different space, right? Not necessarily these numbers, correct? And this- Yes, yes. Go ahead. So there, it's just that probably we have to work out the scenarios, how this is all going to line up, right? I mean, basically, like, hey, this gave out like, say, 100 to 200, now that region is unusable, right? From the spot perspective, because that function, that node died, and then now it connects to a new node, we have to see what that new range, that don't start, right? Correct? Which may not be the same. Let me ask the simple question here. In this picture, who do you think is assigning the VLAN ID? The ENSM is assigning the VLAN ID. Okay, good, good, good, good. That's my understanding as well. It's always good when we share the same understanding. And you're saying when the ENSM goes down, then the next place that someone tries to connect to may be a different ENSM, which may have a different set of VLAN IDs that it's allowed to manage within this context. That's considered, yes, exactly. It may completely different range, yes, yes. Yeah, and the way the current auto healing would work is essentially the pod still has its kernel interface. The cross-connect to the tunnel would just change, right? So as long as it is the case that the other ENSM does whatever has to happen to the physical network, such that now let's say a VLAN ID 200 is assigned, goes to the same or an equivalent physical network function, the pod never sees any of that. The pod literally never sees the VLAN ID, right? It doesn't know. Yeah, ideally it should never see a VLAN ID. It should just deal with the V-switch locally. Yep, I do apologize. We're running up against the top of the hour. Yeah, so what I'm saying is, these are all like, it can get a little tricky depending on different scenarios, right? I mean, basically in some, there may be a V-switch or there are SREOV cases, this is some details you have to really work out how this is all gonna come together. Because typically, how this is all handled, it's all through some global ID management, right? But this is basically taking into account all the policies. If you recall, we are discussing the use cases, right? In some cases, you're assuming like, hey, these are all point-to-point, some it's a little, I mean, slightly different from that. Yeah, so Nanti, is your proposal that the downstream guys should allocate and that's a better model? And it kind of takes a page out of the MPLS? So I'm also, I would love to think slightly beyond just a simple downstream allocation because so the point to notice like, no matter what we do, right? Even if we do like a point-to-point allocation, we are reserving VNI, sort of from a no V-switch perspective, right? And we have to see the global effect. At least to my mind, I don't think we have worked out all the scenarios, when things go down, how the ID management is gonna happen across. We're gonna have to cut it short at the moment because we're already five minutes above and we're holding people back at the moment. So can you add this to the meeting notes for next week and we can continue the discussion? Yep, yep. Cool, thank you so much. Thank you very much. These are super good things for you to bring up, Ramki. They are things we need to walk through. I think at the end of the day, they don't end up being tricky. They're just the things that make them simple are super unfamiliar. And so it's really important to talk through and make sure we got them nailed down. No, exactly. That's what I said, because you're getting to the next level of detail on implementation, right? Want to make sure this is all fully understood and nailed down, including what are the different strategies for handling these complex policies around isolation and all those. Cool, cool. Take everyone, see you all next week, same time. Thank you, bye.


you know that this season every day it's a new injury to update us on so what do you have for us tonight. Ryan unfortunately this season has been plagued with injuries for the Yankees as we know 13 players currently on that injured list there's two more that are not in the lineup tonight Gio Arcella and DJ LeMayhew both out of the lineup but neither of those guys in jeopardy of landing on the IL DJ LeMayhew actually had an MRI on that right knee yesterday it came back showing only inflammation so they'll give him a little bit of time to rest Aaron Boone did say that he would be an option off the bench today but it seemed like he'd like to stay away from LeMayhew if possible it's a short series here then they have an off day on Thursday that would give him a lot of extra time to rest again he is available if a situation would dictate as far as Gio Arcella is concerned I spoke with him in the clubhouse he actually got in the cage earlier today took some swings and that hand not really an issue he did say it is a little bit sore after getting hit on the hand and exited in that game early on Sunday in San Francisco but those x-rays came back negative and he was definitive when he said I believe I can play today so that's another option off the bench for Aaron Boone. Clint Frazier was out here during early batting practice on the field for the first time since landing on the injured with that ankle issue he took BP also did some light running and played catch in the outfield the next step for him will be to continue to do so then they'll really ramp things up as this week goes on he'll probably wind up at Yankee Stadium on Thursday on that off day to see the doctor and then they'll develop a plan from there as long as everything from now until then goes as planned with him Frazier is said all along he believes that he can play through this injury the Yankees being overly cautious right now just making sure that that ankle doesn't turn into a long-term issue for Clint Frazier now we originally told you that John Carlos Stanton was going to join the team here and continue his rehab on the road here in Arizona however there was a change of plans he's actually still in Los Angeles he'll continue his rehab there then when the team heads back home he'll join the team in New York see the doctor there and then continue his progression no word on the timetable for John Carlos Stanton at this point there is some good news on the injury front when it comes to Miguel and Doohar and Troy Tula whiskey both those guys were active in Tampa today playing for the tarpons as you can see and Doohar 0 for 4 he started at third base that right shoulder strain that has kept him out he actually has a slight tear in his labrum Troy Tula whiskey one for three as the DH to run home run in that one Aaron Boone saying that if and Doohar continues to progress they do believe that he could be back as early as this weekend Troy Tula whiskey not too far behind them with both of those guys they are not concerned with the results right now they are more concerned that both of them get through the games healthy and and do harm in particular that he is able to play back to back days at third base


So my computer, Tom takes all the pictures, including that picture. So where do you suppose he took the picture? Pardon? I can't hear. Backwater of where? Canada? No, actually not Canada. Not India. Almost certainly not. That's close. Somebody said Australia. So up until 300 years ago, in European languages, the word black swan meant an impossible event.  They didn't exist. Until a Dutch sailor exploring the furthest regions of the world sailed up a bay in Australia, and here was just a whole horde of black swans. This actually happens to be in New Zealand, where there are also some black swans. But he didn't quite believe it, so he captured a few and brought them back to Europe to prove that there was such a thing as a black swan. And so now the word black swan got changed to mean something, an event or something that was completely surprising, and you certainly didn't expect it. But if you thought about it, you should have probably known that it might be possible. However, it wasn't really apparent until it happened. So let me tell you about a guy who understood this concept of black swans. His name is Jan Wallender, and he in 1970 became CEO of a bank called Handelsbanken in Sweden. And he was rescuing the bank from some sort of disaster, which I don't know. They're trying to grow, and they tried to grow too fast, and ran out of money, and the board was fired or resigned or whatever, and he had been running a bank in northern Sweden. And he said, okay, I'll come run your bank, but you have to let me do it my way. Now, Wallender was an economist, and for all the bad things that I'm able to say about economists, there's also a few good things. In this case, he said, as an economist, there is one thing that I know absolutely about economics. And that is, every approximately 20 years or so, there will be a serious economic downturn. Absolutely guaranteed. And do you know what else? When it happens, we're all going to look at it and say, oh my gosh, I should have seen that coming. But you know what? We'll never see it coming until after it's happened. That's just the way these things are.  How come we didn't see it? So he set up his banking organization, not for growth, but for survival in those 20-year black swan events, which have actually become a little more frequent these days. But still, the idea was not to set up a bank that kept trying to grow, but that was profitable. And the other thing he did was he tried to create an environment where there were decentralized folks making decisions, not a central organization making decisions, but small separate organizations making smaller decisions, a lot of them, independent agents. So he got the idea that every branch manager should run their branch as if it were a separate business. And he held them accountable for things like cost to income ratio and income per employee. But outside of those things, they could do whatever they wanted. They could hire however many people they wanted. They could pay them whatever they wanted. They could have their own IT departments and typically did. That's not really cost-efficient, is it? But it was a concept that the branch manager could run that business how they wanted. And if they wanted central services, they had to ask the central organizations to deliver stuff, that's the only way central organizations were allowed to exist. Handelsbanken has company-wide profit sharing in one of the early companies that did.  Because the profit sharing has been very successful because the bank has been very successful. It's not in all countries in Europe. It chooses which ones to expand in most of the Nordic countries and the UK and a few others. Because it's not about growing big, it's about being successful over time. And so one of the things he did is he set up a bank where the things like loan decisions were not made like every other bank by a central organization. In most banks, if you happen to want a loan and it's a great big loan, you have to get central approval. In Handelsbanken, if you happen to want a great big loan, well, there might be some central scrutiny.  So what they have is a loan portfolio that's vetted by the local branch manager of the company. And branch managers care a lot about making sure that whatever loans they make are good loans. And so every time there's an economic crash like this that they should have seen come but didn't, it turns out Handelsbanken has a much healthier loan portfolio than all those other guys. In fact, they have financed the Swedish banking system oftentimes during economic downturns. So you can see the blue line here. That would be Handelsbanken. You can see the gray line where every other bank in the environment in the Nordic countries just tanked. And so the idea of setting up a company so that it can deal with black swans, things that are rare and unexpected, is an interesting concept because in our world, in this technology world that we've been living in, and I've been living in for a lot of years, stuff changes all the time. And you think we should have seen these things coming. We're not going to see them coming until they're passed. And only rarely do we have the insight to look ahead at the way the world is going. But I can tell you one thing for sure. There will be all kinds of changes. And nothing really lasts that long. Companies even don't last very long. And certainly ideas don't last more than a decade or so before there's some other new great idea that replaces it. So black swans are going to happen. In fact, if you take a look at this, this is a picture of the average lifetime of a company in the United States on the S&P 500. Now to get on the S&P 500, your company has to be one of the top 500 by market capitalization in the country. Which means you have to be a very successful business just to get on. And yet the companies are falling off at a pretty fast rate. Right now, maybe in the vicinity of a company might last for 15 years. And these are the successful companies, not the startups that don't make it. So it turns out, if you're not careful, you could be working for a company whose lifetime is shorter than your career by a factor of two. So why is it that this happens? Why do companies disappear? Well, there's a theory. I'm going to go into a little bit today. And it's a theory that one of the biggest problems is that these companies are focusing too hard on their own success. And they're not focusing on those black swan events that are sure to come. So it's what I'll call the optimization paradox. This is Clayton Christian's classic work on disruptive technologies. You may have heard of it. He studied disk drives in computers. He said, biologists study fruit flies. The fruit flies of our industry are disk drives. But this happens to be a picture of cameras. So on the blue line, what you have is a digital camera. And it was quite successful. And the solid line shows the advance in digital camera technology. And then the red line is a digital camera on a phone. Now, when phones first came out with digital cameras, if you were a serious photographer like, say, Tom, you wouldn't go near it, right? Because it just wouldn't have the kinds of capabilities and features that you were looking for. But as cameras got better on cell phones, they eventually crossed the line. Can you see that dotted blue line? That's what customers are really looking for. Now, the digital cameras themselves got way better than customers were looking for. And there was a point in time when digital cameras on phones got so good that most people stopped buying cameras anymore. Yes, now there's still a few crazy guys like Tom that still buy digital cameras, but not that many. It's going to be harder and harder for the digital camera world to support its future growth. So, if you think about that, here are some disrupted technologies. Film. I worked at 3M, that was one of the big businesses. Film. Who does film anymore? VCRs. You know, record stores, movie rental places, pay phones. Have you seen a pay phone anymore recently? I saw one in London, wasn't being used, but landlines. My son has a cell phone and will not have a wired landline ever. Long distance churches. I heard from a telecom executive that he said, do you know what Skype and those other voiceover IP organizations have done to the telecom industry? They have taken $3 trillion a year of revenue out of the telecom industry. And I'm thinking, oh, that breaks my heart, but still, $3 trillion is gone. Those telecom companies used to have money, they used to have long distance charges, and that was the thing that fueled their growth. Another thing, newspaper classifies. Newspaper classifies have fueled the growth of newspapers, have been the revenue stream forever. All of a sudden, over the last decade, newspapers start failing why? Because they don't get the classified revenue anymore. That advertising has gone online. Maps, who needs it with a GPS? And on and on, dictionaries, Wikipedia survived for a while, and all of a sudden the other two encyclopedia organizations just evaporated, and so on, and even email. I mean, mail is getting to be less and less useful, and my granddaughter tells me that email is for old people. So she says it's all about texting. And there'll be something new after that. So the thing is, there's all kinds of stuff we take for granted now that's going to disappear in the future. So this is a company in Finland, and these are actually slides that we saw from the president, CEO of this company, Christian Helverson. And this was a little about a week and a half ago, or two weeks ago in Oslo. And he's talking about his company. His company is a subsidiary of an organization called Sheepside Media Group. Now, what I have here is a graph of the revenue stream of newspapers, typical advertising revenue stream, starting in 1950, and going through 2000 when it peaked down to 2010. And the typical red, the red line, see that little tiny red line? That is the online revenue of newspapers. You can see why newspapers are failing. They miss something. This is the revenue stream of the Sheepside Group. And on the top, you can see that right now, 35% of their revenue comes from online sales, and it's growing rapidly.  How did this happen? In fact, I don't know of another newspaper in the world that made the interesting decisions that this little group of Norwegian newspapers made. Right now, Fin.no, which is an online stuff. They do jobs. They do real estate advertising, cars, general merchandising, display advertising, travel, all that stuff. And in Norway, they are the number one supplier of that. And number two in travel. And they have 96% brand recognition. And without even prompting the name, 82% of the whole Norwegian proper relation will say, oh, Fin, yeah, we know them. They have 2.5 to 3 million unique visits per week. They have like 110 visits per Norwegian. And the Norwegians spend like 21 hours a week on their site. And I think it's a week. They spent on, yeah, I think that's a week, but I'm not positive. But in any case, something like 20% of the Norwegian GDP is advertised on their website. It's really popular. And by the way, financially very successful.  And back in mid-1996, or in the mid-90s, they said, hmm, maybe we should think look into this online advertising thing. So they started a project. The focus was to defend their print revenue in classifieds. It was a sub-brand of the newspapers. It was a, it didn't have dedicated folks. It was just a project. And they bundled it in for their normal business model, and it failed. So, so much for that. That was an interesting experiment. Now, come four years later, in its 2000, and they try something else. They created a subsidiary called finn.no. And there were chartered to attack the revenue of the newspapers. What company charters a brand new subsidiary to attack its most important revenue? That's an interesting behavior. You don't see that very often. This is why most of the newspapers in the world are failing, because it didn't occur to them that it would be a good idea to attack their basic revenue stream by starting a subsidiary that would take it away. This brand was advertised in newspapers. And in fact, here is an advertisement in the early 2000s in a newspaper saying, if you take a look, the little print, see the boat? And see right below it, there's a little bit of print.   So in the newspapers, they were driving their major source of revenue to the online subsidiary.  So, you know, kind of an interesting organization. These days, they're doing very well. They're revenue from, on the left side, from cars. Okay? You can see in the light blue, the revenue from car advertising in print and on the right, from car and the dark blue, car advertising online. And what happens is, they have more car advertising online revenue than they ever had in print. And if you look over on the right, these are revenues from general merchandising. It's like, you know, I sell, I have something at home. It's sort of like what eBay does. And this is revenue they never had before. And here we see an enormous amount of revenue even now. So his conclusion is this, a will to cannibalize your own revenues has made the global leader in transforming traditional media to online. It's easier to act before market changes than to play catch up afterwards. And disruptive innovation is not just a way to transform your company, but it's a way to add entirely new revenue streams. So think about it. What companies do you work for that would have, you know, just the plain old wisdom to head in that direction, to attack their major revenue stream with something entirely new and very small when they started out? In fact, he noticed that now these days a lot of the browsing of their online site is coming from mobile phones. And not the purchasing, but the browsing. And he showed us a graph. And he said, you see this graph? This is the mobile phone interest in our website. It looks exactly like the graph of online interest in classified ads 10 years ago when we started up. We would be crazy not to realize that mobile phone is where we have to go. Because it's just the same pattern. But they are picking up the signal, that little tiny signal, way earlier than other people, because they've done it before. You see a change coming. You have to pick it up on the front, not on the back.  And most newspapers haven't been able to do that. Very, very few. And this is a story you see in industry after industry after industry. And guess what? It'll probably be in whatever industry you're in, too. Now, I say that to a big crowd. So I'm confident that it's true for 50, 60, 70% of you.  So here's the idea. If you disrupt yourself, you're going to be in a lot better position than if somebody disrupts you instead. This is a case of Harman. Dinesh Patwell took over. I hope I said his name right. Harman. Now, what Harman does is Harman makes information systems that go in cars. So we were just in Munich at a conference, and it was being sponsored by BMW because they're looking for software programmers, yeah? Everybody is, even BMW. And so they took the speakers out for rides in BMWs to show off their BMWs. You would be amazing. All these screens in the front, in the back, huge GPS systems, really, really high-end stuff. That's what they made up until 2007. In all of the high-end cars, they were the ones that made the special information systems that were aimed directly at the special car line. However, when he took over as CEO, he had an interesting problem. They already had 70% of the market of high-end systems. So do you see a problem there? There's no growth. There's no growth if you own 70% of your market. And so his charter was, hey, find us a new market. Well, it's not like they hadn't tried. They had a group of German engineers. They were organized by specialty or competency area. And all of their bids to car companies were special, individually designed for each car environment. And they tried to get the engineering group to design an inexpensive deal information system. And it was really a failure. It was not saleable. It was cheap. It didn't work very well. It wasn't modular. And so here we have a company where 70% of two-thirds of its revenue is kind of not going to grow anymore. So he basically said, we need to start something new. And at that time, Laundoy happened to be also working in Germany as an architect in the product development environment. He'd been there for a couple of years. Both of these guys have gotten their degrees in India and had been involved in global commerce in all kinds of countries around the world before they got these particular jobs. And so Laundoy said we should design something that's modular, but he wasn't making a lot of sense in the engineering department. So the CEO commissioned him to start up his own little separate organization very much like Fin.nl so that he could target at a high margin product at very low cost cars. Because they decided, hey, where's the growth in cars going to be? Let's say India and China. That makes sense. That's where people don't have many cars yet. So we need to get information systems for cars that are going to sell in large numbers in India and in China. In order to do that we have to have small modular things, not specially designed for each model, that are still high margin but come off as inexpensive when buying them. And so that was the charter of the group. Now, one of the first things that Laundoy did was say, I can't do this in Germany. Can't do this with the current engineering structure. So he created a software team in India. I think it was Bangalore but I'm not sure. And a hardware team in China and he had about two or three German engineers. That was his team. Now this is a just truly distributed team. We hear that's a bad idea but in this case it was a much better idea than having a team sitting in Germany. So this group in the software group here used classic agile iteration processes. The group in China was working on the hardware stuff. And one of the decisions they made it was a really interesting decision was there's so many GPS systems out there. Why don't we just create a socket that any GPS system in the world can just plug into instead of trying to figure out how to create a GPS system.  It was simple it was inexpensive, it was modular and it was high margin. And they couldn't get the sales force to sell it. Because the German engineer said it's just going to be cheap junk. You can't design it that way. You got to use our method of product design in order to make a really good product and the sales force said why would I want to sell something that's so inexpensive. But they had to be ordered to sell the product.  And so it turned out to be a very nice successful product after that first sale sort of opened everybody's eyes that Toyota would put them in a whole bunch of stuff and now it took only 18 months first of all 18 months to develop the product. And another 18 months after that they had double the sales of the company. They already had 3 billion dollars of profitable sales in the first 18 months after the product was introduced. So that's growth. But it's growth by understanding that you have to take and disrupt yourself. What car makers are going to still want to spend all of that money on this high end system if suddenly they can get a really nice low end system instead. So very few companies have this courage to go for something that's going to cause their own basic revenue stream to die. But if you don't take that attitude you are going to have an interesting you know future. So the next thing I want to tell you about is I told you the telecom industry has lost huge amounts of revenue. So they are being disrupted like crazy. Telecom companies are having extremely difficult time surviving because they have very intense competition. And they have very much less money than they ever used to have. And they also are operating in what I'll call an unhealthy marketing environment. So I can't tell you how many companies have there would be the product management group and the development group and we would work with them both and the product manager said the development says this guy he promises all this stuff we can't possibly develop. The product manager says I know that. But all of our competitors bid all of the features they pretend that they're going to be able to deliver them. So we have to pretend we can deliver the same features as our competitor or we'll never get any kind of bids at all. And in fact that's probably true. So you have a really unhealthy environment which in the end just drove all kinds of ridiculous excess work in long hours into the development environment. Now Ericsson is a company that sits inside of that particular unhealthy dynamic and the question is what were they going to do about it? As a very proud Swedish company of engineers who are really good they decided that even if they had to bid stuff that they couldn't do they were going to figure out how to do it anyway. And they weren't going to compromise on quality. What that meant is the way they used to do product development simply wasn't going to work anymore. They were going to have to figure out a way to develop the same amount of stuff in half the time. And starting about four weeks years ago various subsections of Ericsson decided they were going to change the way they developed products to some sort of lean or agile or flow process. They experimented lots of different ways because the problem, you know we had a couple of Ericsson managers in one of our classes and they sat quietly for two days and at the end they said these are all good ideas but our product is immense. It's testing, I mean just testing it takes all of this hardware it's impossible for us to imagine how to apply these kind of small team ideas to something as big as we are. And you know I said I actually don't know how to tell you the answer because I don't have it. But I can tell you if you follow these principles of flow and of high quality and the kinds of things we talked about and apply them into your world where how it works you're probably coming up with something because you guys are pretty smart. And so they did, they went home and they came up with an idea and they did two cases of how Ericsson did this. The first one was an organization the first thing they decided to do was to quit having projects. Projects are great big batches of stuff that take a couple of years and they decided they had to do stuff in a small flow environment. The second thing they decided was that they were going to completely decouple their releases from the development cycle and they were going to release on a regular basis not as frequently as you know the web companies going every day but twice as frequently as they had been and whatever was ready to go into final testing system testing was going to be tested and go. And by the way if the engineers put anything into final system test that was not going that had any defects in it that wasn't going to be acceptable. So they were going to have good solid well tested stuff going into their final test and integration. They created small multiple multi-discipline teams in so much as it was possible as the teams came off of projects they would form small teams and they would say take a table like this okay guys we got a new way of doing things here. Now here's the deal our product office we are going to select features the ones that we think is most important for the customer next. We're going to tell you how long you have to develop the features and it's going to be usually about three weeks. And you guys your cross-functional team you have different people maybe not everybody you need but you're smart you're competent you've been engineers here at Ericsson for a good long time and the thing is we're giving you a deadline because we just don't have any more time than that to use but we'll do one thing for you. We will give you a sales engineer and you talk to that guy and he and you can come up with what does it mean so you just get this one line feature and a customer engineer and you work out what can we do in three weeks to make that feature work and then you do another one and then you do another one. Now they kept intact their competency groups but the architects and component specialists became consultants to the teams like this instead of becoming really people who told them what to do and they also found that if there was a team here and a team here and you had a specialist that they needed and they had a specialist that you needed the teams would start swapping specialists under the table without any management involvement because they had these deadlines that they had to meet but they could decide what it meant to meet the deadline so a concept where scope is truly variable but deadlines have to happen because they had a contract they had to deliver on and they couldn't be late and so that's pretty much how they did things after a couple of years as they started delivering this into the market their results were able to develop stuff twice as fast which they needed to I mean that was the whole plan here was faster time to market but interesting additional things that they got almost as they said for free a much higher hit rate why because this team here is talking to a customer engineer and they have only three weeks so guess what between the team talking to the customer engineer and the deadline you only get the important stuff you don't get all sorts of extra junk and they had significantly higher quality which was a surprise to everybody although not to me because that's very typical of a rapid flow process and they basically had engineers that were much more engaged in the development process in solving customer problems and here is another case of a different group with Ericsson similarly to that one side team this one here was basically China Stockholm and one other place in Europe this group had five different organizations around the world and they had a similar problem but not quite the same this was formed out of the old Siemens group that got bought by Ericsson Networks and the story we heard from Hendrik Esser who used to live in Dusseldorf but then moved to Stockholm and started their reorganization that he was implemented so anyway the same thing the past wouldn't work for the future they had to do things faster and the problem here was that they were making promises and not coming through and they were a very proud company and they were not going to do that if they made promises they wanted it to happen and so the first thing the management group did is they started as Kenji would say they started talking to people project managers and developers and say how come we think everything is okay and then it doesn't work and as Hendrik told us we discovered the uncertainty cone the cone of uncertainty which says the further out you estimate something the more uncertainty there is in the estimate and if you happen to be the management group you kind of are way too optimistic to boot so they realized the further out they estimated the more uncertainty there was and their conclusion was uncertainty exists it lives we have to figure out how to live with it and so the most important thing to Hendrik yes there was let's not increase the uncertainty by pretending that we can estimate by pretending that we can put pressure on teams and get them to deliver let's trust our teams to know what they're doing and let's figure out how to get rid of as much uncertainty as we can now again they started forget projects let's stop doing those let's manage features instead and Hendrik is the head of their program office which decides which features the features are not necessarily quite so small but they're going to take still about three weeks and they might be two teams or three teams of these relatively small teams now in both cases we have multi-site organizations but each team is at one single site so and in this case you might have two or three teams working on a feature and so they'll all be at the same site whichever site it is they decoupled releases from development and the next thing they did was they reorganized all the managers which I found to be rather remarkable they said they kind of told them next week you get to re-interview for a new job they had a central planning organization which coordinated which teams are going to work on which features next and the features were they said look the thing that estimates do is they introduce uncertainty let's stop estimating one deadline dates let's ask the teams to estimate 10% likelihood of completion 90% likelihood of completion then we'll have a range we can live with and so they had every team that was working on a feature every two weeks they estimate or maybe every week one or the other can't remember estimate how long it was going to be until that feature was done so Henrik drew the sketch he says let's say I got four features I have a release coming and I need these four features in that release and here's the data I have from the teams the data shows you can see the range of when it might be done the 10% 90% so let me ask you is this going to make it feature two sure features three and four nah I don't think so and so he says okay so that's fact the first thing that I do is accept that that's the way the world is the teams are honest they're doing their best and what we need to do is one of three things and we do it early we either take some scope out of the feature we add some teams onto the feature or remove the feature to the next release but you know what as the person who's making those decisions I have long range visibility into those decisions and I can make them in a way that make sure that I deliver what I know customers want so I get a lot more certainty by creating a range of estimates and by believing the estimates that the teams give me so I don't know the rest of the results from this but these are cases of a big organization making dramatic changes basically because they're in an industry that's being disrupted and again I'll say if you are working in a big company and the company is in an industry that has somebody chasing after it you're probably going to have to make some big changes better to make them earlier rather than under pressure later because they're so much harder when your company is running low on money and the market is eating you up so the big dramatic changes of how you do things are much better done sooner so let's talk about this concept of innovation one of the more interesting companies in innovation is Pixar Pixar made oh these are various of the movies I don't know if they show whole bunches of you see any of these movies Toy Story and all that okay yeah I actually had to go watch one when I first read about Pixar but anyway really cool movies and Ed Camel is the head of the graphics design before Pixar was formed when Pixar was first formed he had a team of really really good graphics software guys and they used to in the bay area they used to do all sorts of public presentations on all of the cool rendering stuff and their tools and all of that but he dreamed of being the guy that produced the first full length automated film with computer graphics and he teamed up with not just Steve Jobs but also who's the artistic guy forget he teamed up also with a really good movie director and they produced the first film and the interesting thing and the unusual thing is that they have produced film after film after film that's been successful the movie companies don't do that they do occasional ones but not these guys they have been a very reliable innovator and what Ed Camel says is what you have to do in order to get creative people and technical people working well together is think very hard about the environment in which they work together and if you want innovation you need to get different kinds of people working successfully on the same team and in filmmaking and in many other kinds of complex product development creativity involves a large number of people from different disciplines working effectively together to solve a huge number of problems having been in product development I can say that's for sure true their teams tend to be about 250 people getting people in different disciplines to treat one another as peers is just as important as getting people within a discipline to treat each other as peers but it's a whole lot harder so one of the things they did for example is they matched the salaries of their technical people and creative people which was like really a strange thing in the area at the time so that everybody would feel they were part of the same organization they created a university where people could study the disciplines of technology and they would be able to do that. He says one of the things if you want to foster creativity is empower the people in your company who are creative. If you have a good idea and a mediocre team they're going to screw it up if you have a mediocre idea and a great team they will find a way to solve it or they will bury it one or the other so there's not any real people at least leading a team. You have to create a peer culture. You have to create an environment where everybody in different disciplines is treated in a way that they see themselves as helping the other people on the team create a successful idea and that they feel free to offer their ideas at all time. It's safe and lastly you have to create a learning environment so you have to think about the mindset that everybody is learning. Another guy who is really interesting and does an awful lot of research into innovation is Scott Cook. I think, well probably yesterday afternoon for the few of you who might have been there I talked about Intuit and how they did an interesting start up here in India but anyway Scott Cook founded a company a very successful company and he started being CEO so he had somebody else become CEO and he started investigating how do you keep a big company healthy? How do you keep a big company from becoming obsolete, from becoming too focused that they don't see the next generation of interesting ideas and he said only a few large companies actually do this. Only a few large companies have been able to sustain growth over time by coming up with a company that is famous for that over 75 years. This company I work for which is 3M as one of the reasons why this seems so right to me because it's kind of the way we always worked. These companies share a common practice. They have systems in place and we certainly did in our company that encouraged small, cross-functional employee teams to conduct inexpensive experiments, frugal inexpensive, don't cost very much to use these teams to try things out and let them experiment. Don't tell them what to do. So when you want this how do we organize for innovation? Here's some of the classic what do you want to do if your company is going to think we need to organize for these black swans that might be coming our way. We need to organize so that we have creative innovative ideas springing up inside of our company. This is what we do for the market out there comes and eats us up. George Saki in this book he wrote and this is about almost 20 years old but the research is pretty current. He says that when development programs are managed and executed by a small experienced dedicated, this is all they're working on, decision innovators can introduce four times more product in the same amount of time with the same number of people. That's productivity and it is the concept of having people from different environments, design, engineering people who understand the customer world all together on the same team trying to figure out how to create an innovative product. You need one or two leaders with deep expertise. If you have three environments you have to have one or two some companies do three. Two is common. In many companies like for example let's take GE Medical GE Medical instituted the chief engineer concept approximately what five, six years ago. They have done an amazing array of innovative new products. One of the things that they did with this concept of chief engineer is put a single leader in charge of having the ideas and making sure this small team comes up with creative ways of doing it. On that team there was always a marketing person, there was always a couple more development folks. There was always somebody from quality somebody from manufacturing usually a couple of other functions and always a finance person because you wanted the product to be digital and you couldn't in almost any company do financial projections without an account. They just don't get believed. So you need an account and that is actually the structure of most startup companies. You have an entrepreneur and you have the various functions and they're all basically equivalent. They don't have like one function telling the other functions what to do. They have various discussions about how are we going to make a great product and you get creative efficient designs. These guys do not have a whole bunch of money. They therefore have to be creative and they have to be efficient. Rapid end to end experiments at 3M we had this mantra make a little sell a little, make a little sell a little. That's the only way you're going to learn and high tolerance for risk. So when you have an organization structured like that you have a much at least some of those kinds of teams you have a much higher probability of innovation. So I want to conclude and it looks like I'm actually going to have some time for questions. But let's just go into this last part here. There's an awful lot of companies that think productivity is everything. How many of you work for companies where productivity is really important? That wasn't actually a really high percent but there were still plenty of hands. So probably what you want to do is not think so much about productivity as impact because you can be as a I think it was a Deming or Drucker who said there's nothing quite so so ridiculous that wasn't the right word as doing very efficiently stuff which should not be done at all. So that's just like lost productivity. So what you're really looking for is impact. You tend to get more productivity in the right areas. Think again about the first Ericsson case study. They got actually in one year they got much higher hit rate on their products. They develop the stuff in half the time. How did they do it? Not by doing twice as much work in the same amount of time. They didn't get it that way. They probably got a little bit more work than they would with other areas because they didn't have multitasking because in fact in Ericsson in both cases teams worked on only one thing at a time but they probably got it because their hit rate was higher. So the stuff they did spend their time on was the right stuff and you have to figure out how to do that. That is more important in productivity and software development by a large amount. How many companies work for companies with the predictability is really important. I have me too actually. When a big company goes and makes a forecast to the stock market if they're off it can really kill the share price. Having a way to be predictable seems like a good idea but actually something that's even more important in predictability is the ability to run experiments. This is a phonograph machine. This is an invention by Thomas Edison who was actually quite a great inventor in the frame around the 1900s and he had the saying that he if he tried 10,000 things that didn't work it meant he learned 10,000 ways that wouldn't work so the next one might be the one that does work. The concept of constantly doing experiments doesn't mean everyone is going to be a success. If everyone of your experiments is successful you're not experimenting. That's not an experiment. Efficiency. How many people efficiency is really important in their companies? Okay. So I propose instead of efficiency what you might want to think about is decentralization which by the way is notoriously inefficient. Admittedly inefficient. But think about Handelsbanken. Think about Harmon. Both of those companies created decentralized organizations moving from centralized one group making all the decisions into multiple units that were not centralized and duplicating various functions. And they did it because they figured over time if you don't have a ton of independent agents making a lot of decisions you're not going to get such good decisions. Now there's another guy that is an absolute believer in this concept of having a lot of independent agents making independent decisions. That happens to be Jeff Bezos of Amazon. He has a concept that teams should not be any bigger than two pizza teams. Teams that can be fed by one or two by two big Seattle pizzas. So that's like 10 or 12 people. I've seen teams bigger than that but that's his idea. And so because he thinks all teams to really be efficient should be effective, should be two pizza teams. When they started moving from the classic architecture, big front end big back end, back in 2000 to a service orientated architecture and they kind of led the industry in this and then started selling it as Amazon Web Services. They decided that they were going to form services around team size. So their services are the kind of services that a two pizza team, that size team can do. They have contract interfaces that are hard between the services. And on a team, they have everybody from people to decide what to do to people in operations that are going to deploy. In fact I kind of wonder if that isn't why they invented the cloud because the only way you can allow tons and tons of people to deploy is basically to have the kind of Web Services that they actually invented. So when it comes to decentralization it doesn't necessarily feel efficient but all those little separate thinking teams, BISOS believes, are what makes your company creative. And so if you want to be innovative and creative, you really efficiency focus, optimization is the exact opposite. So there you go. What are you going to do? Be efficient? Are you going to be innovative? And lastly from making money this sort of feels off my talk this morning those of you that heard it, to making a difference. Because getting people interested in helping your company be successful has to do with creating an environment where people care about what it is that they're doing. This is really interesting. In the US we have a new kind of corporation. It's a B Corp. Now why would we need B Corps? Well, it turns out that most companies if you happen to be a shareholder and you don't like the way the company is doing you can sue the company because they're not maximizing shareholder value. B Corps were formed so that companies would be allowed to do something other than maximize shareholder value. Which is really interesting. They had to have a new company structure in order to make that sort of legal. So anyway when it comes to the things you might do in order to foster innovation in order to be ready for all those black swans you really need to change the way you think about what's important. And this isn't just you. It's at the management level of the companies that you're going to be managing in a few years if you're not managing them now. So I think it's time I could maybe take a question or two because we actually have a few minutes left. Anybody interested in asking a question? Well you're still sitting there. There's a question back there. Can you shout it out? Yeah. Go ahead. For you to carry out effective decentralization you really need mature trusted experienced people otherwise you have a risk of you know, waste like in case of loan I absolutely agree. But in case of high growth you also take time to get people to a level get them accommodated to your culture how do you deal with this contrast situation in that case? Okay. Every single person in a decentralized group does not coming out of the box need to be brilliant. They need to be good in their technology but you need one or two leaders who really do have deep experience and competence that you can trust to create the team around it. So you do need leaders with deep expertise and skilled team members. If we go back to the Hartman case study just think about this because Hartman is a really good example of it. One more. Hang on. Landui when he formed his teams he formed a team in India and a team in China. Those teams were not part of Hartman before and he took a couple of really good seasoned engineers from Germany but when he put a group in here in India and when he put a group in China he hired people who really knew how to lead and had good ideas. He was also a system architect so he had a very clear idea about what the modularity had to be. So yeah, you need some good people not everybody but you need a really good leader that has vision and you need a few good people to start the other people up. Yeah. So the question is go ahead and say it again. What checkpoints? Now I just wanted to know how does the leadership know well let's say they decide that they're going to disrupt the organization and they're going to do something different. What checkpoints? How do they know that it's working? So let's go back to Scott Cook's concept that you need to create small and employee teams that are encouraged to run experiments. That's the world I lived in for I don't know 20 years and I had checkpoints. The first checkpoint was that we could spend some amount but not really more than 15 or so percent of our time unless we could attract people like from R&D to work on our teams and if we could if we were able to entice other people to be interested and excited about our product and form enough sort of mass to create and move the product along we got to do it. Why? Because the assumption was if you could attract people to the team on a more or less volunteer basis then there might be something there. So there was a little bit of we believe in whether it's popular. And then the second thing was at a certain point in time when that product was about to become something that was real it could not go out for test market without branding so it had to have 3M brand and had to have 3M quality and the team at the point in time when they started offering it for sale had some classic 3M commercial targets to meet once it was out there. So as soon as it was but the targets were not this year. The targets were profitability in this window of time and margins going out of the door at a certain point. So we were looking for margins and a eventual profitability which are actually easy for teams, small teams to put together proof that they're headed in that direction. So with that just margins profitability window and the ability to attract people to get started there was enough controls to make sure that not too much stuff happened that wasn't really going to be successful which isn't to say there weren't unsuccessful products. But if you're not doing failed experiments you're not learning anything. So the environment which encourages that stuff to happen we see more and more of it in the software world these days is an environment that you can put in place with reasonable controls and the last thing you want to do is pour a whole bunch of money into experiments. You really want bounds because creativity and innovation always comes from constraints. It does not come from lots of money. Okay, one more question. There we go, last question. You're going to have to say this again. How will this idea be applicable for a small startup? Okay, for a small startup you are sort of in the innovation phase already. You're trying something brand new, yes? Black swan can affect their also, right? I didn't hear. The black swan problem. Well, I'm hoping that if you were in a small startup you are trying to be a black swan, right? Isn't that the game that small startups are trying to play? And so probably unless you're doing something really old and conventional you're not trapped by the focus thing.  And for that you should go look into all the literature on Lean Startup and that might help you. But I don't think right now you have to worry. But let's pretend you're smashingly successful. Come five years you better start worrying. Thanks everybody.