Thank you. I know you said that your talk was not going to be a lecture. Mine is definitely going to be a lecture. And you should definitely take notes. Today, I want to talk about our organization, the progress we're making, how happy I am to have as a part of the Linux Foundation. Our organization has brought together some of the most important shared technology investments in the world. And we are home to the largest technology investment in the world. It's a period, right? Open Source and Linux and the projects we are host to really do represent a multi-billion-dollar investment and are ubiquitous part of all computing. You're adding two members a month globally at the Linux Foundation. We're adding a new member every single day. We, every single day, have hundreds of thousands of developers working across the different open source projects that we host and have vast representation from thousands of companies representing hundreds of different industries around the world. Our membership represents is about a third here in Europe, a third in North America, and about a third in Asia. So we're pretty globally distributed in terms of representation. We'd like to see more representation from areas like Africa, the Middle East, South America. But I think the regions that we represent generally are where tech is being made. We're definitely more than Linux these days. GAB has been a real leader in terms of bringing financial services into the world of open source, both from a consumption and a contribution perspective. And I've spent about the last 20 years bringing many other vertical industries into the world of open source and really being able to realize in each of those industries the benefits of collaborative innovation. Examples include the film and entertainment sector. Our Academy Software Foundation is a partnership with the Academy of Motion Picture Arts and Science in LA, the home to the Oscars, of which someday I hope to be a seat filler at through our Academy Software Foundation effort. But this is something where we spent several years working with the CTOs and legal teams and engineering groups at Lucas Films, Disney, Pixar, et cetera to show them how they could collectively develop the computer graphic software, the CGI software, that creates the most beautiful films, the Star Wars and Marvels and all the different films that you know and love. And they ended up doing it. And the reason is they're in the storytelling industry. But the software development aspect of that community was insane. Every time you'd make a film, you'd have to build up all the technology. You would create the film. And then afterwards, you'd break it all down and you'd do the same thing somewhere else. And each studio was doing it differently. Now, because they do it all collaboratively, they have a standard set of development. They're able to really produce much better films more quickly. And that's what you're really seeing in the visual effects that are in so many of the brilliant films, Top Gun, I guess, being the most recent example of today. But we've also done it in cloud computing. We're home, as Gav mentioned, to CNCF. The CTO of CNCF, Chris Anicek, is in the back of the room here today. This is one of the largest and most fast-growing open source communities in the world. This fall, we'll be hosting 10,000 developers in Detroit at our cloud-native computing event, KubeCon. In the energy sector, we work with folks here in Europe. RT in France, the national grid operator, has a big problem in making the distribution of power more efficient. And as a national organization regulated by the government, they don't make money off of software. And so they came to us and said, we would like to give away the advanced software that we're building in order to create smart grid technology and collaborate on it with energy and utility companies in Germany and the United States and China. And that is exactly what we're doing. Over and over and over again, we're realizing the value of collaborative innovation. Each of our projects has a goal. And Finno's projects are no different than this, which is we want to find good project market fit. The technology isn't that interesting, or it's someone's PhD thesis on GitHub. That's not something we'd really care about. But if it's a global operating system or a core component of the financial services sector that really creates value, that's project market fit in our world. When you have that, people can code develop that software, and then they start using it in the products and services that they're offering to their customers. In the case of Kubernetes, all major cloud providers have a Kubernetes service offering. They make money off of that service offering. And as part of the profits that they're making from that service offering, they pay their engineers to fix the bugs that they discover in offering that service and improvements that they want in that software back into the project. Better project, better products, more profits. This is the positive feedback loop that we always focus on at the foundation. We have a detailed methodology about how to achieve this positive feedback loop, and it's what we're always focusing on. I think my slides continue to disappear with each click here. I'm not even, I'm just gonna remember what I was supposed to say on this slide. It was a last minute addition, and because we're here in the UK, I wanted to quote Dickens. And so with all the good stuff going on at the Linux Foundation and these critical open source projects that run almost all technology products and services in the world, we're really in sort of the best of times, but we're also in the worst of times. And I'll tell you what that's all about, and that's really the core of what I wanna talk about today. In terms of the best of times, we're really realizing the true value of open source. This is no longer a novel thing. Open source is not a bell that can be unwrung. Open source is now a fundamental part of how any technology, product, and service gets built. In 2022, you would have to be insane to build your own operating system, middleware stack, different components. It just, there are literally millions of reusable, very high quality building blocks out there that you can bring together, integrate, and build incredible technology products and services. The tech sector, now the financial service sector, most folks have figured this out quite a while ago, and that's why we have this incredible, two trillion package investment out there that people use to create this amazing technology. And so that is the best of it, but there also is in particular a big challenge that open source is facing right now, and I wanna spend the rest of my time talking about the challenge. I wanted to kind of butter you up with the good stuff that's going on in open source, and then now unfortunately talk about the bad stuff that's going on in open source. There have obviously been a huge increase in high profile security vulnerabilities in the global software supply chain, and in particular in some very high profile open source components. I know a lot of people mentioned that they had heard of Logforge vulnerability, but how many of you actually were involved in remediating it in your firm, like the working on Logforge? Okay, so a lot of you, yeah. Still doing it, I suspect. It's every audience I talk to who's in the tech sector is still working on remediating this horrific vulnerability that put all of us at risk. The amount of time, effort, money, and risk that has been injected is in the billions of dollars, from just one vulnerability. And we're seeing these vulnerabilities increase at a very high rate, and I'll tell you why that's happening. So while open source is the best of times right now, it's also the worst of times. I picture some popular node package maintainer and I'm like, can I have some more sort of asking for resources to help with security? And that's something that I think we all need to do. And the reason for that is because, as I stated, open source is a fundamental building block of technology. Next week I'm gonna be at the White House in the United States, meeting with the Commerce Secretary, talking about cybersecurity, the head of the national's cybersecurity directorate, Chris Inglis. And what I always talk to regulators about is, no, you can't just get rid of all the open source. Like that's never going to work. It would take 20, 30 years to rewrite all the software. So that belt cannot be on wrong. But now because of these incidents, we know how critical security is. To reel people's lives, energy grids get shut down.   So we understand how important this is. Adversaries, they're just getting better. And adversaries aren't just criminals, they are now nation states, as we've seen in the conflict in Ukraine. Sort of a low level cyber conflict that that has created. And then finally, regulators in this industry does not need to have this explained too much. The regulators are a common. So we need to either clean this up collectively or the government is going to start potentially doing it in a more ham-handed way. And that's something that I think we all need to consider and work on fixing. And so in order to do this, we have to answer a few simple questions. One, we have to decide what is the world's most important shared software? Package, version number, in rank order. Figure out if this particular component has a vulnerability, how many of us are gonna be impacted by it, right? Is it network facing? Is cryptography involved? Rank up, figure out what are these components by importance, figure out who writes this software, and then figure out whether or not it's secure. And not only figure out whether or not it's secure, continuously measure the security of this so that we all can continually understand these critical components are being maintained effectively or maybe they've been abandoned or maybe there's a certain number of vulnerabilities or maybe somebody new just showed up and started writing weird code, right? We should know this stuff. We don't know this stuff. It's embarrassing that we don't know this. The last thing we need to do is look at weak points in the global software supply chain and fix them in the way that code flows and I'll show you how we're gonna do that. The first thing we wanna ask ourselves is who writes this code? And this is interesting. I talked about the 760 odd thousand developers that work across the Linux Foundation every day but our research department did an interesting set of work in partnership with Harvard University to answer the first question I asked was what is the most important shared software in rank order in the world? And we did that and we came up with a list. What's interesting is the power law of open source. Of the top 49 of the top 50 software components, 23% of those projects are only maintained by one developer that writes about 80% of this code. So one thing we're just thinking about is just calling these individuals up. Can we offer you a secure coding course? How about a little bit of testing help? How about additional resources and so forth? And I'm gonna show you how we are literally doing exactly that through our open source security foundation effort at the Linux Foundation. Second, it's important to understand, this is the call to action from the folks in the financial services sector is that as much as you all think money solves all problems and I like to think it too, that is not what solves problems when it comes to developers. We went and surveyed like why do you work on open source and money was in most cases the last reason that these maintainers of these critical open source projects work in it. They use open source because it gets their job done more easily because they wanna be a part of something bigger than themselves. They wanna be a part of a shared mission. They wanna show their technical prowess and advance their career. Money was the last thing on their mind. The money's not gonna solve everything. Lastly, when we talked to these developers, one of the things that we realized is that they need better training. Literally, there is not a single computer science department in the United States that requires a secure coding class for graduation from a computer science degree, which is totally insane, which is in fact the topic of the meeting that I'm having at the White House next week with Secretary Ramondo. We've got to do better than this. And so the Linux Foundation has created a plan. I'm gonna go over that quickly with you because now I have 10 seconds left and as I said, it's a lecture, it's going over, take notes. I'll go fast. To solve this problem, what's important to understand is how code flows. Now I don't need to tell most of the folks in this room because you're all technology professionals how this works, but it's important to sort of simplify and understand how we attack this problem by looking at this. Code starts with the developer in their mind, in a repository, often on GitHub, GitLab, some shared repository, where they're writing the actual software, right? And in that part, there's all sorts of ways to mess with the code. You often don't have code review in a lot of open source projects, or people can bypass it, or you can compromise the actual source control system itself, right? After that, code keeps flowing and now we start to get into the whole remix aspect, which in particular in the last five, six years has become this fundamental way to create software. You're using node components from NPM, or Ruby gems, or you're using Java components from Maven Central or so forth. In each of these package managers, there is many opportunities for bad actors to squat on particular packages to sort of usurp these packages. We hear constantly about people coming in and compromising various node packages and then compromising all of us, right? Through these dependencies and bad packages and so forth. And so what we're identifying at the foundation is ways to address each of these things. And while I wish that the type was on this particular slide, I'm not going to remember every single component of the effort that we're doing here, but I think you get the gist. Our open source security foundation has efforts to look at how do we do best practices in writing code? How do we work on vulnerability disclosure? Does an open source project have a responsible disclosure policy period? Do they have a security mailing list? Do they have testing and tools that they can use in order to write their code, to fuzz their code in order to lint their code more effectively and so on and so forth? And so we've totally lost all of the text for my presentation here. What you should be seeing here for the first time ever is a comprehensive plan that we have developed. Can we get a little help in the back there because this is really the meat of the presentation? But all kidding aside, about a year ago, the open source security foundation kicked off with about a $15 million investment. So just before Log Forge, I called up the CIOs and CTOs of Amazon and Google and Facebook and Microsoft and many of you and said the same exact thing, sort of gave the same pitch and said, listen, we gotta get ahead of this. And we were able to get the open source security foundation together with sort of a CEO minus one executive commitment on the board to go work on these problems. And an initial seed fund of about $15 million. Then of course, Log Forge happened, I looked like a genius and we were sort of ready for this. And for the first six months of this year, that organization has developed a 10-point plan, which I hoped to show you today, but now I'm gonna have to describe it by memory, the first really truly comprehensive approach to how do we go answer these questions and solve these supply chain issues in a scalable way across the entire industry and across the globe. We presented that plan in May at the White House. We had the participation from a huge swath of technology industry participants and we raised an additional $30 million in resources toward that plan. And the plan really involves the stuff that I'm really hoping we can get up on the screen here any moment, but it involves a few things. One, training. We really need to create secure coding courses. I would argue that the Log Forge vulnerability would not have happened, and I hope the maintainers of that project are not listening to this because they'll be all pissed off at me, but the vulnerability, the nature of the vulnerability for anyone who had taken a secure coding class would have been quite obvious. And I think that that happens over and over and over again because a lot of open source developers, well very talented computer scientists and developers, just don't think with security in mind. I'll give you an example. Remember back in 2003 when Microsoft was having a lot of security problems? Bill Gates writes a letter to the whole company with his famous letters, and he says we're gonna stop all production at Microsoft. Everybody has to take a secure coding course, we're gonna introduce better testing, we're gonna have a trustworthy computing initiative, we're really gonna take security seriously. And the implication is stop all coding, do this, or you're fired, right? Because all these people work for Microsoft. But we can't go fire a million developers globally who work on these critical open source projects. So we have to provide them free resources and training, there you go, in order to allow them to know how to write code in a way that won't put us all at risk. This is securing open source production. We need to just figure out ways to then, once folks are writing more secure code, look at that code, trust and verify, right? Do better ways of testing, presenting analytics about the results of that testing and so that all of you as consumers of that open source project can understand the health of each of those components. If you go to the Linux Foundation's website today and look at our LFX security tool, this is the start of an open source reputation platform that literally measures based on scanning of code with commercial code scanning tools based on testing each of the projects on whether or not they've taken secure coding classes, whether they have security policies, whether they have security mailings and so forth, to give sort of a risk score for each of those projects. Go check this out. This is something we're investing in in order to make all of us safer. We're looking at doing just dumb, simple things like how about using package signing with cryptography? Wouldn't that be actually important? NPM finally got this done just recently. Just recently, they're using two-factor authentication for critical open source component maintainers so as to prevent that attack vector and so forth. Now, once we establish that baseline, we really wanna work on vulnerability disclosure and remediation. One of the things that we're working on right now is a project called Alpha Omega. We're taking the 10,000 most critical components and applying that reputation framework and a set of scanning and sort of automated testing components for security vulnerabilities and we're taking the top 200 most critical open source projects and we are literally doing an audit, we're using third-party audit companies to audit every single one of those projects in detail. And has anybody here been through a security audit of code where you actually have third parties read the actual code itself? Every time we run one of these, the Cloud Native Computing Foundation does this all the time with Kubernetes, we find zero days every single time. This is something that's critical to all of us and doesn't even cost that much money relative to the cost of a log torch. The last thing we're doing and I'm kinda skimming through this and there's a white paper up on the Open Source Security Foundation's website that presents this in great detail is we're looking at creating more liquidity in the world of software bill of materials metadata. So one of the big projects that we're investing in is automating the ingress and egress of software package data from repo to package manager to consumer so that all of you have a much easier and commoditized programmatic way of knowing whether or not you're vulnerable to log4j or whatever the latest vulnerability will be. Right now, just figuring out what you're using is so bad across so many sectors that we need to improve that. Software bill of materials metadata that's automated and easily distributed and is very liquid across the industry will be a force multiplier in enabling the faster mediation of things like log4j. Here's the punchline and I know I'm way over but it's really the people's fault in the back because I had to riff for so long while I was waiting for those slides. The punchline here is this is just a rough estimate of the cost and we're already well on our way as I mentioned to raising the money to do this but we're talking like $150, $160 million over a couple of years to seriously have an order of magnitude impact on our collective cybersecurity. This is a complete no-brainer. Like it costs a lot more than $150 million to remediate the log4j vulnerability, just one. I think some of the Equifax fines, I think for just one vulnerability, the struts vulnerability, just the fine to Equifax, I think was more than $150 million. One company and the fine itself wasn't even remediating it. This is something that I do want to encourage all of you to look into and participate in and it ain't just about money. If you have expertise at your firm that you can lend to the open source security foundation that can help with auditing, creating that reputation framework, helping with the automation of software build materials metadata across the supply chain, help with building a package signing across all the package managers that are out there in open source. We need both money and expertise. This is a totally doable thing. It is not a moonshot, it is a mobilization plan. There's nothing particularly innovative that I've discussed today. It's just a lot of freaking hard work that will make us all more secure. And so with that, I really thank you for listening to me today. I encourage you to do this. As bad as the security problems are these days, the value of open source is so much larger. Let's spend the next couple of years fixing this one problem that we have and I think all of us will collectively benefit. Thank you very much. Thank you.


Live from Silicon Valley, it's theCUBE. Covering Mobile World Congress 2017, brought to you by Intel. Okay, welcome back everyone. We're here live in Palo Alto for theCUBE's special broadcast presentation and coverage of Mobile World Congress, which is happening in Barcelona, Spain. I'm John Furrier here with SiliconANGLE Media is theCUBE. And of course, we're covering it here in Palo Alto, bringing in experts and friends who are following all the action, as well as have commentary and opinion on what's happening. We're going to roll up the news. It's the end of the day in Barcelona. We're just getting our sea legs here for day two of 8 a.m. to 6 p.m. coverage inside theCUBE. And of course, we want to break down the counter. Our next guest is Val Berticovici, who is the CTO at SolidFire. Also a governing board member of the CNCA, the Cloud Native Compute Foundation, which was KubeCon, which is now part of the Linux Foundation, which if you know theCUBE, you know we've been covering that like a blanket. All this shows theCUBE has been there. This is in the world of DockerCon, et cetera, et cetera. Val, a CTO, 19 year veteran at NetApp, of course, knows the storage business, knows the converging infrastructure, knows the cloud, part of the original cloud already. Val, great to see you. Thanks for coming in. Thanks for having me back on after all these years. Yeah, I mean, it's great to have you on. One, you know, we've, you know, we see each other at some of the parties and Georgiana's place in particular. Georgiana brings all the storage and cloud already together. But you and I have had conversations a few years ago about, you know, where cloud was going and you almost kind of connect the dots, not to pat ourselves in the back, but I think we were right that the cloud was what we thought it would be and probably more. I mean, for me, I think I underestimated certainly the Amazon impact. Me too. But you look at what's happening in Mobile World Congress, you have a bipolar show. You have a device show, ra ra, look at the fancy devices. And the other show, you have a telco show that's trying to figure out their future. And that's interesting because the telco's power, the big networks that everyone is using the devices for. So you have a consumer market, but the real conversation is 5G, IOT, you have a collision course of enterprise issues, enterprise data center, enterprise technology, colliding with a telco infrastructure, AKA mobile, head on. So it's not just more wireless, 5G, certainly the story we talked to folks like Intel and others around that, but you have essentially all these core problems that are gonna scale up this next generation use cases are enterprise like. This is your wheelhouse. So are you like looking at this saying, I've seen this movie before, what's your thoughts? Actually, I haven't, that's so exciting, right? There's, as you said, there's so much innovation happening for me. Probably the big story is what's not in the headlines at Mobile World Congress, which is the back office work to support a 5G rollout. And I've had a lot of experience, particularly on a solid fireside most recently, speaking with all the big telcos globally, and their NFV implementations. And what's interesting is they're all going through a gen two or a rearchitecture right now. A lot of first generation NFV was done based on traditional or legacy now cloud architecture, which is very VM based. And all of them are now architecting and implementing microservices based implementations. And a driver for that is just the explosion that 5G will enable in terms of connectivity between devices. So the least interesting stat to me is how fast I can download a movie off of 5G. The most interesting is how many hundreds of thousands or millions of devices within my domain are going to be communicating with each other on 5G. We had Sark Ghalai, who's going to come back today. He's also a guest analyst, former HPE, NHP Enterprise, he ran there, built their communications group all the NFV. I like it, and he was commenting the same thing, and he made a point I want to bring up, which is I don't really need more, a gigabit right now, I want more battery life. So he's kind of being pedestrian in his use case, but that really is kind of the consumer issue. You're pointing about things that are going to be harder to do. And NFV, you mentioned as one of them, can you explain the NFV current situation? Because also we've been doing a lot of the OpenStack, all the OpenStack shows since it started, that has become kind of a telco NFV storage show as well. Absolutely. So what is the real issue with NFV, and why is it important and relevant to the service providers right now? So if you take a look at all the services we depend on our phones nowadays, there's obviously the basic connectivity, there's additional services around location mapping for GPS, the layered service is not top of that in terms of the collaborative apps that we use and depend on every day, sometimes on S3, which is not always available as we're recording right now. There's a lot of layers there, and from an NFV perspective, from a backend data center perspective, everything amounts to a session. So even though it's packet switched, it's still a logical session you have to set up. So for every session, and imagine this happening millions of times at every tower and more than millions of times at every regional or central data center, you've got to have a session set up where you've got to authenticate, actually, who the user or the device is, make sure they have permission to be on the network and accessing certain things, you've got to authorize them to do certain things, you've got to log what's happening, then you've got to slap some firewall or security around them, then you've got to layer in and access to all the other resources, they're trying to combine into a service back to the end user. There's a lot of things going on. You have to set up these sessions for every connection. And if you try to set up a VM for every connection, you'd have to fund a multi-billion dollar data center, Google can't even afford. So this is where microservices are becoming essential right now in a 5G hyper-connected world, is where you have to have much more efficiency in the speed by which you set up these sessions, the efficiency of number of sessions per server, and just the end-to-end cost of processing all these transactions. This is interesting, and I want to just kind of translate a little bit out for the folks that aren't CTOs out there, essentially, you think about mobile. We've all been, you know, since the iPhone in 2007, we've seen this just accelerate with data and whatnot.  That is essentially the base station saying I can't get a session. Now, as a user, you have a phone, so you've been provisioned by the telco.  That's a known problem. Now, when you think about IoT, Internet of Things, and now people, your watches, your wearables, sensors on airplanes, and industrial equipment, to traffic lights, those are devices that are going to be provisioned and turned off and on, so it's like a new phone every time, so you've got the complication of not knowing the devices that are coming on, and then trying to establish the connections, and figuring all this out, this is kind of a really hard problem. It is. At scale, it's really, really hard. This is a really hard problem at scale, at many levels. So to me, what we're hearing at Mobile World Congress is, you need a dynamic network. Absolutely. What are some of the tech involved? What's the real enables? You mentioned microservices, we know about containers, Linux foundations opening up their kernel for a lot of variety of new kind of configurations. You got solid state memory, and you got new memory architectures. What are some of the key things from a technical perspective that are going to change that complexity to be seamless for users? Probably the most fascinating trend to me, and we're just beginning to see some stories and a merger on this is the rise of edge computing. So I kind of hark back to where I started my career and dating myself now, but the client server era that succeeded from mainframes, we've seen a huge pendulum shift towards cloud computing and centralizing a lot of processing. Well, back to 5G, back to millions, if not billions of connected devices right now, there is no way Einstein kind of introduces problem for a speed of light. There isn't a way to process the exponential amount of data we're dealing with right now at the core, and still provide useful feedback at the edge. So the rise of edge computing as a bit of a counterbalance to cloud computing and having more powerful, more intelligent processing at the edge, filtering a lot of data because we can't possibly store the exabytes and the other bytes of data. This is a paradigm shift.  So what you're bringing up is an interesting architecture that is an enterprise data center thing. We were talking yesterday and then I was talking to the Intel folks, I pressed them on this because they're obviously in the data center business, that a car that's fully instrumented like a Tesla or a future autonomous car is essentially a moving data center. So it just changes the notion of data. Yes. This is a paradigm shift. You agree with that? Moreover, IOT is maybe the first technology buzzword that takes a lot of this digital world that we've been talking about. It's really been largely abstract and virtual for the common person, and it makes it physical and real. So the impact of IOT is an actuator changing your traffic light.  That's actually impacting your physical world. It's no longer just a virtual thing. So that's where IOT is going to become really, really significant in our lives. And the software program that needs to be created, this is an opportunity for entrepreneurs. Certainly, you know, Peter Barris and I were talking yesterday morning about the edge computing and he's got a big slew of research on this. And he talks about IOT and P, IOT and things and people. And we were also talking yesterday about the relationship of the people to technology. So for instance, in telcos, they view the phone as the relationship that's coupled to the carrier. And that, you know, the premise we put out yesterday was that that's going to be an uncoupling. It's going to be a person's relationship to multiple carriers, if you will. So the question is, how does business extract the value out of this? And this is something that, again, Peter Barris and Ricky Wands are digging into, which is the business value of technology. In the paradigm we just talked about, which seems pretty obvious how we get there, not so obvious, but people are working on that. How should companies think about getting value out of this massive shift? A lot of moving parts. The examples are already there, you know. Let's talk about one of the most talked about companies around here, Uber, right? And not for some of the headline reasons they're attracting right now, but some of the classical disruption they enabled. When they take a look at the fundamentals of what they do technically, it's interesting. It's somewhat impressive, but it's not revolutionary. What made them revolutionary is digitizing the transportation relationship you and I have with our transportation providers. And when they tapped into that, they realized the potential of that is limitless right now, because we're all physical beings. We still have to move ourselves or our food or our packages around. But digitizing transportation is really, you know, a great example of any industry, whether it's a 100-year-old industry or brand new industry, when you digitize a distribution and when you actually add digital efficiencies on the back end, you end up with that 100x effect, more than a 10x effect, and truly earn the term disruptor. Give us some more examples that you've seen. I know you mentioned you talked a lot of telcos, but what other use cases? Because the whole notion of 5G and this new architecture is really coming down to use cases. And certainly there's the sexy ones. The car, the smart cities. I mean, there's a lot of policy and societal impact issues that need to be thought through, but just generally, what's the low-hanging fruit right now? You know, instead of low-hanging fruit, I might be giving you the most pedestrian example I could think of, which is when I meet with some of the waste management companies, years ago, I took them for granted. There's no innovation here. It's going to be an old enterprise discussion with the conservative tech leaders. I'm not even going to try and phone this in. I'd show up and meet with them, and they were truly innovating because they realized the whole customer experience of putting out your trash bin, your recycle bin, your organic bin, and so forth, your compost bin can actually be improved and efficiencies added when you put GPS trackers on all the trucks. When you figure out when they have to go to the dump because there's like an exponentially high amount or an inordinately high amount of garbage put out early in the route one morning. And the ability for you to know when your bins are picked up so you can actually go and pick up those bins, put them out minutes before, instead of the night before, bring them back minutes after, just reinventing that very pedestrian-made experience tells me there's opportunities for innovation everywhere in our lives. So really, let's just pick a spot to make efficient. It's probably the easiest, kind of laying back. Great, great feedback, Val. Thoughts on developers, because it's something that we didn't get time, and we'd love to have you bring you back for more time on this. But the CNC, yeah, the CloudNative Compute Foundation is, it kind of came out of the Kubernetes con, which is the Linux, now part of the Linux Foundation. You mentioned microservices, Kubernetes orchestration, there's a lot of software around composability, whether that's an artisan-like developer, or the hardcore developers down low on the stack. How does autonomous vehicles, and this new future use case, whether it's programming drones, or writing cool software, to what's going on in the developer community? Can you share any color on trends around what being done in traditional, classic developers? Autonomous vehicles are really a perfect example because CNC, fundamentally, is about what we call CloudNative technologies and applications, and a typical CloudNative architecture is container package, not VM package, and it's dynamically orchestrated. So we say it's basically declarative, as opposed to opinionated, to use back-end speak, technical developers speak. But what does that mean? Autonomous cars are not interesting if one car is autonomous. Autonomous cars are interesting when dozens, hundreds, perhaps thousands of cars along your route, if those are autonomous, the interaction between them, and making sure they don't all try, and occupy the exact same space at the same time. That's pretty essential. But also keeping traffic flowing smoothly and preventing unnecessary traffic jams, since the coordination of multiple processes, multiple things, that's what really makes CloudNative computing happen at scale. So CNCF actually is five projects right now. Most people know it as the Kubernetes project, and it is very much that. I think a year ago no one knew how hot Kubernetes would be, but certainly it's taken off right now. Thanks a lot, and part to Google. We've got new technologies around monitoring. So you've got to monitor your app, obviously, in the Cloud world to see where the efficiencies and the performances for the end user are on logging, distributed logging, and monitoring. We've got new projects around actually debugging at scale. So debugging one process, as simple as any developer knows, debugging multiple concurrent threads or processes, that's still a black art. And so we've got open tracing technologies around that, and there's a new style of project. It's actually something known as Linkerdee, but it's around now in the IoT context, the most important thing to be a really valuable IoT device is discover your context. What other devices, what other sensors are out there? It sounds like an operating system to me.  This is a global operating environment. It's a great, I think, insight. It's not, I don't like operating systems. Not like a classic sense, but there's some systems. It is a new operating environment. I think in the CloudNative world, operating systems still kind of takes one PC or one device or one host on a data center. It really is a coordination of services, new modern high-end services. And declarative things with containers that's essentially assembly-based. You can manage things component-wise, those kinds of concepts. Absolutely. And you see that as a key part of making the CloudNative, as a key part of enabling these new use cases. There's actually no economies of scale if you don't go CloudNative nowadays, as 5G networks become more prevalent, as IoT becomes our mainstream, it doesn't play without microservice architectures. And the trade-off for not being CloudNative is what? Being disrupted. It's literally, you know, there's some great recent blogs. You've heard this title before, the coming SaaSpocalypse. So the disruption of the legacy SaaS vendors. The economics of them forced you to basically have fixed subscription models with your customers.  These new CloudNative architectures are going to enable disruption in the industry because they'll only consume resources as a sessions for an app, you know, as common SaaS app are used. And the licensing and business model can now be that much more efficient, 10X more efficient for people that actually want to get charged per use as opposed to per subscription. Val, you know, I've gotten to know you over the years and a great guest to have on theCUBE. Thanks for sharing that insight. But it is a robust, it's exciting time. You had a great run at NetApp. I mean, you look at what NetApp's been doing. I mean, they were the darling of Silicon Valley classic success story. Multiple reinventions, great founding team, great investors, just the classic run that they've had. As you look back now going forward, looking back and now looking forward, what are some of the things that gets you jazzed up right now in terms of things that are, that are that next wave that that's coming? What do you see that's exciting and what would you share for folks for insight? I'd say the most exciting thing to me at a high level is just the opportunity that, you know, 5G IoT enables. I think there's a whole new market segment. Some people might call it the real evolution of HCI, which is edge computing and all these really fascinating new workloads. They're not gonna be necessarily virtual desktops in terms of running or operating your business, but entirely new revenue streams, entirely new services that all sorts of companies, digital or analog can offer. That excites me. And of course we've talked about the backend rise of the shift towards, away from fast storage, fast storage towards persistent memory, that in itself is gonna open up a whole new category of apps that we've yet to see. Yeah, and we gotta get that Linux kernel rewritten and open up all kinds of new stuff. Great, great commentary, Val. Thanks for coming on, Chief Chairman. We certainly want to have you back and really unpack and drill down and double down on what cloud native impact means. And certainly edge and IoT computing really is going to be a fascinating run. I think that's going to open up a huge can of worms and an opportunity for really changing the game and creating great value and risk too. I mean, Amazon S3 is down as we speak. We're joking, but you know, we see insecurity problems out there and we'll stay on top of it. Of course, theCUBE has got you covered and that's the hot themes. Really that's not being reported at Mobile World Congress that we're reporting, which is the surge in IoT relevance. Obviously AI is super hype but that gives a mental model. This is the story of Mobile World Congress, a 5G as a fabric connecting in with hard enterprise data center like technologies end to end for dynamic experiences. This is the challenge for telcos and someone will get it done. Let's see who it will be. Of course we'll be watching it. This is theCUBE with more coverage of Mobile World Congress after the short break.


We're super excited to be here. Now before we get started allow us to introduce ourselves So my name is Adriana Villela and I am a CNCF ambassador I'm a HashiCorp ambassador blogger podcaster by day I work at service now cloud observability the artist formerly known as light step And I spend most of my time doing open telemetry work Reese and I actually work together in the open telemetry and user working group soon to be sick super excited And by night I like to climb walls and completely unrelated. I'm a huge fan of capybara's Hi everyone, my name is Reese Lee. I'm a senior developer relations engineer at New Relic as Adriana mentioned we work together on the open telemetry and user working group Where we're focused on connecting end users to each other through enablement content as well as events and facilitating a feedback loop with project maintainers and end users to help improve the project as well as drive adoption I guess my fun fact is I like anything spooky and paranormal So credit for the idea for this talk actually goes to our lovely MC Austin Parker They had written up a today. I learned type post where they discussed a conversation that had had about Who's someone about how open telemetry deals with error? recording which made us wonder oh Dude, where's my error? Where's my error dude? so how does open telemetry handle errors and What options do you have for recording errors using open telemetry? This is what we're going to answer for you today in our session We're going to first set the stage of some background then we're going to get into how errors are represented and handled an open telemetry We'll then give you a demo on How the same open telemetry instrumented error is represented in a few different backends? And we'll talk about why that matters for you as an end user and Finally, we will do a quick wrap-up So Adriana is going to set the stage for us shall yes before we talk about all this lovely stuff Let's do a little bit of background So here's the deal So happens that different languages approach errors and exceptions in different ways So for example go doesn't really have the concept of exceptions But then languages like Java and Python have things like mechanisms for catching and throwing exceptions Which is awesome But what happens when you're in a situation where you have an app that's made up of multiple microservices? Written in different languages and we're trying to do the observability thing Well, we need a you know a standardized way for capturing our telemetry And that includes a standardized way for capturing our errors So what do we do in that case when we can't even agree on you know How we're handling errors in different languages and of course The answer is open telemetry to the rescue our good friend and Which leads me to the open telemetry refresher, which I'm sure everyone here is familiar with it So this will be a very quick refresher So open telemetry cncf project open source vendor neutral Allows us to generate ingest process and export telemetry data to one or more observability backends for analysis and interpretation awesome so Back to errors Well, we talked about errors. We talked about exceptions But what is the difference between the two now if there are different definitions of errors and exceptions? And we can probably argue all day about this But we've come across definitions for errors and exceptions that we particularly like so these will be The basis for our talk here note that this is these are definitions for errors and exceptions within the context of technology purely and not like But not not in not necessarily in the context of open telemetry So what is an error an error is an unexpected issue in a program that hinders its Execution so an error can be something like a compile time error. You forgot parentheses on an if statement Maybe you forgot curly braces. Maybe you forgot a semicolon or It can be something like a logic error The program is executing sure but it's not doing what you're expecting it to do Then we have exceptions an exception is a type of runtime error That disrupts the normal flow of a program and that can be something like dividing by zero or Something like referencing a memory address that doesn't exist shout out to you know old days of doing programming in C Where that sort of things seem to happen all the time? So there you have it Thank you. Okay, so now that we have set the definitions. Let's get into the main course error handling and open telemetry So as I didn't just mentioned languages have their own ideas about what constitutes errors and exceptions as well as how to handle them So how exactly does open telemetry deal with all these conceptual differences from language to language? This is where the hotel specification or spec for short comes in The spec provides a blueprint for developers who are working on various parts of the project it standardizes implementation across the different languages and Since language apis And SDKs are implementations of the spec there are general rules against implementing something that isn't covered in the spec This is to help provide a guiding principle for organizing contributions to the project in Practice though there are a few exceptions. So for example a language my prototype a new feature As part of adding it to the spec usually designated as Alpha experimental before the verbiage is added to the spec As another example the spec allows for some degree of flexibility for a language to implement something as naturally To that language as possible. So for example, most languages have implemented record exception And since go does not have the conventional concept of Exceptions it has instead Implemented a record error Which isn't he does the same thing and we'll talk a little bit more about that in a little bit So now that we have a unified framework for how to handle errors Let's see what options open telemetry provides for us So first of all we can record errors using either spans or logs In open telemetry a span represents individual Represents an individual unit of work within a distributed system for example an HD call or database call and They can they then make up the building blocks of a distributed trace Spans are related to each other and to a trace do something called context Context is the glue that holds at sorry that turns a pack of data into a unified trace And we like to think hazel weekly for that awesome definition Context propagation allows us to pass information across multiple after systems and tie them together You can learn all sorts of things through traces of our systems such as how our services are connected and talking to each other As well as what occurred in our application at a given time Open telemetry enhances spans in several ways One of the ways with which you can enhance our spans is through metadata or attributes in the form of key value pairs So when you attach relevant information such as user IDs or cross parameters you can gain deeper insights into what occurred within a given trace Spans also have a field of span kind Which is additional information that can provide Developers with further context for troubleshooting Span kind is determined automatically by the instrumentation libraries that you use Open so I'm to define several span kinds each of which has unique implications for error porting We have clients which is for outgoing synchronous remote calls such as outgoing HTTP calls server which represents incoming synchronous remote calls and then internal spans represent operations that don't cross process boundaries and Finally, we have producer and consumer which are typically used for message queue operations Spans can be further enhanced with something called a span status as well as a description of that status For example, here we have an acceptance message message that was captured along with a status By default a span status is marked as unset unless otherwise specified you could also set it to error or as okay, and finally we can enhance spans with something called Span events a span event is a structured log message embedded within the span on which it occurred Span events can help enhance spans by providing additional descriptive information and you can also capture additional information on the span event by using custom attributes When a span status is set to error a span event is created automatically that captures a span error message and stack trace Earlier we mentioned a method called record exception Since go doesn't support the conventional concept of Exceptions it is implemented instead as a court error However, with both of these methods you have to make an additional call to set the span status to error And that's what it should be because it's not out of my automatically going to be set to that So this means that you can actually use span events to record additional information on that span event So by decoupling the span status from being automatically set to error when a span exception occurs you can support the use case where you have an exception of That has a span status of okay or unset and this allows the most Flexibility for instrumentation authors as well as end users We can also record errors using logs with open telemetry In open telemetry a log is a structured message emitted by a service or some other component They include a message a time stamp as well as a severity level and Severe two levels Represent the type of message that's being emitted so debug info warning error or critical Open telemetry allows for the correlation of logs to traces In which a message a log message can be associated to a span within a trace Vias trace context correlation which we talked about earlier So if you spot a log with a log level of error or critical You can navigate to the corresponding trace to find out more information about what happened And if your back-end UI allows it you can also navigate to the log from the trace UI So is there is one better than the other for recording errors? Spans or logs the answer is everyone's favorite. It depends Perhaps your team primarily uses logs perhaps a primarily use traces Another thing to consider is you the back-end that you're using does it render both logs and traces does it support trace and log correlation and How easily curable or discoverable are your is your data? So if you've been using a proprietary agent to monitor your applications and have migrated open over to open telemetry You might have noticed that An error that was captured by open telemetry instrumentation is Expressed a little bit differently than the same error captured by the proprietary agents instrumentation This is mainly due to the fact that open telemetry simply bottles errors differently than how vendors might have been doing For example vendors have their own notion of what represents a logical unit of work within an application or system You're probably familiar with the term transaction which can mean something slightly different from vendor to vendor and in open telemetry This is represented by a trace which is made up of spans so already Vendors have had to adjust How this data is populated in their UI because it is a different data model and Finally, we have span kind which impacts which has a bill Possibly to impact your error reporting Some backends might have opinions on whether a Sorry might have opinions on like only server and consumer span kinds should be counted to an error rate and not internal errors for for example And in a second here Adriana is actually gonna Take us through a little demo and that's we're gonna where we're gonna demonstrate kind of like some of these differences you can see from vendor to vendor In Yeager, oh another thing to consider to you is some backends might have created a different Signal type for span events in Yeager. They're represented as logs because that's Basically what they are so we'll show you an example of this in just a little bit Demo time So we are going to do a demo. It is not a live demo because in my experience any time there are live demos bad things happen So it's pre-recorded demo You still get to hear my lovely voice as I live explain the pre-recorded demo. Okay So we have a demo application written in Python There is a client and there is a server. It's a simple application The client makes a request to a roll dice endpoint on the server. It's a flask application The server rolls a virtual die i.e. It outputs a number between one and six and outputs it to standard out Now as I mentioned, there is a client and a server portion to this application However, the client's not so interesting. We'll be looking at the server code and here are the notable parts so here we have the roll dice endpoint and This part here basically calls another function called do roll which actually does the work and Then we have this section at the bottom which does the initialization for our application It initializes our flask application. It initializes our traces and our metrics now This application has been instrumented with open telemetry using auto and manual instrumentation We are adding some logs for the purposes of demoing all this good stuff and for fun Z's We're also capturing a metric Now inside the do roll function. We are creating a span called do roll You can call it Bob for all I care does not matter, but it should be a name. That's meaningful to you Now in our span, we are doing a few things. We are creating an attribute We are also creating a span event and that span event has attributes of its own We are also creating a log message. It's an info message We are creating logs using the Python logs bridge API aided by Python auto instrumentation Which means it does magical things because we are embedding our log within our span definition it means that the log gets Automagically correlated to the span and then finally we have the our one metric. It is a counter Instrument and this counter gets incremented by one every single time this function is called Now this is an error's talk. So we should be throwing an error at some point I have forced an error basically every time a die roll is divisible by two It will throw an exception and this exception is caught in the rule dice function Now within the rule dice function If when if the exception is caught then we create a brand new trace and this trace We're saying, okay, we're capturing the exception By using record exception now record exception is basically a span event But it embeds the stack trace as part of it and also just for fun Z's I'm also creating a log message here and it is an error. The log level is error So here is the video of the demo now this repo is publicly available and we'll include a link to the repo at the end of this talk This is available on github. I've made it available through github code spaces So you don't have to set up a local environment test this thing out So hopefully it'll set save you some headaches if you want to play around. So here we go So first we are opening things up in github code spaces and it takes a couple minutes to start up I sped things up so you don't get bored. The other thing is The hotel collector config for sending stuff to the observability back ends I put it in this hotel call config extras file, which I've get ignored because it's got some Some keys to access the observability back ends, which I do not want in github So that's been added separately next. We are building the Docker images for the Python client and server We'll be running this thing using Docker compose in a minute once the images are built a little hamster is going And It's almost done. Okay. Now it's done. We are going to run Docker compose So this thing is going to start our Python client server the hotel collector and Yeager and we can see that it's running here Now Yeager is automatically Exposed through port one six six eight six in github code spaces And if we go over to the ports tab and click on the little globe thingy Then we can open up Yeager in github code spaces and we can take a look at our traces And as you can see we have some traces that contain errors represented by the big red dots and the ones that do not contain errors which are represented by the little green dots So if we go to a non error one, we can see our trace with four spans and we've got our do roll span and it's got our span event and The with the message and the two attributes then going to our error trace You can see that our span we have the one span with the error. It's marked in red And it's got our usual span event, but also because it threw an error It's also capturing the stack trace as a new span event automatically captured And because it's throwing an exception that extra span that we created shows up there as well And if we click on that we should be able to see our span event that we created That captures the stack trace as well. Obviously, this is overkill We don't need both of them, but it was just for the purposes of demonstrating this now We're opening up a different observability back end and as you can see Similar interpretation looks slightly different the red triangles represent the errors that the traces with the error spans and The green dots represent the traces with the non error spans. So similar thing we can see that All is good We see our span event with our message and our attributes But we can also see our log message that we added in because this back end supports supports logs and That the log is actually correlated to this particular span and if we click on the log We can see the log message and we can see it also in within the context of other log messages that belong to the same trace So we have that correlation now we look at and at one of the error spans And clicking on that we can see the do roll incurred an error so it's got it showing up as red we have our Additional span event with the stack trace which we expected and again We should see our same log message and because it was an info message. We don't expect it to look You know scary and red But we can see again There's the correlation to the the overall trace and we also see our even number span That was created because we incurred that error which has our exception message and also our log message Which we didn't see in Yeager, right because Yeager doesn't support logs So if we click on this log, then we will be able to see the log message in a minute Which because it was an error it shows up in red and again We can see it in the context of other log messages that were part of the same trace So now Reese is going to show you what that looks like in a different and in yet another Observability back end just to give you an idea of You know, it's kind of interesting to see how how different products Interpret things in different ways Yeah And I think it's really valuable for end users to kind of understand and get a look at how they are represented differently So this is an example of how traces might be visualized So here we've got a group of traces which we can see Easily which one of them had errors and from there we can click into one of the traces with an error And look for the span that has the error and From there we can see metadata about the span in this case The span status code the status code description as well as the fact that there was a span event captured when we click into the span event Which you can see here is referenced as a Span events versus a log as it was in Yeager. Sorry, I clicked too fast and I don't know how to go back But also we're running out of time so I gotta go This back end has created a new signal type for span events called span events and So in Yeager it was represented as a log here is represented as a span event and you can also navigate to the Associated logs and vice versa because this back end supports Trace and logs correlation yep, and There you have it. I believe we have one minute to wrap up so error handling as challenge open salametry provides a an open standard blueprint for how to handle these errors as well as providing different ways for us to record errors do spans logs and it supports correlation you can enhance spans with Metadata a span status span kind and a couple of different ways that we covered and How your data is visualized in your back end? Maybe a little bit different than how you may have been used to you by using a proprietary agent instrumentation simply because open salametry Models errors differently than how vendors have been doing Not all images are created by humans Dolly, but honestly Adriana is an amazing prompt engineer and did all the penguin images. It was fun And we have some handy QR codes for you to check out Yeah, I have a podcast called geeking out and I've had guests like Kelsey high tower and charity majors and Reese So you should totally check it out Also come see us in the hotel observatory at cube con starting tomorrow We've got some really cool stuff going on Including end user feedback sessions So please if you have any feedback on open telemetry sign up for one of the feedback sessions We would love to hear from you And come join us on Thursday for a party that we're hosting with our friends Docker plumey and tail scale and We also have some handy licks here to reference from the slide from a talk today. Thank you so much