Welcome everybody for the first day of the L15X and today we're going to be taking off our embedded track with Joel Fernandez from the software. Hi, so thanks for the nice introduction. So today we're going to talk about BVFD, which is a project I started to make it easier to run BCC, which is a powerful tracing tool set on an embedded remote target. So I'm going to start off with a demo and then I'm going to build up towards BCC talking about Linux tracing in general and then how that builds up to BCC and what BVFD is and hopefully the demos will make it clear how I'm using this stuff. So a little bit about me first. I work in the Android kernel team at Google. So we ship all the kernels on all the Android devices out there and my two areas are Linux scheduler, which is a piece of code in the column that takes care of placing tasks on different CPUs and load balancing the tasks between the CPUs and scaling the CPU frequency and stuff like that. The thing I work on is tracing, which is f-race and perf and all these powerful tracing technologies available in the Linux kernel. And BCC is the latest thing that I've been working on. So I'll start off with a demo of a tool that lets you look at all the file reads and writes that are happening across the system. And also this thing is how powerful BCC is and because we're in the kernel, we can't miss anything, anything that's involving the kernel itself across all processes. We will see that in the kernel. So I'll just show you an example to a file top. Just showing my setup, I have EmbeddedTargetHikey964 which has the high-silicon 960 SoC and it's running Android and it's connected over Wi-Fi to my laptop and so everything I'm going to be doing is over the Wi-Fi network. Someone will be tracing the Linux kernel remotely over Wi-Fi. So before I run file top, I had to set up some environment variables where my kernel sources are, how I'm communicating with the target, what the target architecture is, stuff like that. So I run file top now, you can see some warnings that happen when the tool runs. I don't want to go into that now. But right off the top, you can see the Wi-Fi service in Android is continuously reading from Wi-Fi stats format to get information about the Wi-Fi interface. So you can see that you have this global view of what's happening on the Android system. And so I'm going to open an app called the Contacts app. You can see it did a lot of reads and it's reading the contacts database there and I'm going to create a new contact, just like how you would create a contact in a regular Android device. I just created a contact and you can see that there is this entry here that shows that the contacts database was written into 104 kilobytes, 53 bytes in total. So it's kind of nice to be able to do this over Wi-Fi to get that kind of observability. You can see that there's not a lot of data transfer happening because all the data collection is happening in kernel. As these like thousands of events are firing, the data has been collected in kernel aggregated there and periodically sent to user space. In this case, over Wi-Fi. So all this builds upon a lot of technology that's not visible. I just ran files off but there's so much going on under the hood that you do not need to know. That's the beauty of this. So broadly speaking, there are these six different signals that you have in the kernel that you can monitor. This is not related to BCC or eBPF or anything. It's just tracing in general. These are the signals. You have the static trace points. First of all, which are points in the kernel that are inserted, trace points that are inserted at compile time. So at compile time, you can have a static trace point that emits some information if it's when it's enabled. So they call static because it's done at compile time. These are the S-trace events that you have. And then you also have dynamic trace points which are trace point probes you can insert at runtime that you don't know in advance. That's how file top works actually. It instruments VFS read and VFS write in the VFS layer dynamically. So these trace points are not decided ahead of time. That's what they call dynamic. That's the K probes infrastructure in the kernel. That's what allows you to do that. Then you have the user space equivalent of that. U probes lets you do that for user space applications. U probes is dynamic. Again, you don't need to do it ahead of time. Then you have something called USDT which is the static trace points for user space. And then last two are Perth events. So you have your performance counters. Those are also signals that you can attach hooks onto like cycles and cache misses and stuff.  That's how Perth profiling works. You have an interrupt that fires 100 or 500 times a second and you can run some code that can observe the system. So these are the various signals that we can build on top of to get the observability that we need. Now let's come to BPS. So BPS lets you write programs easily that can run when any of those signals are triggered, when any of the signals happen. The way it works is you have a BPS program that's compiled to BPS bytecode and that's loaded and verified by the kernel. So once it's loaded it's sitting in the kernel ready to run. And then you can attach that program to any of those signals I just mentioned in this slide. And program runs in kernel, collects some data that the signal might have emitted or you might just want to read some kernel data structure or something like that. To get that information that you're trying to observe and finally the BPS program in the kernel can output that in different ways using maps. Maps is a data structure that lets you aggregate information as you get it. So for example histograms is a kind of a map. And there are various other ways to image information from the BPS program. So BPS is a technology that lets you do that. So now let's come to BCC. So what is BCC? BCC actually puts all of that together. So it lets you set up all of these signals others talking about, the dynamic signals especially, activates them and all of that. And it also does all of this stuff, takes the program that does the observability like file top in the demo I just showed you and builds a BPS program out of that. And finally it does all that automatically so you attach it to the signals that are required by the tracing tool. And it also has a set of libraries written in Python and stuff that can read those, read the maps and present the information to the user periodically. BCC kind of makes it easy. The beauty of it is it does all this powerful stuff behind the scenes that you don't even know. Like when I ran file top you didn't see any of that happening. You just had to say file top and it was showing you all this information. There are like 100 plus tools that you don't even know, need to know how they work and just run them. That is opposed to running tools like Perf or something like that where the running those commands might be more complicated and interpreting the information. So BCC does all the powerful stuff for you but really it's very easy to run. So feel free to interrupt if you have any questions so far please. Okay. So traditionally BCC has been typically run on the same device where you have your kernel sources and your clang stack and all of that and the kernel is running on the same machine where you're running the tracing tool. And that would have been fine but unfortunately you also need kernel sources to be able to run the tools. As I showed you, you needed to set an environment variable to run file top that pointed to the kernel sources. So all that is, that doesn't work very well for cross development kind of development model where your target is remote and your development tools are on a different machine. So that makes it very difficult to run these powerful tools. So what I did is I separated out the whole stack into two parts. One is the stuff that doesn't need to run on the device you are tracing and then the stuff that needs to run. So the stuff that needs to run on the target is just the stuff that does a low level things that are not compiler related like loading a BPS program once the bytecode is available or creating, you know, attaching the BPS program that was loaded to a trace point for example. These things are low level and are not like, not related to compiling the tool or compiling the program that produces the BPS bytecode and stuff like that. So, you know, and the way I got that to work is I started this project BPSD which kind of encapsulates all the low level things that BCC needs and have that have BPSD run on the target. So in the demo I just showed you BPSD is running on the on target and BCC is running on my laptop. So just to summarize why we want to do this, you know, we don't want to sync kernel sources to our remote target. We don't want to download them onto the target. We might not have space for that or the sources might get out of sync with the target. It's really unnecessary. With this kind of a model, you don't need to cross compile Python, Clang, all of that stuff. It can run on your host. Quite easy, very well run on your host. And this fits the cross development flow where as I was saying, you build on your host and run on your target. You don't do it the other way. And also your host machines are more powerful. They have more processing power typically than battery powered devices. So it makes sense to do the compilation of the tools on your host. And also many embedded targets don't have symbols in their executables. Things like that. On the host you have a greater chance of having that information. And the tools need that information to need the symbol information in many cases to produce meaningful results. So that's why these are all the reasons why we want to do it. Any questions so far? No, I'll move on to the demos and show you some more demos. So I guess I'll just move on to the demos then. So this is a tool. So I just showed you five top. It's a source like hardirqs.py which summarizes the amount of time that is spent in interrupt handlers over the period of time that is specified in this case 10 seconds. So every 10 seconds it's going to show me the total time interrupt handlers spent on CPU. And that's useful because you might run into real-time issues or latencies and stuff like that. If interrupts are spending too much time you need to, as a designer, you need to make sure that your interrupts are not taking up too much time. So these are all the interrupts that spend time in the last 10 seconds. The second column is in microseconds. So I'm just going to do some UI activity and show you that the GPU interrupts are now going to take a lot of time. You can see that Mali here took like 141 microseconds, milliseconds, out of the 10 seconds. So that's hardirq.py. Just to show you the tool itself, what it looks like. The tool itself has an embedded, it has a C program embedded in it that is compiled into BPS. Oh, shoot. Yeah, no problem. So where this tool works is it uses K-Probes to instrument handle IRQ event per CPU function in the kernel and it attaches the count-only function. And then when that runs, there are two modes, actually. You can either count or you can time. So I think the default is time. So for the mode in which it times let me show you how that works because that's what I demoed. So for that, trace start is executed whenever handle IRQ per CPU runs and it makes note of the starting time stamp and trace completion runs whenever handle IRQ per CPU finishes and it uses the time it stored in the start. It takes a difference between that and adds that to the histogram that I showed you. So some other tools that I wanted to show you. This one I won't be demoing but it's useful to see how much time was spent doing block IO operations. This summarizes, shows you a summary of every so often of all the processes that were doing block IOs, how much time they took to run. Here, Biotop and Biosnup there in the slide. The difference between Biotop and Biosnup is that Biosnup shows you an entry. It shows you, it doesn't refresh every so often. It shows you like a tracing type of output. It doesn't clear the screen, but Biotop clears the screen and both these tools show you the latencies that the bio operations took. So this can give you an idea about whether the disk IO is slow or not, for example. Another tool is CacheStat which is useful to monitor what's happening in the page cache in the kernel, whether you're having page cache hits or misses, stuff like that. If you have too many misses in the page cache that can indicate that that is what is causing a performance issue. So I'll just show you a demo of CacheStat. So I'm going to remote into my target board over ADB and I'm going to run a while loop that reads a 100 MB file. So there's a 100 MB file. I'm going to write a loop that reads the file. And then in another window, I'm going to run CacheStat every two seconds. You can see in the third column here, CacheStat is showing me that all the cache accesses were hit because the first time the 100 MB file was read, it was cached into the page cache. I'm just going to drop all pages in the page cache. And so the next occurrence of CacheStat, you can see that there were a ton of misses. That's because the page cache had to be repopulated with the file. So the fourth and fifth column are not working because it's reading files that are on my laptop instead of the remote target to populate them. But the first four columns are valid. So again, to run this tool, all I had, you know, it was very trivial. I just had to run the tool. I didn't have to specify any complex arguments or anything like that. It's very easy to use. If you want to do more complex stuff and don't want to write your own program, there's this set of tools called the Multi-Tools. It's one of my favorites, the Trace Multi-Tools. And the Multi-Tool will automatically, will generate an EBPS program for you and compile it with BCC based on the arguments you give it. And you can do a lot of powerful things with it related to K-Probes and stuff like that. In this slide here, I want to see all the open, the file opens that are happening on the system. So it's functioning in the kernel called Deuces Open, which receives arguments from user space to open files. And this uses K-Probes to instrument Deuces Open behind the scenes. So I'm telling the Trace Multi-Tool, I want to instrument Deuces Open and print the second argument that is fast to Deuces Open. Second argument is the file name. I can just do that. It's already showing me. I think that's the while loop that I wrote in the other window that's continuously running. So I stopped that. And so if I do some activity in Android, it shows me all the opens that are happening, surface flinger and all that. Some files are being opened a lot of times. You know, it might be better to open them just once. And that might give you an idea of maybe something is unnecessary done and this tool will tell you that, yeah, you're doing this and it's not necessary. So it's very useful for that kind of performance debugging. And I wouldn't be running a demo of this, but it's still called RunQLength, which uses the Perf Sampling mode. And every time Perf Sampling event occurs, the BPS program runs, and it looks at the run queues on all the CPUs and adds information to a histogram about what the run queue length, like how many times a certain run queue length occurs. So histogram has that information in the end. This will give you an idea about whether there are too many tasks running on a system causing performance issues. Sometimes you might have, so you can either have too many tasks running on system or you might have too few CPUs. Like if you have too many tasks in the run queue, then the run queue is overloaded and you might have performance issues. So this will let you look at, get that information. Many things are not working right now, but they are very close to working. I call these the boring issues because there's a clear path to getting them work but just need some work to be done. So that involves user space, dynamic and static tracing, symbol lookups, new tools, tools that need to read stuff on the remote target but they read stuff on the local machine. That is my laptop in this case. There's a full set of issues I've created in the issue list. I already welcome any contributions there. Maybe we have some more interesting issues that it's not clear how to solve. One of the issues is that BPFT itself, the reads and writes it does, can cause, or the other system calls it makes, can cause a lot of tracing activity itself. So sometimes that interferes with the tracing tool output. So I'm thinking maybe we can blacklist the BPFT process from being created somehow. Some tools can generate a lot of output. So we have to reduce the amount of data that is being sent somehow, maybe batch the data together, stuff like that. Perf polling is another issue that we're looking at. And also logging is not working. So there are a couple of interesting issues that it's not clear how we solve but we have to solve them. So that's all I had. This project has come to the open source so you can go to GitHub and download it and contribute to it. I wrote an LWN article that goes into much more detail about how it works. And Brandon Greg has set up nice resources on how to run these tools. So I would encourage you to go check it out. So I guess we have a lot of time so we can just do a Q&A and any discussion on... You said that this works around some of the issues that you have often tracing remote targets like all those problems... So usually as an embedded developer you have the kernel sources available because you build them on your host drive. So you just find BCC to that using the work that I did. Now it's possible for you to set an environment variable pointing your kernel sources in that and BCC will look at the kernel sources on your host, not on the target, on the host and use that to build your BCC program. Debux symbols are not needed in the general case to run a lot of these tools. They're needed to interpret the output sometimes. Like there are tools that do profiling of stacks and stuff like that and so then the output will just have instruction pointers if you don't have symbols. So for things like that you need symbols on the host but otherwise you don't really need symbols. It seems like this has an applicability beyond that it's so serious if you had any intentions for authentication, authorization and securing the connection between the client and the host and also what you do in failure mode. So say I've launched something against my host my Wi-Fi connection graph or something like that is my BPS program going to continue running on that host or does BPS need a hand? Yes, so the first question about the other ways you can use this the way it's been written, the architecture right now is very plug and play kind of thing. So for Android I have an ADB module that does Android specific stuff but BPFT itself is written with a standard IO interface kind of thing so every command is like one line and you send that to be, you can run it locally and just give it a command string and it'll respond to it. So to load a program we give it the byte code in base 64 encoding. So somebody else was telling me they wanted to use it on servers and over SSH so that's definitely possible to add very easy to add I would say. The other question is failure mode so right now it's very early stage so we haven't really thought about that so any kind of cleanup will have to be done you know for Android what happens is the ADB connection dies the processes that were started remotely will also be killed automatically so and then all the resources will be freed that way but yeah but cleanup will have to be thought about there's a lot of those kind of low hanging fruit issues that have to be worked on so yeah feel free to pull the code on GitHub and ping me if you want to work on this stuff and contribute yeah right now it's just a single device but potentially that could be done yeah I don't see it's just a matter of somebody taking it and applying it to their use case I don't see a reason why that cannot be done yeah


Hey, so I'm Michael A. Day. I'm the least important person in the room because we have a fantastic panel of people. We've got Nigel Cook. Nigel's an Intel fellow. He works in their software-defined infrastructure group. And generally a wicked intelligent guy. And kind of fun to party with too. We've got Martin Kiss. Martin is a healing MVP and is an OpenStack ambassador and founder of Hungarian OpenStack user group. So Martin, I've not partied yet with Martin but I look forward to the opportunity. We've got Sri Ram. Sri Ram doesn't really need an introduction so I'm not even going to introduce him. You all know Cloud Don? If you don't know Cloud Don, then you need to know Cloud Don because Sri Ram's cool. Thanks, Mike. Thank you. And he's easily embarrassed. Should I continue because it's kind of fun to watch him? No, I guess we have the entire session so that's okay. Thank you. So we're here to talk about a number of things but mainly around Cloud and the Enterprise and understanding what kind of obstacles exist in getting Cloud into the Enterprise, what kind of barriers to adoption exist. They asked me to kind of moderate the panel. I'm going to ask a few questions but these guys are super intelligent and the majority of the time should be them talking.  So I'll drop my mic and then let things proceed. I'll throw out one question just to get everything started. So OpenStack and the Enterprise, is this real? Is this something that we should be thinking about? What are the barriers to adoption for OpenStack and the Enterprise? So it is interesting. It is definitely real. There are challenges, obviously. We just had a talk about workloads, the applications running on OpenStack and we had a great panel yesterday, right? So what are the barriers? Traditional Enterprise applications are not Cloud-aware applications. What I mean by that is they need to be highly armed. They are mostly consistent. They are not like scaling up, scaling down kind of applications and they have high SLAs. If you look at OpenStack infrastructure, it doesn't support highly availability always, right? If you lose a VM, it doesn't wake up again by itself. So those are the barriers that Enterprises have in their mind. The other thing that's also happening is that they don't want to miss the train of OpenStack. They want to have OpenStack but the problem that they have is, okay, I need to have OpenStack but I have this old legacy application that I want to, that can I put it on OpenStack? Can I run my SAP workload on OpenStack? Or can I run the CRM application on OpenStack? I see that that's not the right way of approaching it. There are two things, right? Enterprises are not going to remain stagnant. They need to look forward, which means that you need to have new kind of applications, right? Whatever you are running, yes, you need to continue the investment there. You don't want to kill them but it is not the only type they're going to ever have later. So plan for those applications, right? And those applications will not be these static applications. Any application that gets written now or will get written in the near future is going to be cloud aware. But that's going to be, you need to have mobility for instance. No enterprise can survive without having or supporting mobile applications. They need to be social. You cannot have employees to run on all desktops anymore, right? So all these are your requirements. So think in those terms and plan for those things and OpenStack will support those applications. So that's kind of a starting point. Okay, in some way I can agree with Don. So basically the huge change here, the legacy application simply won't fit into this cloud model. And I guess this adaptation will take some time because deploying just a simple cloud won't solve all of our problems. And we need to change at organizational level. We need to change the development culture we are using. And especially we can learn a lot from the startup companies. They are using this typical DevOps culture when CISAD means have some developer knowledge and they can work together. And basically a lot of things is different in cloud. If you want to see a well-performing application, we need to build a cluster solution and stateless applications. And of course we need to do a very, very different development model. And we can learn about this typical continuous integration story, how we are developing software. It is changing because we think cloud software differently. Maybe the earliest dates will be different. Let's see the OpenStack continuous integration system as an example. We are using OpenStack to develop OpenStack. And if we want, we can do a release every day, basically, using those new tool sets. So I think it is a very early stage of our huge revolution and change of the entire IT industry. So I guess I'm the contrary point of view here, which is probably why I'm on the panel, I guess. So I have to take odds with some of these statements here. And I think OpenStack is getting towards the enterprise of private cloud. In fact, there are some distributions which are great for that. And that's been because of additions people have made into that cloud distribution, which they're now up streaming. So it will be available to everybody. But I don't buy into, hey, cloud, it's got to be stateless. It's the only thing that works. Cloud infrastructure is by nature unreliable. I don't think that's how the public cloud providers build and run their infrastructure. And I don't think that's how the enterprise private cloud should think about or run that infrastructure either. So to me, it's a matter of a cloud infrastructure delivers a certain SLA. And if you can, once you can deliver to a particular SLA, it's a little bit like driving down the highway, you can drive at 20 miles an hour, you can drive at 40 miles an hour, you can drive at 60 miles an hour. As long as you can set the speed that you want to run at, you can go different speeds, different distances. What's bad is if your car is going 20 miles an hour and then 60 and then back to 15 and then to 30, that that's unreliable. I think what we've got to get to is a reliable cloud infrastructure. And then all kinds of applications can run from stateful to stateless. So so I was just going to say that's an interesting point. I think that as we look at enterprise cloud, right, and if I may just summarize that there's this assumption that infrastructure that is cloud enabling is inherently unreliable in all cases. And anything that's taken to the extent, right, that that it's being taken in the current model, right, is is wrong, right. Public cloud providers, many public cloud providers run a tight ship. They have a great management infrastructure, they deploy good hardware, private cloud infrastructures should should learn from what they've been practicing for the last 20 years, right, which is deploy good hardware, deploy good infrastructure, deploy good management. And that resolves weakness in the cloud. I have a point to Nigel's right. So you're talking about cars, the only problem that enterprise seemed to have is car is not the only vehicle, you have multiple modes of transportation, mopeds, bikes, bicycles, anything, right. So if you have a car and you think about a car, you only you are only thinking about having a fancy car or a reliable car, you are only thinking in that pattern, right, cloud is not like that, it's going to cater to multiple ways, right. So you have a different approach here. And so so it's the it's the error of arguing by analogy, because you can always extend it. But I think, you know, the great thing about OpenStack, and in fact, this was highlighted by the keynotes this week, is that people are going to it because they want to build a tool chain on top of an open set of APIs. And then under that open set of APIs, they can have bare metal with ironic, they can have a KVM hypervisor, they can have a Docker container, they can have Hyper-V, they can have VMware, they have all that sort of choice in what they deploy on underneath, you know, a consistent and open API set. And I think that's how we need to look at OpenStack is enabling choice, both in the size of compute, but also then in the quality of service from storage and the quality of service from I totally agree with you on that point. Yeah, I agree on that. So we are not saying that you cannot move your existing application stack into OpenStack, you can definitely. But referring back to some real life examples, we were using OpenStack cloud and there was some transition periods. And for example, if we have the proper tool set to automatically launch or application stack inside the cloud using the OpenStack API, for example, one time one of the or each Republic cloud instance simply died. And it was much easier and faster to launch a new one, instead of calling the support and waiting for their resolution to bring back this instance to life. So I think this is the big deal in cloud. Oh, I think so too. Yeah, yeah. Well, you know, you think of the, the, the, the legacy application running on two bare metal servers, that first bare metal service, don't you have I got a four hour, you know, five day a week support policy, or is it eight hours, is it 24 hour with cloud? It's 30 seconds, I'm going to go back to my console and re provision. That is the benefit of cloud. So that's the benefit. Actually, that's the point that I think you are talking the other side of that one. Probably the cloud is not necessarily inefficient, or it's not necessarily failure prone. But what we are saying is that that assume failure and develop for that. So that's a better, but that's a better approach. The legacy applications are the developers, they're like pampered with highly available infrastructure, always on infrastructure. But what is reality is that it's no longer the case. I think the paradigm has changed and developed towards that. That's what is changing. Yeah, yeah. Yeah. And one, one more comment. I don't agree that public cloud is much more stable than private cloud. So Amazon, for instance, Azure, for instance, they have their downtimes, they have their outages, right? So they absolutely do. But you know, I was listening to Tim Bell talking about the joy of upgrading this opens that cloud from, you know, Havana, Iran. And what I'm saying is, you know, I think we've got a few steps to climb here. And it's definitely doable. But so to that point. If I were to ask each one of you to give me three obstacles in priority order, that you see around enterprise cloud adoption of OpenStack, what would you, what would you say those are? Oh, now he gets to go for it. I can go for it. I can go for it. That's fine. I didn't want to. Now remember, in priority order. It gives me time to think. Mine is like incorrect expectations. Second is the processes, internal processes. Third, I would say it kind of goes with the expectations, the kind of work, the approach that you're taking. So by expectation, what I mean is that you've had something going, it's been working for a decade, at least a decade, right? And you have the same expectations that this new shiny object will give you. You need to fix that one. Whether it's correct or not, I'm not here to argue whether it's correct or not. I'm just saying that it's a different set of expectations you need to have and think in those terms. And second thing is like enterprise IT, the way it operates. So anything that you want to have, you're creating an SLA, the IT will, you need a server now is going to go through an entire set of the question process. You create a ticket, it might take its own sweet time, a week, a month, depends on how fast your organization is, right? That's going to change. People, the developers expect to have a self-service. They're going to expect that one and they're going to operate on that one. What implies is that you have more agility, that's great, but it also create a lot of friction. It might seem to be a threat for your system, I mean, IT department, right? They're going to resist you naturally. And your entire way of operation, you are building any moving piece that have been operating with that expectation, they have to change. So that's a new process, right? You need to educate that. It's not that just you're going to swipe a credit card and get your system when you are done. It doesn't happen in enterprises. You need to change that one. Third thing is, again, like it goes with the expectations there. You had set of setting applications, you need to think forward, right? You don't want to spend your cycle just for thinking about porting or rewriting. Just let them go or let them run. It's okay to have two set or three set of infrastructure. But anything, like Nike's example. So they were moving their applications, they might have had their old applications. Anything moving forward, they were thinking mobile first, right? And they were thinking social first. So they were thinking in those terms and releasing their applications. Those, you can argue whether it's going to be enterprise or not. I mean, if it's being adopted by a big company, a Fortune 500 company, it might have a different usage characteristics. But like depending on the adoption or the revenue involved, like these are new set of applications, you should think in those terms. Those three are my top three careers. I think that cloud adoption into the enterprise mustn't be started by moving the business core applications into the cloud. Let give some time for the team to build up a pilot cloud solution and learn the new model there. And I was at the Linux conference in Ducado at the OpenStack booth and I met with three of the customers. And most of the questions came from that they had problems. First of all, most of the customers don't want to start a brand new IT infrastructure from scratch. They have existing network and OpenStack need to fit into this existing network. Nobody wants to buy a new device just for OpenStack. This is one point. The other thing that is very serious, that they want to upgrade their OpenStack cloud systems. And I think it requires a very special way of finding of OpenStack. So I see these problems currently that we need to run to start the cloud adoption into enterprise. So this is the first steps we need to do. Awesome. Nigel, have you had enough time to think?  So, you know... It sounds so good. No, no, no.  That was really good. And the legacy system piece is high on my list as well. So I'm going to leave those. And I'm going to give three more. But just assuming these other ones are already covered by the other speakers, right? So let me get this straight.  Yes. And giving us the rest of the list. I guess I am. Okay. All right. Just check.  But those are all very valid points there. But I think, you know, specifically if we talk about OpenStack. So assuming that you've already made the mental model that hey, we're going to do cloud and, you know, now OpenStack specifically, if I narrow my view to that, right? I think that deployment and upgrade would have to be like the number one thing that needs to be addressed in OpenStack to get real mass adoption. I would say the maintenance functions in OpenStack, the operational aspects of it, are another piece that need hardening. So particularly here, you know, I have to do some firmware maintenance on this row in my data center. How do I do that without complete chaos in my OpenStack cloud? But that's a great use case that needs to be addressed. So and then the third one then, if I've got one left, I'm going to have to reuse one here. Definitely the uptake of a legacy environment. So I've already got a data center full of my favorite hypervisor. You know, maybe it's a VMware environment. Maybe it's a Microsoft environment. What I need to be able to do is move that into the cloud versus build something separate and have to sort of migrate it in. And so I think that would be, if we had all those three, I think there would be runaway sort of uptake of OpenStack anyway. So I'll ask kind of a fluffy question. A little bit of a fluffy one because we've had a couple of really hard questions. What's your favorite project in OpenStack? Swift. Why? Sorry. I like, generally, I don't like the need for specialized hardware. So Swift is a great example of how you turn commodity hardware into a very simple set of hardware into highly-layable, highly-rend and distributed storage. So that's why it's personal. If you're asking me in a different way, I would say Neutron because of its problems or at least how we sail to those problems. Interesting. Yeah, my favorite project is not an OpenStack one. It is Seth that solves this commodity storage problem. So you're super setting. Yeah, but anyway, I love all of the OpenStack projects from Trove to Nova. Yeah. My favorite project or my favorite latest feature in OpenStack, let's say, would have to be Horizon. There was a metadata repository that was called, added into Horizon that was called Graffiti. And the idea of that project is to provide annotation of particular images and requests so that as a user requested an instance, it could be best matched to the right space in the underlying infrastructure. And that gives, that gives choice, it gives the ability to have a sort of a differentiated cloud environment. So that one wins my, at least my vote for now. Ideally, making the scheduler, giving the scheduler the ability to make informed decisions. Informed decisions, correct. Informed decisions, yeah. Can I ask you guys back? What do you think of Triple-O? You've got to answer that one. Oh no, I mean, I'm not going anywhere near that. Audience can answer that. What do you think of Triple-O? What do you guys think of Triple-O? Question for the audience. What, do you guys know about Triple-O? So Triple-O is, Triple-O is a project called OpenStack on OpenStack. And hence Triple-O. All right. And it's a deployment capability. So it allows you to install OpenStack using OpenStack. Kind of inception. Yeah, anyway, we are using OpenStack into a lot of different things because we are using OpenStack to develop OpenStack on the test. So, you're not getting an answer from the audience, I guess. Does anybody in the audience have any questions? They come down to enterprise. The bugbearer of most things is you can spin up a machine in seconds nowadays. And we all hear about your favorite projects. But the one thing that enterprise hates is the network. I'm surprised that you don't include any kind of integration and neutron and all that sort of stuff into legacy critical networking. Because I can tell you that's where most enterprise networks lose. Or let's say OpenStack loses its agility. So you've only got two legs on this chair. And I know there's a lot of work going on with the neutron. But OVS is kind of a disaster as regards enterprise networking. And there's lots of things like that. So I'm kind of surprised that that wasn't high on your list because everybody goes to the compute and the storage. So why wasn't that high on your list? Because I'll tell you that's the most unagile part of OpenStack. I mean, neutron is an interesting, the networking is an interesting space. And I think that, so when I included sort of set up in deployment, I was actually including that. And when I was calling legacy ingest, I was also including that networking component. But you're right. Networking is, you know, is a very interesting and inverted column of space in cloud. But I do think though, you know, despite some of the butchuring issues around the V-switch, that the notion of using an overlay network to build your, you know, your interconnectivity in cloud is a great direction. And is one that lets you basically overlay on top of an existing infrastructure versus, you know, having some deep point integration into the, into that existing infrastructure. So I think it's, I think it's on the right approach. There are actually, I must say, technologies from Intel, which are being upstreamed, which dramatically improved performance of the V-switch. We integrate it with DVDK, which will give you high performance and lower CPU utilization, for example. So I love the direction of the V-switch, but clearly it's, it is a, a work in progress, but great progress has been made. I want to add to that one. As a community, yes, we understand that neutron is a mess, but the community has taken good action so far and trying to get it to a good stable state. But I wouldn't call that as a, a barrier for adoption, right? You still had, until last release, right? You still have no one network working for, for your networking needs. So for example, like Rackspace Public Code users used, at least used no one network for the networking. So it is, it is not, from the operational point of view, it is not really a blocker. It is painful, but I wouldn't call out that neutron is messy, so OpenStack is not ready for adoption. There are a lot of features to that one that we, we talked about a lot of varying features, varying factors, but I would like to separate those things, right? And, but having said that, the community has really has worked well on that one, particularly that are, that are cool features that have gone into the Juno release, DVR, Distributor Virtual Reuter for Instance. Yeah, that's a great feature that is going to remove a single point of failure, making it much more scalable. So that's kind of my take on that one, how to, I personally like the way that, that the community has taken and realizing that, okay, it is messy, so we need to fix that up. You as the developers, anybody would accept that how messy it is, but we have worked on it, and we're still working on it. Yeah, yeah, basically, I think that Neutron is not about it, because it is just an obstruction layer to manage networking operations. And I think when we have some bad experience related, maybe some drivers that Neutron is using, and I think it needs some time to evolve and, and some, some basic evolution to go into the right direction. It was not designed correctly from the beginning, so it just took more time to fix it. Yeah, yeah, of course, of course. But I think it is an advantage of this open collaboration that we can fix things. Maybe it requires some time, but, but we have a very strong ecosystem inside OpenStack, and I'm sure it will be fixed, because if we want to run OpenStack clouds in the future, in a stable way, we need to solve that problem. So definitely, I mean, I'm not going to mention names, but there are distributions, which I think have this well in hand already announced and sort of pending announcement. So I think the networking, I mean, networking always is a challenge, but I think that the distributions are addressing that. And I must say, I think that the, if anything, the interest in NFV from the telco end is if anything helping to accelerate that, because those guys are pretty sensitive about their networking and are pushing requirements and are pushing the community in a direction that flows back down to the enterprise, you know, in a very positive way. Interesting. So we've got one question over here on the side. One of the complaints from Enterprise is the cadence is too fast. Six months for de-cycle, it was even mentioned on the first day keynote. So that, well, I'll ask another question. What do you think is the appropriate timeframe for back porting features into previous releases to support enterprise adoption? No. The enterprise doesn't want to adopt it, because it takes a lot of data to change them. Well, if the enterprise, so if the, I can see your point, but if the enterprise adopts ice house, for example, and there's a two-year back porting requirement, for releases, right, then they can effectively get those new features and still maintain their older version. So I guess my question is, what do you think is the right timeframe? Yeah, I think we are seeing this entire listing from a wrong point, because maybe we could focus on the future to make... Nugget of wisdom, would you give to the audience to leave? We can start with Shreve. Yeah, yeah, I know. That's one sentence. Nugget of wisdom is the most important. You know, I've watched this, the OpenStack community grow over, you know, this is my fourth or fifth summer. I mean, it's amazing about how much it's growing. It's amazing the diversity of interest is coming. And I think that this definitely is the... You're going to catch a train. This is definitely the train to catch for a cloud, both public and private. Yeah, I think OpenStack is still in these early days. And the next five years will be very exciting for each of us. And I'm sure we will solve most of the problems that we have now. And anyway, it is a fantastic moment and I enjoy every minute of it. We'll make two points. I wouldn't expect anything else from you. So, first thing that's kind of... Since we're talking about expectations. So, we have a right set of expectations is that we just succeed. As a community and as a product and for the users and the developers, everybody is having a better idea about what the right expectations are. So, go with that. If you need any help, I think you are getting help in the community. So, it's always very helpful and you get enough feedback. Second thing is like, I want to mention that we are in the inflection point, I would say, in the community. We've been there for four years and it used to be a lot of projects now. And now it's the time that big vendors start to remain, which is good for enterprise customers because they need a lot of support, which is the big, big play of the catch here. I had IBM, they're going to provide it, right? Naturally, they come with their own distribution and they understand the enterprise paying points better. So, FTE and OpenSpot, for instance, like we have enterprise pardoning, we have scale and fit with all those things, right? So, what I'm trying to say is that those vendors will give you the stability and maturity that the enterprise customers want. Having said that, it is not the end of the system. I'm seeing, I'm expecting the second way of thought as to where more innovation is going to keep happening. Like, you can expect a lot of, we kind of solved the infrastructure problem, we looked at how do we deploy it in here, right? How do we make the automation in here? So, we're kind of in a state where like we have a good answer for that. But now, next step is like, how do we make it much more secure? Can we make cool things about it? Can you try bolder things like making daughter companies and stuff like that? Those things, these big vendors might be different to crime, but that's where the second way of thought as to where it's coming. So, it's an active, equal system, it's going to keep happening like that. So, I'm very positive about the infrastructure that I'm learning. Together, we're going to learn, and together, we're going to follow that. Well, thank you very much for your time. I really appreciate it. I'm sure the audience does as well. And thank you guys for enduring the panel discussion. I hope it was worthwhile. Thank you guys. Thank you. Thank you.


What do you do for a living? What do you do for a living? What do you do for a living? What do you do for a living? I run a TikTok, where I ask people what they do for a living, a nice car. YouTuber It's Daniel Mack asked over 100 millionaire super car owners what they do for a living. What do you do for a living? You don't do anything? So naturally I decided to watch all of his videos, all 118 of them, and this is what I found. And in this video, we're gonna go over three main things. First of all, what percentage of people went to college? Did you go to college? What industry they work in? You know, I took my dad's trust fund and just a little bit of insider trading. And what they do for a living. Hey man, I love your car. What do you do for a living? I'm the CFO of only fans. All right guys, this was a ton of work that I did for your enjoyment. So if you haven't done it already, make sure to gently tap the like button and hit subscribe. First of all, what industries were these super car owners in? Well, by far the most common industry was technology at about 26.7 percent. And this is something that I've been talking about quite a bit on my channel. Tech has so many opportunities and if you think about it, it makes perfect sense. One person who knows how to code can create an app that literally millions of people can use. Usually in business, if you're just one person, there's no way you can scale the business to get to millions of people. However, there are a few skills, a few types of businesses out there where you can and technology is one of them. And it really doesn't matter whether it's a skill, a degree, an industry or a career, technology is just good. And so I'm not surprised that this one was number one. Number two on the list was finance at 14 percent. And this one has everything to do with investing. Finance has always been a fantastic industry to be in. It's probably created the most billionaires out of any industry out there. Very similar to finance is number three on the list, which is real estate at about 11.6 percent. Real estate will always be a great way to build and grow your wealth. And so it's no surprise that there's an entire industry around it. Whether it's building real estate, renovation, buying and selling houses, investing, this is a booming industry that has created countless millionaires. Number four on the list was healthcare at 9.3 percent. And this is my industry because I'm a healthcare professional. And the cool thing about healthcare, in my opinion, is this is probably the most stable industry. Doesn't matter if it's the end of the world, there's still going to be doctors and nurses taking care of people. They are truly essential and they'll be the last people to lose their jobs. Number five on the list was entertainment at 4.7 percent. And this is anything from having a social media following to being a professional actor. With the onset of social media, entertainment has become much more scalable.  You make really entertaining or informative videos, post them on YouTube. And eventually, with a lot of hard work and a little bit of luck, you'll probably blow up. This is an industry that's on the rise, in my opinion. And one really great thing about this that you probably don't realize is that it's relatively recession proof. When countries go into a recession, people end up spending a lot of time indoors and just consuming entertainment. And that means if you go into a recession, you'll likely make even more money if you're an entertainer. Tied for fifth place was automotive as well as aviation, both at 4.7 percent as well. And here's the entire list if you want to pause the video and just check it out. Next on the list, we're going to be talking about careers. And I actually think this is kind of genius for him to do this, because from what I understand about him, he's the type of person who wants to be able to not only have a millionaire lifestyle, but he wants to be able to do it at a young age. Now, one thing I always talk about on this channel is if you're thinking about going into a career, for instance, the best thing you can possibly do is reach out to people who are already working in that career and ask them questions. And so he found a genius way of basically doing this. People who have supercars, generally speaking, are going to be young, and so they're millionaires, but they're young enough where they can actually enjoy their money. And so he asked a bunch of people who had supercars what they do for a living, and here's what he found. Because the truth is anybody, even if you have a relatively low paying job, can get rich over the long run if they consistently invest their money. And I've talked about this quite a bit on my channel. And the math here is stupidly simple. You can make a million dollars easily by investing $10 a day because of the magic of compound interest. But the unfortunate thing is you'll likely be too old to enjoy it. And realistically speaking, most people want to be young enough to where they can actually enjoy their wealth. So here are the careers that will give you the best chance of getting rich quick, which means having a supercar under the age of 40. So by far the most common career was entrepreneur slash business owner at around 36.5%. When it comes to having enough money to be able to afford a supercar at a relatively young age, there's no question here that becoming an entrepreneur is going to be your best bet. Now I will mention that a lot of the people who started these businesses were in some of these industries that I talked about before. So you could sort of put them in multiple categories. So we're talking about like finance, tech, entertainment, etc. But the second most common profession was actually a three way tie at 6.8%. So most of the people who fell under this category were social media influencers. Tied for number two was tech. And this didn't include people who owned their own businesses. These were people who were employed. So they were employees and tech related jobs. So we're talking about software engineers, for instance. Also about 6.8% of the people said that they were rich because they had some kind of relationship with another wealthy person, whether this was inheriting money or maybe they were married to somebody. Now you might be surprised at how low this is, but research has shown over and over again that most people who become millionaires actually don't inherit their money. Depending on the study you look at around 80 to 90% of millionaires are self made. Doctor came in at number five at around 5.4%. Physician is obviously one of the highest paying professions, but it's also very difficult to become one. And you're going to be working long hours and it's just pretty challenging in general. Lawyers, finance professionals, and executives like CEO, CFO, etc. all came in tied at 4.1%. Now I do have to admit here that it was a little confusing sometimes to tell whether someone was the CEO of a business or whether they owned it. And yes, there is a difference there because business owners can hire CEOs to run their business. So whenever there was a question there, I just assumed that the person was the owner of the business. However, there were several cases where they were likely just a CEO. So chances are executive was actually higher than 4.1% and entrepreneur was likely a little bit lower. Also, I would like to say here that most of the people who were employees and they weren't business owners were likely in leadership or management positions.  And a lot of the findings here are consistent with what I found when I did the video, the careers that create the most millionaires. In the top 1% of earners, you saw a lot of doctors, executives, managers, lawyers, finance specialists, marketing and sales people. In the top 10%, you saw a lot of managers, software developers, marketing and sales experts, executives, finance specialists, and physicians again. And here's a quick look at the full graph in case you want to pause the video and take a look. Now the last question that Daniel would ask people is, did you go to college? And this was just a yes or no answer.  However, surprisingly, 75% of people said yes, they did in fact go to college. And this is consistent with my findings as well, which I've gone over in detail on this channel. 80% of millionaires, 90% of billionaires and 95% of Fortune 500 CEOs have a college degree. So the myth that you should drop out of college if you want to be a millionaire is just that, a myth. But with that being said, I do somewhat agree with the fact that the value of a college degree, especially with how expensive it is, and especially with certain majors that aren't going to help you get a job, has gone down over time. And this is a topic I've gone over quite a bit on my channel, so check out my other videos. Getting a college degree is kind of a double-edged sword. And that's why it's extremely important that you plan things out and pick out the right degree for you. Another thing I want to mention here is every time I meet someone who is a successful business owner in the millionaire range, I like to pick their brain and ask them questions, kind of like Daniel does. And I can tell you that by far the most common story for people who ended up becoming millionaires is they got a normal job, they learned a bunch of skills that were extremely valuable, and then eventually they recognized some kind of opportunity, they went out, they started their own business, and they didn't quit their job right away. What they did is they started their business on the side, they grew it, and businesses take a long time to grow, and most of them end up failing. So if the business failed, they kept their job, and then they tried to start a different business, and if it succeeded, they would see if they can grow it, you know, see if they could get it to maybe $1,000, $2,000, $3,000 a month. Once they get it to a certain point, they would quit their job, go full time in the business, and that's how they became successful. This is the most common path that I've seen, and all the people who want to sell you a $1,997 course don't want you to know this. The reason for that is because they want you to be in a scarcity mindset.    But anyways, I think this whole idea from Daniel Mack is absolutely genius. You should do it in real life yourself. No, obviously be respectful to people, you know, don't bother them, be respectful of their time. But yeah, I think a lot of people are more than happy to share how they found success. And you can see that with these videos. Most of the people are more than happy to help him out. If you haven't done it already, go ahead and gently tap that like button, hit the subscribe button, ring the notification bell, comment down below any thoughts, comments, criticisms, etc that you have on the video. And before you go, don't leave. Check out my other videos right here. I made them just for you.