Hi everyone, can anyone hear me? Okay, I'll shout at you a little bit So first I have to apologize so I'm I'm Revamping this talk. So you're we're doing an experiment you might guinea pigs For a new version of this talk where I'm shifting a little bit the focus I have material for about two hours. I'm showing every other slide and we'll we'll see how we get through that Quick introduction, Daniel Rieck is my name work at reted in the CTO office. I've been in Linux forever Did a startup in the late 90s and I was the PM for rel for you know a long time So a lot of experience with the operating system. I was involved in the Container strategy in reted now I focus more in machine learning, but I'll I'll say how that's relevant The title of the gray beards was nightmare originally was inspired by by the the Debian fork over system D and The what happened is that system D changed how? Linux works right. It's a pretty big move away from Unix tradition to more actively managed operating system and That was for for some people in the Debian community that was too much change And there was a fork of Debian when Debian accepted system I think that that is gone away in the meantime and everyone has accepted to live with system D Some of the things that we are doing to the operating system right now are bigger changes and you know So Linux is changing more than system D did and so you know the the idea was I was this is This is even worse. So let's see what happens if we introduce Kubernetes into the mix So what's a traditional role of the operating system right? They are there two views traditionally that you can find in in in reted definitely or in general right? There is the infrastructure up view. We say oh, it's a hardware centric view right? we have an operating system in order to bring up hardware and It ties you know the operating system. It's very kernel centric, right? That's what talks to the hardware and the purpose of the operating system is to make my hardware work There's another view which is the application centric view which is a top-down view in the user space century It's the oh the operating system is what gives me a runtime environment for my application and Well, of course the truth is it's both right the whole point of an operating system is to allow you to run applications on hardware or something like it, right? It's not hardware any more necessarily It could be a virtual machine But it is there's always an intrinsic conflict between the two views because you know One reason like do I tie my operating system lifecycle to the hardware? Do I tie it to the opera to the application lifecycle and what's the ultimate purpose and You know traditionally You know back in the day This wasn't a big question because you had kind of a vertically integrated Combination of hardware operating system application right if you go back to the main frame. It was basically Here in the IBM model at least to some degree black box application running on leased hardware Right, so you you you had access to an application was very well integrated. It did what it needed to do But you could never take the application to a different piece of hardware Right, or you couldn't change the use of the hardware. You couldn't control how you use the hardware. It was vertically integrated You put data in and you never got your data out, right? It was like it was a one-way Roll and even in Unix so-called open systems. You still had the integration of hardware and operating system Right, so you know if you had a Sun server you had Sun hardware and a Sun operating system And that was true for all the major Unix what linux did the role of linux was to break up this vertical integration Right with linux. We got the ability to choose Hardware that we like from multiple vendors One little sort of and then choose software from an open from an open ecosystem, right and That linux wasn't the only operating system that had some choice windows had some of that Right, but even in windows you had The the you had the integration of one vendor with the operating system Microsoft itself Right, and they controlled to some degree still what you could do, right? And they controlled the tool chains. They controlled what ISVs could do to certain degree linux is the only thing that gave you full at Large you know in the mass market is a full free choice both on the hardware and on the application and provided a binary portability To the degree that's possible Based on the hardware limitation, right? So it you know you can take one One application on the source code level across different hardware architectures on the binary level within one hardware architecture You can take the application run it everywhere And that is still the role of linux today, right? It's a common binary runtime across Across hardware from different vendors across virtualization and across the public cloud That's why by everything the public cloud runs linux basically nowadays, you know because it gives you this abstraction and so You know so that you know that's always been what linux did and you know There's the the traditional of I'm running I'm running one machine, but we're seeing some shifting Paradigm some some big trends, right? And so one one thing we're seeing is that everything is software today, right? There is no business at large that doesn't depend on software one way or the other right even you know any kind of Any kind of what product like you know a car today is a data center on wheels Right, and you don't even have to go to like self-driving cars for that like any car has a whole bunch of computers in there And the functionality of the car is defined by software There is you know in the in the whole Electric car move with Tesla, right? What what? Tesla stated a point is that there Their challenge or their competitive advantage wasn't the car itself Right, it was what's the electrical piece, but that is a software to run the car, right? That's where the differentiation between cars happens nowadays So that shows you like the software is eating the world, right? Everything is software every business is driven Business value is driven by software And we have a fundamental change in how we how how we interact with With things right so It's like when I grew up where we lived in the broadcast model, right? So they were like schedules when things happened, right and you watch TV You know when a show was on right and you listen to radio where you had no control over the schedule, right? so it was like it was a life where You know what you could consume and how you could use things was Relatively like controlled by others outside of your control that has completely changed right nowadays our expectation is that is Things are always available when we want them, right the Epification of the world and that has not stopped it says that that has happened in consumer Things, but it also has happened in the software industry, right and modern Because of everything depending on software Model where like software is a one-way route and doesn't work anymore, right? And most lines of business has their own software developers, right? So they're creating their own software and their expectation is that the value they want To put in their business, you know the business the the requirements to create the business values I want to create in software are always available, right? So it's it's basically an application. It's a consumer centric idea so software Software developer if you go to modern software developer in a line of business who is pressure to create business value for their company Which is not a software company in the traditional sense, right? It's some company Building some consumer device They will not when you go to them and say oh you have to standardize on this Version of a library that has this limited functionality while there is some newer version somewhere available They will not understand what you're trying to do. They will not follow your standardization They're going to try to consume the newest version of something that has the features that enable their business value, right? so that's an important from for the operating system and the software stack right because You know traditionally we tried to standardize things and that has changed fundamentally You know, there are other things You know the complexity of software stacks has grown so we see an aggregation of services rather than monolithic systems And and you know and well what's nice for us is open source has become the defaults for software, right? the The differentiation of proprietors, so they're still proprietors often is of course always going back and forth But at the core of most software systems today is open source. So that's the foundation for for software that's That's become and customers consume it that way and then the stand open source. So we have a lot of Interactive is whether any of some of you probably, you know I'm talking a little bit from a retic point of view because that's my world but you know many of you probably are in in non-software companies and You understand open source you interact with open source even though you're not in a traditional software vendor Right, that's a fundamental change and that's very common in the cut with the customers that that I interact with that They are advanced users of open source that actively contribute to open source actively participate in open source which Which changes the dynamic, right? It's go it goes back to like it. It also makes open source From emulating a proprietary broadcast model of software, right? open source business now has to and and open source systems have to understand this value cycle of interaction with Everyone in the value chain also contributing back to open source There are other changes move to cloud native behavior And I'll talk a little bit about that is a big change deaf ops agility those kind of months are changing how we treat software A big piece You know in all of that is software stack complexity and you know, I put this yesterday from module counts It's from the internet, so it must be true But you know it like even if it's of like, you know, so this is just a number of of projects in common Repository and these could all be for like forks of the same project But that's that's basically what the line of business developer pulls from right and You know if you see like npm is this like explosion, right? and That's that's a really interesting problem, right? So so I think in Fedora we have a Reader of about 30,000 packages and Debian has slightly more So that would be like somewhere somewhere down here, right that what we can package as binary packages So there's no chance that that for example linux distribution is traditional sense can package all of this and Make it available in the way we have we make available the core system, so and so that that's an interesting Change in how software life cycles work And like and so this npm stuff is really interesting. Well, this is not just no jazz that might actually that's a good question So there's a lot of code in there that actually like isn't executed on the server Right these are frameworks that put things that are executed in your web browser. I hope my presentation is basically it's right. It's Google Slides I'm using right and that's basically a JavaScript app that's running in my browser That's pulling code. Well with Google. They have pretty good controls over but You know in the in the traditional linux model what we usually see is that? You you package software and it's on a server and the security problems are confined to the server And which can have significant impact if that's insecure, right? But what we're seeing now is like the way that modern software works is that the browser itself downloads additional software packages when you execute an application and so like so so the whole like concept of Still of like software main stability and and security like has a much bigger implication now it's much more Complex than what we traditionally had to deal with In in traditional software that which is like a server with a terminal or a local machine a workstation And so and you know the I call that app Send up centricity, right? So the one of the problems we're dealing with in linux I know I said the there's like a diminishing return on the you know against the complexity of software stacks I mean because packaging everything in the linux distribution works well for the core system Right, basically the kernel g lip see and the pieces you need virtualization the pieces you need around But it doesn't it can't keep up with the complexity and even the Use and deployment model of these complex application stacks today. It just you can't can't keep up and You you know everyone who develops software will go to the native package format for the software stacks They're using and which in npm with these npm packages then in the browser application It's downloading random stuff sometimes and like there's some it gets really scary what what if you go like to Agency created or line of business created software And you watch what what it's downloading. No one knows what's in there anymore, right? Like so no one can tell you where this stuff is coming from what it does in in most cases, right and You know in go it's all source code and then You know that gets compiled into static binary, so like there's no way to know what's in there like once you get it Right, so they're the interesting problems. They were where like what we did traditionally was packaging Code in an rpm for binary distribution, right? Which was a huge step forward against the fragility of compiling things on that each machine But it it's not sufficient anymore to keep up with this because you can't capture all of it another big change is That you know In the modern world like it the cluster is a computer I see the whole concept of deploying an individual machine is not a reality anymore. No one no one runs one machine, right if Sometimes you still look at an individual machine, but that's usually a virtual machine Right. I'm running on a cluster of virtual machines or in the cloud in most cases You basically look at services that that that You provide so it's it's basically One service that gets deployed somewhere that service might be implemented as a VM or as a container progressively it's in containers and with you know with in Linux, that's where where Kubernetes comes in right so the Problem we have You know traditionally everything we have done in the Linux distribution is confined to one machine, right? That's how we looked at it and so You you had the special case of clustering so so system. These are single machine Orchestrators that takes care of your machine Really well, but then you know when you cluster them that becomes a special case and you have some failover But there's nothing in there that takes care of like how how do services move? Across multiple machines. How do you get connectivity, right? And that's where Kubernetes comes in and Kubernetes now has become the default orchestrator for services on Linux That run run in in clustered services The It inherits the the features from Docker and or like Docker like OCI containers Which I'm sorry, I have to look at the time All right, so so it inherit its built on The container concept those introduced by Docker first standard as an OCI now implemented with potman and and and cryo in in red heads stack and What it does basically you can look at it when we sometimes call it a meta kernel, which of course is More meta meta system D right it it orchestrates services across Multiple machines and manages the cluster and then on top of that it services that that orchestrate how you talk to the service direction service mesh how calls get routed in that server and What's What's interesting is that is that it Does two things so the container concept encapsulates the dependency stack of an application Into a namespace Which removes one of the biggest problems that that you have in the traditional model of of an operating system, which you know You can call dependency hell, right? So if you have in the traditional single node operating system, you have one user space, right? And when you want to install things they have to They cannot conflict in that user space, right? That's how we our PM works, right? I PM solves that with its dependency management But when you go into this model of many different package sources that gets increasingly complicated, right most of them have some way of isolating like a virtual environments, but The dependencies on the underlying system then still need to be Consistent or interdependent between these systems and so traditionally that has been solved by running thing in virtual machine So you could do one virtual machine per service, but that's relatively expensive So containers of a very nice light-weight way to give you a user space isolation to manage one service Dependent binary dependency stack in a nice way and then Kubernetes gives you a way to orchestrate these together to serve applications out of a cluster, right so Bunch of things happening here right move to from one server to a cluster of machines and from kind of the binary like one application binary interface to the operating system to abstracting that and splitting the operating system into the hardware piece and An application piece that lives inside the container and and can move around so so the Equivalent to the earlier picture, right traditional in the traditional server We had red head enterprise Linux between the application and the infrastructure in the hardware or the virtual machine and now in the new app-centric platform we have Redhead OpenShift, which includes enterprise Linux, right? That's still that's still the foundation for that So it's enter president press Kubernetes plus a bunch of other servers that becomes a new abstraction layer for applications across the different footprints of infrastructure now This is happening in the context of cloud, right? And I talked a little bit earlier how cloud is changing how we do software, so to go a bit a bit deeper so the cloud It's not only that That clusters are the default the default for most large customers today are cloud deployments for new projects If you're not in an regulated industry Then you default for new projects to the cloud in many areas and There are a bunch of reasons for that right one reason is OPEX versus capex, which I think is The least important one. It's it's like it's from a budgeting point of view It's easier to go to the cloud because you don't have to buy hardware with a long commitment The key points are elasticity and self-serve you can just go there and just do it It's much easier to get the cloud instance than to go to internal process even to get an internal VM in many companies I just go there with your credit card and then the companies have to deal with that You can move up and down as you need right if you buy hard with the traditions and moving down isn't really an option You already bought the hardware cloud gives you the ability to reduce and be elastic the cloud has managed to Establish data aggregation all your data and and progressively today You know, it's a separate topic that that we have all thread on is you know the concept of intelligent applications applications that Will bend and define that for me applications that are learned from data and improve their behavior based on data Right those applications need to live with data or need data access Cloud allows you to aggregate data very easily and so intelligent application have a high affinity to cloud They are the service of cloud provides great integration of different services They often have better security right so you can trust that Amazon's infrastructure is very secure It's very expensive if you run your own infrastructure in a public-facing way to keep it as secure as they will keep it For a bunch of reasons It's their core business, it's probably not your core business And they have operational excellence right they they because it's a core business to operate that cloud They know how to operate that cloud no individual company can compete with that directly They run everything as a service right so they provide pre defined services They are now moving on prems. It was a hybrid model So even if you have things there you say, oh, I need to keep certain data or certain processing locally for latency for example, they have For industrial IOT for example, they have on-prem Satellites now that you put in your in your factory and they're basically part of the cloud But local data is processed locally so they move hybrid which you know removes a lot Is very attractive for large users that needs the data aggregation to central views integration and You know the key from my point of view the key piece what they're doing is the service abstraction and time to value in the public cloud right that you interact not you You can choose your entry level right traditionally like it was VMs, but now it's services All right, so you aggregate your data your application from a bunch of predefined services And then you add the piece that that's different for you So you can focus purely on your own differentiation And you don't have to build all the other pieces right and and that's You know you can focus on your core value and I think you know the concept that like shows that the most is function as a service so Amazon Lambda Where the platform like you basically only deploy code snippets in a predefined framework You know usually for things that act on triggers on data triggers and messaging triggers So you deploy only like you basically you don't have to you don't even have to deal with the operating system Was a software stack anymore as a runtime right? It's all predefined That way abstracted and then it talks to it through a messaging bus through some other predefined applications So you know that the the that gives you this level of service abstraction You don't need to be an expert in any of these other things right you don't need to storage admin You don't need an DBA you don't need To you don't need sys admins to install messaging you can just use that now in an environment where your data is already there and They give you this on-premise thing to aggregate your data and and push some of these decisions locally. That's a very compelling Value proposition, right and and that has changed how we look at software in general right like so You know traditionally it was what we call enterprise software So someone writes software that someone else then Deploys and the third person uses right now with with you know everything being software line of business having their own developers we are do people just writing applications on top of a of a stack that is completely controlled and provided by third party And and so so there's a big downside to the whole thing And that is the lock-in. It's black box services. You have life cycle dependency data gravity You can't get your data out once it's there or it's very expensive to get it out You have no reproducibility because you don't know what these black box services actually do right You can't look inside you if they change how it behaves You you have no control In you know, it's really bad for open source and you know probably a lot of you have have seen some of the Fights of our licensing where like open source project or the companies that built an open source project Are changing their licenses to make it harder for the cloud vendors to offer their software as a service Because there is no room for ISVs anymore in this model right Amazon It's Amazon basically competes with the whole rest of the software industry at the end and and so You know, I said it's bad for open source. So who cares right? Why why does it matter if it's good for customers? Well, the problem is that they're there good reasons why you want open source, right? It's a transparency It's long term strategic independence. It's innovation, right? If you go into monolithic system You basically they basically recreated the mainframe, right? It's black box services running on leased hardware That's that's what they did. It has all the advantages that the that in the early days of computing the mainframe had and You know, you can just use it. You don't have to deal with it. It's very secure very reliable But it has also all the downsides. You're totally locked in you can't innovate you can't differentiate you can't do things differently and You basically it becomes more expensive because you're in it in this dependency, right? So so And and you know a big conflict is like customers who want to have strategic independence We don't want to buy into this monolithic system and a choice of like four big vendors that are relevant, right right now For all kinds of reasons And the software vendors that are not one of the four Big ones have a shared interest to find an alternative to them And what you know, what are the challenges for that alternative? The big challenges is is the service abstraction time-to-value. That's the first one right if you like if if I go to a customer and say, okay You want you know, I want I want to enable you to run things offline on premise, right? Not in the cloud a service. Well, you need to install Linux and then you need to install Your database and then you need to install this and this and this and I end up with this huge software stack that they have to maintain They have to become experts in You know, they need to deal with storage. They need to deal with hardware. That's sometimes that's a very tough Value proposition, right? The sheer number of servers are easily available, right? I can just click a button and I can just deploy a service and I it covers my whole application architecture That's late. That's a lot of work if you want to do that in the traditional model of you know, how we bought software traditionally right Because you have to deal with five different vendors to get to the same in Amazon It's just one vendor you need to do it. So that you know, that's that's a problem Application portability right like if you go like can I can my application? Move between the the cloud right so so one of the big advantages of a third-party Model over a single cloud vendor is that if you know If you can take your application move from one cloud to the other cloud or on prem Then you immediately are in a better negotiation position against the one vendor, right? So we need to provide Application portability and finally we need to have the operational excellence We need to make the operational excellence available to customers that Amazon provides as a benchmark was the other club I don't want to single them out But but we need to make that available in a model where as the average IT company Can can or the average IT department in an average company can operate their IT in a Competitive manner right which is really a hard challenge because you know The expertise is never going to be there And in a way, you know, you can look at redhead's traditional business model as making Make using Linux to allow companies who couldn't Who couldn't afford a Sun server to run the IT like they could right that was in the early days was our business model So it was about enabling people at a lower cost on PC hardware to do the same things that that you know the pros did in in with Unix service You know at the end like now our challenge if we need to enable the average IT department to run their IT like Amazon does right So how do we do that? so we think that Kubernetes is The answer here right as I explained earlier Kubernetes is a service orchestrator in a cluster it provides you OpenShift is redhead's distribution of Kubernetes, so I use in In synonym here, although like technically Kubernetes is kind of the kernel of OpenShift and then the other services around so OpenShift gives you you know and extends Linux to give you kind of a service application-centric deployment model across clusters, right? abstracts from the underlying hardware it takes care of connectivity of Applications in that cluster right between the application how you get to the application it has application lifecycle management So it has it's a cloud platform in that sense that it gives you this Elasticity and service-centric or application-centric view Applications are portable because they're defined independently of the underlying infrastructure right and even So the portability can happen again like in traditional Linux on two levels Like if you have the same hardware architecture It will inherit the binary compatibility and you can take your your multi-service application definition and deploy it in parallel to multiple places I think we you know we had a workshop yesterday to deploy a whole the data hub data aggregation and machine learning stack and It's basically a one Click deployment of like if we have 13 service stack So you get this Abstraction that's possible as OpenShift and the portability of the full service where you don't have to be an expert on every individual piece and the the interesting piece here is that like so you get these pre-package services, right? So we have the ability for third parties and we we work with many third parties that define applications themselves consisting of multiple services and you can put them up and They are the end customer can then consume them in a way that's Compatible or comparable to how you consume things in the public cloud, but you can take them wherever you want, right? so you you get basically instead of Having documentation how to install things you get the higher aggregation of the full orchestration model down to like pre-encoded Scaling rules and things like that in the cluster as an application you can provide, right? So that that gives us the ability to create the integration and the time to value that you see in the public cloud independent of where you run it And the key here is the ecosystem obviously, right? So you need You need pre-package services Available from many vendors in order to have the same number of service available that you would have in the public cloud so we End up with kind of two ecosystems here one is one is component level It's a download to to build so as a developer I want to download some Java package or an npm and I've write my code and that goes into kind of the application life-cycle management But in many ways in many Scenarios, I just don't I just want to download a predefined service that I just want to operate without having to be an Expert in doing it like I want a database, right? I want I Want storage? I want a message bus. I want Kafka in there, right? It was one of the things we deployed in that service. So So it's it's pre-packaged services that go beyond the traditional binary to the full Spec of the operational parameters of that service and make it easily consumable like you would in the public cloud And and so this gives you it's basically an open alternative to the same model And it gives you a standardized operational model Which you know, so so you know at this point I should have put checkbox in there, right? So so at this point what we solved is we've solved the service abstraction, right? We can package full services makes him consumable very easily We manage the time to value there What we haven't managed yet is the operational excellence, right? And and the standardization of operation that you see in Amazon, right? If you follow the Amazon path, you know you're following an industry best practice and You know, they're like interesting Discussion is like, you know it in open trip. We have a trend to immutable infrastructure for example. So right now you can Right now you can take you can install rel and you can install open shift on top There is a discussion whether that should still be supported going forward or whether you should just have in have a core as immutable host and you can't even lock into that anymore because Usually when you want to log into your host on an open shifting you want to do some things That's not the best practice right because the best practice is encapsulated in what redhead provides you It's it's a debate, but You know in Amazon yet if you go the Amazon way you automatically follow a best practice and We are trying we think there is Kubernetes we can create a standardized operational environment that allows you to then You know Just the same things operate in the same way independent where you run them, right with the operator model We encapsulating operational aspects of an application with the application. So For example, if I'm I'm I'm a database Provider, I ship a database to my customer. I've write an operator for that that operator encapsulates Upgrade procedures and backup procedures for that database right so So that the customer doesn't need to create custom backup rules for example a custom upgrade It's like, you know in windows. There always was a service called HSS VSS volume of shadow. So it's an interesting concept, right? I used to work at a backup company for brief time and on windows You could when you wanted to back up a windows instance running Microsoft SQL Server in a VM You basically had the thing that just said oh, I want to back that up and then VM Where you know in the VM there thing would like go to the service in windows They are you going to be backed up and the windows so VSS would tell the SQL server. Oh, you're going to be backed up The SQL server would flush its buffers pause VMware would take a snapshot of the machine of the data and then say I'm done with the snapshot and The service would restart Right, so you didn't like there was no custom script in that it just was in the system Look at operators as a way to standardize that for services as they do operators to other things But that's one example right the application vendor will set the procedure how to do the backup and that becomes basically a standardized model so that You know if you want to back up your data The system will take care of making sure that it's clean at the point where you can then take the snapshot And so that's what we mean with standardized operation mode. There's another part of that. It's It's I have two more slides so and then we go to questions so So we're standardizing metrics with Prometheus and famous and You know for many things For many things that that you're trying to do in these clusters You want to presently be able to Drive application behavior based on metrics for the full cluster, right? And so that's another so so standardization of how we meter things how we aggregate locks how we Look at for example performance is another aspect of this standardization of the operation environment And then with Kubernetes you already have a standard way to enact action, right? So You know so there are there's an API for everything for every action And so you can automate things on top of these air so we have the way to have a standardized model on how to do actions right gather data and then do actions and so on top of that what Retta is doing we have a service called insights which basically Aggregates information from multiple customers learns from it and then gives guidance to customers based on that So today it primarily Does it for configuration data? So it will tell you whether the configuration you have is bad, right? It's just wrong, you know, it's a or suboptimal configuration or it can oh if it's this configuration It can relate to issues we have seen with other customers So, you know customers with this specific configuration ran into this specific problem and will warn you Proactively about that our last summit we demoed the service where it would look at some of your configuration to say Oh, you're an outlier or you know, 99% of our customers have this configured differently and by the way your performance is off, too Right, and we recommend you change this setting, right? So the idea behind that is that So in a way like if you look what redhead's role traditionally is in open source and you know similar to other open source companies is that we We aggregate knowledge in how to maintain the software provides a stability aggregate knowledge on how to fix the software and So that if you run into a problem if you're a redhead customer you call redhead and you have a high trust level that either We already have seen the problem with another customer so we can address it for you. You know, we may be already Provided a fix if it's a software problem or we are able to identify the problem and then fix this and then So, you know, it's already solved for the next person and basically that's always been the model And now we are taking this model and automating it by aggregating Observations from customers learning from then and providing guidance back So that's the idea behind that is that we can provide some of the operational experience, right? The observational excellence that you see with these big With the big cloud providers or very large ITD partners and we can make that a shared experience for our customer base and grow grow that and so the system will Help you it will have basically a guidance system to help you Achieve the same operational excellence without having to have the expertise always in-house and and that's you know, usually correct performance reliability stability and Correct behavior of the application reliability stability security and performance, right? That's the things you care about and that are all things that we can We can solve to largely agree with this guidance system Now the second piece the next step is self-driving clusters, right? So so putting AI into So into this system, right? So the the first guidance model is possible without the standardization Of course, but it becomes much more effective if you have a standardized operational environment because you So if you if you so insights and traditional rel is Helpful, but of course limited because each deployment is too different So you cannot always make the same conclusions when you go to the model where Where you have an established operational environment with immutable infrastructure, right? Every host is the same every open shift is the same their parameters in different But they are observable and they're in a controlled environment suddenly we can We have a high probability it's something we observe in one customer applies to the other customer as well, right? Or we can measure the differences Now we with a standardized way to measure all of this and the standardized way to act in an act change We can also try to automate that change right the action and enable us to do self-driving cluster So we put learn, you know, we would predict a predictive statistical systems and machine learning models in there to learn from behavior and then automatically react to it And it starts out very simple right with just anomaly detection and things like that So it says oh, you know something unusual happened, right or alert filtering, right? But it progressively will get more and more interesting right from predicting behavior or optimizing Optimizing placement We will be able to for example direct service calls, right? So you can You can If you have we have the standardized monitoring so we can monitor the full stack right up to an application now We can measure the efficiency of an application down to how much CPU heat does it generate, right? So I met someone who wants to do that for machine learning itself, right? So how do you know a machine learning model is good or not? You can of course, you know, they're they're hard factors like how well does it predict, right? How correct is The outcome, but you can also look oh one uses much more energies than the other one So maybe we go with the one that loses uses less energy because that's a factor I care about So I can monitor things old stack or service rooting, you know, I can see machine load and I can push calls For my application to another note because I have You have all the knobs to do that and we can do that intelligently with you know learning from the behavior of the specific environment So that's the ideas behind that So, you know, basically in many ways takes a concept of self-driving car and apply to a much simpler problem self-driving clusters Which will help self-driving cars because you know, it's a data center. So you don't want to suss admin in the trunk, right? Um, okay, it's a cheap joke. Uh, yeah, so so right? So the you know The the storyline, you know summarizes like so Linux's historic role is to be this neutral runtime, right? Open shift does that for the modern cloud world as an extension of Linux The key values that drive cloud adoption are service abstraction time to value and operational operational excellence and the availability of services and we think that we can get To place with a hybrid ecosystem to provide the same And then with the standard as operational model we see there Drive the upper you know provides the operational excellence through Aggregate data aggregation observation and guidance services and putting intelligence in the platform itself And that's possible like because you know because of Linux Kubernetes And then adding machine learning to it any questions Take a very rude But there's a so we joke right like so there's a the head is like so We should change our pricing model and that is a joke right? I mean the city office disclaimer nothing I say can constitute to a product announcement and nothing I say anything like that But but you know someone had the idea why well we should like charge more if you want the root account on your on your hosts Because it immediately means you're doing something. That's not standard. I think it's a cultural change, right? But I think I Really like so I really see the analogy to self-driving cars right which and My you know, I my wife has it has a model Tesla Model 3 that from time to time. I'm allowed to drive And it's really fun So I'm like on the highway in autopilot mode. It has like it's like it's not fully autonomous, but You know, I'm not steering the car and I'm driving well in Massachusetts Let's say 65 on the highway I So 65 miles per hour. I can't you know, it's a hundred kilometers per hour or something something above 100 110 or so and and the car is steering on its own in full traffic so and and and that's that's a behavioral change But You know, it's just the you know the car like I don't have to worry about Anything right especially like in stop and go travel like or that have it like it just Breaks for me and it's much better that than I will ever be it has a lighter and radar Future that look like three cars ahead So it breaks already before I realized that the cars before me are breaking and I think I think once we have These guidance systems in place and and we have automatic Optimization it's gonna be a self-controlling model There's a talk. I think that was yesterday on soft. Was it yesterday? Yeah, so We out of time. So last sentence. So we we are looking at that also for content guys So so this was the operational side, but when you write your own software you you use all these stack components from somewhere and It's the same problem, right? How do you know which version of what will work with which version of what and So we have a guidance model for that as well And we think that it will basically lead people to to the mainstream, right? People will go to the best practice as soon as they have a realistic ability to learn what the best practice is And then people will break out of that only if that's a real good reason to do so All right, I hope this was interesting


Hi, my name's Corey, I'm going to be talking about the title of the talk for this was Insecure by Law, but we're going to be talking about hacking heavy trucks and the dongles and whatnot that are associated with each. I work as a senior security researcher at Iowa Active and prior to this I did some auto dongle work under a company called Digital Bond doing the progressive dongle research. Basically for me what gets me hot about cars is when we start connecting third party after market stuff and expanding the tax surface and making it more interesting because cars by themselves are sort of insecure by design in my opinion and we're not really going to fix that. We'll get into that in just a minute. So basic overview what we're going to talk about is we're going to go through a real quick can primer. We're going to talk about heavy trucking because that's important to this particular research understanding the context of why this matters. Because heavy trucking even though a lot of the technologies are the same, fundamentally and culturally and its impact on the world is a little bit different. And then we'll go into the analysis of some of the logging devices that we got. Talk about this cool little event that happened about three weeks ago called the Cyber Truck Challenge and then we'll go through the summary of what came up. So back to sort of what I had jumped in earlier when it comes to car hacking if you haven't spent time sitting down in the village out there or looking at different stuff sending some can bus messages and that kind of thing I encourage you to do so. But when we talk about car hacking in my opinion it's a little bit interesting and kind of boring at the same time because when you're talking about can bus and then vehicle messages you're really talking about an eight byte payload maximum and then an arbitration ID. So you've got an address and a payload and there's no authentication. There's no anything on there. It's just a basic bus. So when you talk about car hacking and you're looking for controlling a vehicle with these types of messages with different auto manufacturers they use different IDs to mean the same thing. So for a GM for example the ID that shows you what the RPM is on the bus that transmits what the current RPM of the engine is might be different than another one. And then it might be a different offset within that payload too. So the address might be 310 and then the RPMs might be the last two bytes. First two bytes are some kind of flag set up for an example. And when it comes to car hacking these are the kinds of messages that people are usually talking about when you're doing reverse engineering of vehicles. So when Charlie and Chris were doing stuff when other people have gone through and published car hacking things mostly the hard problem when it comes to car hacking is figuring out what messages do what. But when I look at it I mean that's very interesting and that is a hard problem but it's kind of boring because you're not going to ever fix this. And it's insecure by design is the term that I'm borrowing from industrial control systems which is where I came from before getting into auto. And the realization that this is a broken protocol and we're not going to be able to put really security on top of it. So when it comes to that I get interested when we add on a different attack surface. So when we're putting a dongle in between this insecure bus and the internet that's when I'm really like okay well we should take this seriously because yes this is insecure by itself but it's not until you start connecting it. Same thing with power plants which is where I was before. When we started connecting those and people want to control and monitor their plants from their living room on a Saturday nobody wants to drive 40 miles out to a substation in the middle of nowhere to do something when you can use remote connectivity to do it. So the advantages are there same thing with auto we want to get as much telematics off of an engine as we can so we can do things like predictive failure analysis like really cool shit but that also means that we're taking these broken fundamentally broken things and then connecting them to the internet. And so that's where the attack surface increases and where it gets pretty interesting just from my perspective. So we're going to depart all that basically what I said was more on the passenger vehicle side and now we're going to talk about trucks because trucks they do use canvas underneath them but it works quite a bit differently. So the United States for one thing because that's what I have statistics on heavily heavily depends on the trucking industry. The idiom in the trucking industry is if you bought it a truck brought it. And if the trucking industry goes down there are three days at one day before you can no longer get fuel at gas stations. Three days before you can no longer get food and items off of your grocery store department store Walmart shelves. So this is critical infrastructure to the United States of America. The trucking industry is significantly important when it comes to just keeping our lives operating. There are the other things to note about the trucking industry is there are regulations that prevent drivers from over exerting themselves driving too long. And all that is recorded in logbooks. So a driver has to record how long they were driving when they took a lunch that kind of a thing. And so you're limited to make sure that everybody's safe. Right. There's there's a good reason that we have those regulations in place. But the problem is that checking logbooks is fairly tedious and people can lie. We don't want we don't want those truckers lying to us in our logbooks. We want to make sure that they're obeying all the rules because naturally you're incentivized to move as much product as possible. Maximize your profits that kind of thing or sometimes you just want to get home earlier than than before. And so so when people are going around our regulation the natural solution is to add more regulation. It's like brute force right. If brute force isn't working you're not using enough. If if regulation isn't working we're just going to add more. And that's how we're going to solve this problem. So that's what they did. And there are other reasons for it not just solving the the logbook fraud or or mis misuse problem. But the government mandated these electronic logging devices. So there was a there was a law pass that said by December 2017 all trucks have to have this electronic logging device and that's what we're going to use to monitor and make sure that everybody's obeying the regulations and and get some extra gravy on top in terms of what kind of analytics we can do on that data. And it is 444 pages long which is awesome. And that's the kind of thing that somebody who's going to manufacture these dongles or use the dongles or whatnot they have to go through sort through that and make sure that they're compliant. So uh in looking through the 444 pages I'm just going to highlight a couple awesome gems for you. So uh I went to the table contest I'm like uh quality assurance alright that's probably where security lives. Uh and it's uh it's right here the entirety of it it says insert quality assurance section here. And this is the finalized uh out these are the guidelines uh in those 444 pages. So the quality assurance section is is uh non-existent uh and that's where I mean that's uh security is sort of glorified QA really. Um making sure that the system does what it's supposed to do and that it doesn't do anything else that's what security is all about. So uh to have these sections empty is uh a little concerning. Uh but there is some good in the 444 pages um if anybody's ever worked in the government or uh known anybody putting these documents together can be uh a process um but uh this particular section uh I think someone was in charge of writing this particular section and they maybe knew a thing or two. Um but under so this section is under certain circumstances like if there's an officer at the vehicle and uh and there's Wi-Fi on but there's not cellular and not Bluetooth and it's the Mars is in the 4th quadrant of the universe then you apply this section and this is the only mention of uh TLS uh or an SSL uh any kind of secure communications. So in this like this one section then we're talking about it. So it's but it's in the document so it's it's it's there somewhere um but then when it comes to actually figuring out how uh whether a device implements this is is trouble. So let's say you're a donger manufacturer and you want to get in on this sweet cheddar that's coming out the government's mandating something so there's going to be guaranteed money there so let's uh let's go ahead and even though we're uh we make macaroni and cheese for our business I think we can probably get in on this get some of that awesome awesome money coming out. So you say 444 pages man how are we going to make a dongle that uh meets all the requirements in there but it's okay you don't have to worry uh because the only requirement is that you have self-certification. So if you self-certify that you meet all the requirements then you're good uh then everything's great uh you're awesome you're going to start making some money you're going to get on the list of all the approved dongles uh and be good to go. And the list is maintained um by the FMCSA uh it is uh they don't make any uh obviously the government doesn't make any charge that uh that you actually meet the requirements um this is the reason for self-certification. If you if you've ever been in that government uh then you totally understand why they did that because that takes the liability off of you and puts it on uh the people who make the dongle uh but there are over 70 uh at the time of the screenshot I'm sure it's increased uh probably uh lately uh but there's a lot of them and so you can just go and uh and get any of these and that will do your electronic logging for you. Um what they end up tend what they tend to look like is something like this um so this one is uh is uh available at Walmart. So as a sort of consequence to this mandate um you've got guaranteed money there uh so you've got a bunch of people entering the market with stuff and as consequence then the uh the barrier goes real low in terms of how commoditized the hardware is how commoditized the solution is.  So it's heavily heavily commoditized uh there's a lot of players in the space and the only requirement is that they're self certified. Um but as we go forward it is actually still under contention there are people who have uh some altruistic reasons uh but um this bill was just put into I had to just add the slide because it was just added uh two weeks ago I think um uh representative from Texas put this bill forward to try and kick back the adoption uh another two years so they want to extend it. Um they cited uh cybersecurity vulnerabilities or cybersecurity security concerns as one of the reasons so that's significant um at least we're being used as a scapegoat even if even if it's not actually true but hey either way uh somebody's gonna care about cybersecurity hopefully as we go forward or maybe nothing will change and then it's just gonna be two years before they're required to adopt it. I don't know uh depending on how cynical you are you may fall on one side or another uh but the other reason is there's logistical concerns uh because there are some uh like livestock uh get exemptions to the uh driving rules and that kind of a thing so they're trying to make sure that their business is protected by this ELD thing and uh I suspect it's actually less about the cybersecurity but so uh after we saw all this and we get excited like alright there's a bunch from let's go shopping so we did and we picked up uh so uh as a caveat this research if you are a POC or GTFO person then you can probably GTFO because uh this I'm not I'm not dropping a full uh payload uh weaponized ODAY or anything right now the whole point of this research was a uh breadth first not depth first and uh to try and figure out uh how to hack a particular dongle and come up with an exploit with uh metasploit module that's not what the point was the point was to get a few of these things and see what the general security uh setup was with these commoditized dongles uh because especially when there's 70 of them I didn't want to get in the position where we got one of them and that happened to be the bad one but the rest of them were good you know what I'm saying so we did a breadth first search of the dongles to figure out what's going on so we got uh ELDs from three different manufacturers again these are consumer off the shelf you just go pick them up or you order them directly from them uh the suppliers were chosen at random so uh but again being a breadth first like this was more to get a sense of the industry and so I'm not trying to pick on any given supplier uh in any of this so please as another caveat do not take this to say if I have a supplier's name or I'm holding one right here that does not mean that these guys are the only bad guys um but I will give you a particular tip about this one because it's cool uh so going forward um we got them we ordered them uh and we started off great uh with a with an awesome start uh so we got some emails to sign up because almost all of these have companion apps or back end stuff um and so it's like uh here you know uh address and they put the password in a get parameter like that's a small vulnerability it's not really big deal people do it all the time but that was the first experience with any of these so uh so it was sort of like alright um the uh the bar for the for the average in terms of what I've grown to expect out of hardware stuff is pretty low uh and so far it looks like we're par for the course uh just starting out so we got them and then we we began to do some hardware analysis uh so that's this this one taken apart um and it's actually just a very simple one board with the daughter uh cellular board but it comes down to an arm processor a uh some spy flash and a can transceiver and that's it basically we're talking very simple very minimal how cheap can we make this thing how fast can we get it out to market this is what we're gonna do and and that's what it comes down to and this one also is equipped with Bluetooth and then has a cellular daughter board as well and you can see uh if anybody does any hardware hacking there's some interesting pads there uh that were fun to play with um the connector which is not exposed you'll see that that connector over the top not exposed uh here um but if you take the top of this thing off which is kind of what I was hoping somebody had a pin for but that's alright uh and then this is uh another example of one similar type thing you've got your processor you've got your spy flash you've got your can transceiver and then a bunch of power regulation and that kind of a thing and usually with some kind of communications uh chip uh on top that they speak with over uh over serial line in order to do the communications uh and then another one uh looks like this this one is interesting because this one has a screen uh so it's a little bit more complicated so there's uh some other cool stuff uh playing with it uh this one's also interesting because this one doesn't plug into the truck so uh spoiler alert this one's the most secure um as in terms of in terms of whether you're affecting the truck or not but uh unfortunately you can't legally use this one because legally uh as the document says it's required to plug into the truck to get information out of the truck so um unfortunately uh but they're self certified so maybe it's fine uh and this is similar so this underneath there's a lot more components in here because we've got a screen and a printer on it uh which was cool so this one you if you want the full logbook out then you go and it prints it um and then you can hand that to your regulation uh officer this is another one uh another screen the same one ribbon cable that goes out and that kind of thing and the uh sim there for cellular back end uh communications but again this one uh this one isn't hitting the truck uh directly whereas this one plugs in uh the other one that we looked at had a cable that you then plug in and then you feed the cable to to the unit so after doing some harbor analysis uh pulling flash uh grabbing uh seeing if we could get the debug going that kind of thing then you jump into the software and of course the first thing that you do is the best hacker tool ever strings is uh the first step that I always do whenever uh whenever I'm gonna look at new firmware or anything because interesting things pop out of there um so like here we've got this is sort of purposely uh obtuse or difficult to read um but uh you've got full path to somebody's uh uh directory structure for uh how they were building out the project. His name is Jenkins uh which is an awesome name um so I imagine like they're sitting around and the team's like God Jenkins we need to get this pushed out right away and so okay. Oh okay. Alright.  I was unfamiliar with that. I was unfamiliar with that. You foil it. Sorry. Can I choose to disbelieve the truth and uh imagine that I'm just kidding no I appreciate that I didn't know that actually. Oh okay okay. Okay cool. So this might be some of that same no this one was from a different unit so this is not uh this is not the one that was built out of Jenkins I don't believe but uh but then we're looking at the bluetooth uh pairing stack for uh for this other unit just doing some basic reverse engineering um but what it comes down to is you're seeing like again this was a breadth research so it was mostly like how are they using like stir copy uh insecure band functions that we shouldn't be using anymore. Let's use memory safe versions instead that kind of a thing. So basic software analysis we weren't again we weren't trying to get POC uh or anything like that on there. So um you don't have to just take my word for it either uh when it comes to what the uh security opinion of these devices are um so like I mentioned about three maybe it was four weeks ago now since we've been in Vegas for a week uh the they had a cyber security challenge out in Detroit uh where the uh Army Tardoc uh sponsored um this event uh where they brought students from 12 different universities 12 is the correct number I believe universities and community colleges uh so there was uh two days of class and then two days of hands on. So we had a couple trucks that were there the students actually got to plug in and so it was kind of like the car hacking village except uh in a more controlled environment um and and with a little bit of hands on uh class learning and that kind of a thing as well. And none of these uh well I won't say none but most of the students had never done any hardware hacking had never had any exposure or anything like that uh any other stuff and so they got to come in and play with things they had a bunch of they had like multi meters uh soldering irons um uh chip readers that kind of thing they're available for them to use and it was a really cool event uh but if you know anything about the trucking industry uh it's an interesting culture and they didn't want to talk about this event in the public very much which I think is unfortunate because it's really cool and and they're sort of going forward in terms of cyber security thinking about it trying to encourage people to look at stuff uh but they they didn't want to talk about it too much but I found this article I don't know if it was authorized or not uh but there was this article in the Detroit Free Press about it um and it was a really cool event but the so the students uh tore apart this particular one that I'm talking about here um extracted the so this is in one day actually so for only one day that they were that they spent down looking at it and I'll give a shout out to Grim Mitch from Grim was one of the instructors who sort of walked them through and uh encouraged them as they were learning about this uh so shout out to them um but this is what they did they extract the firmware over single wire debug uh because the debug interface is wide open on this as it chips out from the factory um there's lack of encryption in any of the firmware or communications uh they were able to do basic dynamic analysis uh using GDB um but ultimately the student opinion of the device security is low to nonexistent unfortunately but uh so the reason that I said I was gonna tell you something cool about this sort of ties into here because this is uh between I think it's around 150 bucks for this thing um but there are other like toolkits to help you do to get into truck hacking or interfaces to communicate on the bus of the truck um and some of those go for thousands of dollars um but if you just buy one of these uh then you've got bluetooth access to the truck uh to the truck bus for pretty cheap uh really I when I when I'm working on it I prefer a beagle bone with a canvas cape and then manually controlling stuff that way but uh this is really a great alternative because it's open you can flash your own firmware to it it's really really straightforward to get your own thing on there so if you buy one of these it can become your truck hacking interface instead of buying some expensive thing uh otherwise or if you don't want to mess with wires and and getting connectors and stuff like that if you're doing a beagle bone then this is a great compact package uh for all of that so uh I'm not sure that I would recommend it as an ELD but as a car hacking device is pretty good um so the the conclusions uh security overview uh again the bar is fairly low um and uh everything was basically par for the course uh devices were shipping with debug enabled uh with the firmware was easily accessible uh you can just grab it straight off the chip um there's lots of use of banned functions within the firmware uh general insecure coding practices uh that kind of a thing which is uh unfortunately expected at this point um and I think anybody in harbor hacking or IOT or anybody who does any embedded security sort of that that's a pretty common thing but it's a little bit um unfortunate uh when we talk about cause there are well I'll talk about it but there are companies who who uh were sort of in this business before the ELD mandate came out and then uh and then like everybody entered the thing and then they kind of get lost in the noise even though they're making significant effort and spending money on cyber security we'll talk about that when we when we get to impact impact at the end uh but what what is the potential impact uh if we've got a vulnerable device here uh we know that it uses insecure coding practices uh and if we can access this uh from the internet or from bluetooth uh whether you consider bluetooth remote attack or physical attack uh given the short ranges is up to you uh I think some people on either side of that but uh if you've got a remotely exploitable device and you're plugging it directly into the truck uh that that does matter and for the the truck in companies uh are doing work to try and isolate this threat uh because one of the things that you can do on the buses is uh oh I meant to mention this at the beginning but if if you've ever done any OEM uh car hacking you know how the the IDs are different with uh a GM versus a Ford versus whatever uh in trucking it's all standardized so you don't have to do any of the reverse engineering or very little of the reverse engineering to figure out what ID does what so there's a straight up document that'll tell you if you want to request torque this is the message that you send so that makes it easier when uh when crafting payloads to potentially exploit it and so when you're talking about the if you're a heavy truck manufacturer you want to probably isolate that if possible not allow a device that's plugged into the diagnostics port to be able to send that message but the problem is that there are uh things that plug in that that legitimately need to send that message uh that are interesting so and this isn't just uh the trucking industry when you talk about like emergency response vehicles I used to work as a volunteer firefighter and uh we would be on the in the city department with the engines and you're going to be sending those request torque messages when you need to use your pump and spray uh spray your hose to fight a fire you get a torque up the engine so it's some of that same kind of thing uh as we're getting into more sophisticated uh trucks and we're getting more drive by wire autonomous that kind of a thing it's going to be more significant and we need to make sure that we're isolating these out uh but critical uh heavy trucking is critical infrastructure so uh it in my opinion it's a little bit more significant and you approach the problem slightly differently than your average passenger vehicle even though passenger vehicles are also super cool because you're talking about uh multi-thousand pound robots uh that are under control of an attacker which is significant because when we start to get to the physical side where cyber security meets uh physical and you do the cyber physical if anybody's got bingo cards I can just ramble through them right now and we'll uh and we'll make sure that you get your blackout things or whatever because you can win your prize uh but the cyber physical aspect to it is significant and important and that's that's what makes this uh this hacking a little bit different than um than some of the others so the uh the the impact is potentially significant uh as an example if there is a slowdown in the mountain pass in Denver uh that gets felt by the carriers all the way back to the east coast so you've got delays all the way all the way back across half the United States uh because of how tight um everything is and how significant that can be so um if anybody has anybody read The Damon by Daniel Suarez oh man nobody that's an awesome book you should read that book uh it was written uh probably like ten years ago or so I think uh but one of the uh it's really great because it starts talking about autonomous cars uh and if they're if they're under control and they become part of a fleet uh then it is a cyber physical sort of where where the uh that can be controlled in the physical world plus like he's uh like any good cyberpunk um like there's also autonomous motorcycles and then they put katanas on the motorcycles so they drive around it's totally awesome uh I recommend that book it's great there's also a sequel um but uh but yeah as as mentioned a problem in Denver can affect it and then um the other thing that I think is interesting is you know these are uh in my opinion easier to spoof than a logbook maybe it's just because of of my own morality but if I'm looking at a person I have trouble lying to you because I like people uh I feel like I'm a generally a good guy and I don't want to lie to your face so if an officer says hey how much did you drive I'm gonna probably tell him the truth but I'll lie to a computer all day long because computers are evil uh and Skynet's gonna take us over and kill us all uh and Elon Musk and I agree on that point um not exactly but it's much easier to lie to a computer uh than it is to a person and when you're talking about insurance fraud or or fraud related to this um when I started getting into the progressive I was able to piggyback off of a lot of research done by the community for the insurance fraud purposes uh they were saying you know we plugged our dongle in and oh we only drive uh to church on Sunday and back and we never go over 25 miles an hour and that was what they were doing with the insurance goggles and so they they had identified how that some of the components have worked and I think that same thing sort of applies to this uh if you're talking about trying to actually solve the problem uh with regulation I'm not sure I'm not sure that this does it it's but because like non-repudiation is a is a hard problem it's a problem that we can solve uh but that's sort of related to security and this doesn't have any of that stuff so how do you actually solve that problem it's uh it's difficult so in summary uh the flash card I guess for the insecure by law talk is we've got electronic logging devices that replace driver logbooks mandated by the United States government the devices are heavily commoditized and fail to follow cybersecurity best practices in general uh and the impact to US critical infrastructure is real we have to care about this massive increase in attack surface uh with commoditized hardware uh and what like I mentioned before it's sort of unfortunate because there are companies who are spending effort to uh to make this better telematics uh companies who have been maybe doing this for a while um but they they sort of get lost in the noise a little bit I mean there was seventy uh seventy things eight pages of different devices and if you're just trying to find one um that's what you're gonna you're just gonna grab the first one you see right uh so I I really don't ever do this in talks but I would like to I would like to do a brief exercise uh where we're gonna do a show of hands I apologize uh so when we're also thinking about the impact of this and it's where it's three days uh before uh let's let's use Walmart as an example where Walmart shelves are starting to get empty do you think I'd like a show of hands one or another uh first if you think that Walmart will be the one that the average consumer brand blames for the problem or even if in the news says like we've got a heavy trucking cyber vulnerability uh we don't have our goods aren't shipping appropriately whether it's just a DOS and they're they're not shipping um that's uh will will the average consumer blame Walmart or uh the trucking industry insecurity so for a show of hands on people who think that the average consumer won't make that abstract leap from the thing that they see in the face to the trucking who thinks that Walmart will be blamed instead of the trucks okay and who's on the uh people will blame the trucks instead alright so it very clearly uh very clearly the lot of you just uh think that uh if we have problem with the infrastructure the transportation infrastructure that is going to be the end unit that is the one who sort of catches the blame for it they definitely won't blame this you are yeah you are right I I think I agree does anybody think they'll blame this if this is what caused it yeah they'll bring yeah they'll blame Russia they'll blame whoever they're told to blame probably is what'll actually happen which is sort of an unfortunate thing and we can rant about that for a long time but when it comes to this yeah so that's an interesting yeah that's interesting is uh they're probably pretty much definitely not gonna blame this it's gonna be the the carriers OEMs uh possibly uh or more likely the end thing that's in somebody's face because the average person doesn't care um they're gonna they're gonna do what they first see and get the problem solved and out of their face do you have a yeah that's true that's true that's true yeah if you get if you get enough that says Walmart they're gonna blame Walmart if they get enough that goes into some kind of detail or it ends up on like one of the longer news programs like say 60 minutes and then they might be able to make that abstractly yeah I think that's an interesting discussion in terms of human uh psyche and how they approach problems and whatnot but I don't pretend to be a psychologist in any way uh I work with deterministic things like computers because they're easier um so uh yeah so uh thanks for coming very much and uh I appreciate it