Yeah, development and testing of autonomous vehicles at scale next 20 minutes Can reach me At the email or send me look me up in LinkedIn. I'm Frank Remer IBM systems architect long-time IBM guy Last three and a half years working very closely with autumn without a motive companies OEMs tier ones specifically in Germany, but also around the world and The idea of this short presentation is to give you let's say a little bit of the view of the project that we see Some of these are references where we work with customers and the customers agreed to share their name Of course, you know development needs AV data or data for AV development. I Want to share my experience with data standard design what we work to specifically work together in this area with equinix and also with entity data and then For use cases if we find the time for that So what we learned what are the challenges and AV data management more or less and I think we we we touch some of these points and Data ingesting and preparation cycles are very time-consuming This is mainly to do with that the IT metadata is different to the car engineering metadata And I think the picture here shows it in a very clear way Yeah, this is the poor engineer who who knows that there is data available to solve his problem but he's not able to find it because He has not the right technology not the right software not the right processes in order to do so As a result, you see many silos of infrastructure cost-intensive. These silos can be on cloud on-prem or any In any mixer we see multiple copies of the same data without a single source of truth and Data gets bigger and bigger so 300 to 500 petabytes of data is typically the play that we see here And as we all know It's expensive more less so what we're looking for is to be reproducible to be efficient and to be resilient And this also over a long time of period of time because these cars are on the road probably the next 15 to 20 years more less Okay, so what do we need? Yeah, so the old saying of data is the new oil Well, it's a little bit more limited for me I think like all data is valuable, but you have to refine it before before you can use it otherwise it's just it is just useless more or less and the refinement of the data and this is use case number one has to be done not in the Refinery is like on the right side, but the new refineries are collocation data centers and I think this brings us to the Idea of what do we need or what do we see in this game? Yeah, and I think first of all, you need a collocation data center You need a place where you can where you bring your data the test data which is run by the test cars You have to bring Them into a certain location This location needs to be connected with a high-speed networking connection And I tell you in a second what I mean with high-speed networking connection because these high-speed networking connections are required to reach the Public cloud providers and as we already said public cloud is a typical play here So we see the major players in this game AWS and Azure as the two big ones Google cloud Oracle cloud IBM cloud and several others Tencent cloud maybe also All of these are important and and what we're doing in the collocation data center space in the first one We put the data into the data space. We try to do some analytics on it. We have to do some CPU computing we have to do some GPU computing We have to use this data for testing hill testing still testing simulation and all the game more or less and If this collocation data center is big or small, isn't it depending on the let's say concept and the costing structure there but I think and Putting everything in cloud is what possible if you have a golden credit card That's the way to go if you want to make it a little bit smarter faster I think hybrid cloud is a very nice play and of course in this game Containerization OpenShift Kubernetes is is the way to go Okay, what do I mean with high-speed networking connection on the left side? You see not a high-speed networking connection. Yeah, this is a slow networking connection This is typically what we see when we talk to car manufacturers this is in their garage in their working environment and Sorry to say Wolf's book is not the center of the earth and it's not very well connected to any cloud they found out So That's reality. Yeah, what do we need for AV development and in this game? This means highways data highways fast lanes fast networking High-speed parallel networking latest technology more or less This makes it possible to cope with the data and this is what we have to do and this is where physics and reality comes into play These clouds are made of or these are existing data centers. They are servers Networking there there's there's GPU and and physics applies more or less. Everybody knows this Okay, what do we also see in this game is that this is a distributed play around the world? Which means we do have typically a three three location around the world in the US multiple sides and we aggregate these Multiple locations in a single color or in twin color more or less. We also see this In Europe and we also see this in Asia and we specifically if these is in China This is a specialized thing because everything have to stay in China But it's a multiplayer game and we have to interconnect that an interconnection also. This is very important Not using old MPLS technology, which is very expensive Using new software-defined Varn technology. This is a perfect match to the red hat product portfolio SD Varn Open run all these new technologies where software defined is the way to go using overlay networking and a very smart way of Orchestrating this these overlay networks more or less. So okay What does it look like then if we go into such a co-location data center? This is now let's say Some work we did together with Aquinix on the left side. You see heavy heavy-duty In-car data capturing and we did this together with a partner here We did this with a company called B plus and Siemens We can also do this with other companies. It's not to be it. Is it a d-space I by gam Seagate and I so same same play more as our driving around collecting data We see about 50 to 80 terabytes per development car per eight hour shift. So this is big data more We have to have some uploading stations in order to receive the data Then of course we put them in a data leg and it's very smart from our perspective in order to work on this data as Soon as possible in order to find the right pieces of data Which are relevant for the for the later stage of the AI training and then then there's still something missing Yeah, so I think the software gap that we see in this area has to be closed Finding the right data in the right language of the engineer and it's not about the IT meta data It's about the engineering meta data. It's about temperature velocity traffic situations weather conditions, etc, etc If we find the right data, then we could put it in a fast file system And this fast files is typically connected to AI training I show you this in a second or we can use and upload the data to the cloud We can also use the data and store it on a local cheap tape drive Same costing structure or cheaper costing structure as aws s3 pleasure Or we can also use it in in combination with and this is something new with aquinix metal This is servers out of aquinix. So they bought the company packer comm and of course they run Linux and they run Kubernetes and that's a very good fit in order to mix and match between your own servers the servers and the infrastructure rented from By a provided by aquinix and also the interaction with with cloud with cloud providers there To make this a little bit more clear. This is a picture This is how it looks like in the reality from an aquinix perspective Yeah, we do have multiple oems or multiple tier bonds or any mix and they are They're collecting data in a very very extensive way We have to have multiple facilities interconnected around the world. We have to use Fireballing between them. We have to use the right cloud providers which these specific oems have been selected and we have to mix and match that and We don't have to reinvent the wheel there because lots of the things are already available When we think about aquinix collocation data centers or using aquinix fabric as the interconnecting and the networking services They are ready to go firewalling technology And I think this picture shows it from a complexity side. This is really big. This is expensive and we have to And we have to fine-tune this and you have to make really sure that that this is what the customer wants and and Also the interaction the multi-tenancy bringing data together over multiple Tier ones things things like that. So this is all important more or less Okay Use case number two. Yeah, why are we collecting this data like? Like crazy because we want to do AI training AI training is the holy grail of robotic Robotic cars in this game and we all know that and I think this picture also shows it very well You can have the fastest car in the world, but if you're stuck in the traffic jam does not Well does not help more and this is really relevant for AI training You can have hundreds of GPUs, but you do not have the right data Well, you just wait What does it need and this is done work? We've done together from an IBM perspective from our data side file system side NVMe software defined data In combination with Wettered because containerization is also very very important here And the certification that we have to get from NVIDIA in order to feed these data be still the like the dgxa 100 n in the future the very new h 100 system We provided Reference architectures certifications best practices performance guides and etc etc So we did these kind of installations and this is just a picture to show it. How is it done more or less? Yeah, you can Inside the collocation data center This is also the best place to to create your AI servers more or less and interconnect them with the data lake that we already talked about We can split a little bit and match it and fine-tune the data lake in order to have hot data very closely available Using very very fast infinity band with low latency in order to feed these GPU systems And keep the volume of the data on on the lower or on a on a colder tier Which is a little bit more less expensive more less. These are the optimization tricks that we can play here Reference customers as I said, it's continental based in germany aquinics in germany big Installation lots of GPUs lots of dgx systems big file systems Though we do this together with that. It's a very very well known reference customer They extended the installation multiple times now from the gpu side and also from the storage side Publicly available as pdf just have a look at it. I think it's a very good reference more or less And we can talk in detail if there's a need for that Um, how does it look like in such a collocation data center? And on the upper left on the upper right side, you see the a real actually the real picture from the aquinic side They posted this on the web. So it's a freely. It's freely available. These are very very large data centers collocation means there are other customers in the same building or on the same campus and specifically for europe all the major cloud providers are In colo data centers, they are just in on the same campus and this makes it very easy to have a fast networking line Because your own data center is more or less physically very close to To hyperscalar data centers are not they spend over multiple collocation providers But typically they have from a networking side. It is very very close A cloud has to be close in order to use this high high speed connections more or less What we also see this is specifically important for the automotive space is that testing is important because it's security and relevance of testing is absolutely critical and testing from the automotive engineering means Hardware in the loop testing which means they they deploy the electronics, which is Typically in a car they put it in a in a in a in a rack and this software and this hardware software Combination just acts the same as it would be in a car and then you have not only one You have hundreds of these hills ricks hill stations and you feed them with the real data that you've That you have recorded on the road not only the data that we use for ai training You feed your complete data stack to those hill systems So this means hundreds of petabyte for each hill hill hill run is typically and if you keep all your data In in in cloud. Well, you have to pay the equest charges for that So that makes it a very expensive operation. Then this is what Several of these customers have found out. So it's a very I think it's very smart to put some of the data Close to your hill testing where it is available for less costing And we can also combine this and this is typically done So the more modern guys are using more software in the loop testing where the where the hardware is replaced We are a software model and typically the software model is is using Kubernetes and open shift and but still I think there is There's some need for hill testing But we see a tremendous increase in software in the loop testing software only but still testing is is Very relevant and this is one of the pictures we could we can we can create this more or less Okay, when we all put it on a on a chart, this is well not The greatest chart in the world But I think it shows what we have to do from an automotive perspective. Yeah, we have to collect data We have to do ai training or data preparation ai training We have to do hotter in the loop and software in the loop and we have to do simulation Which I come to at as a as a use case number four and I think this fits very very nicely to the overall architecture of Of open shift that we can do all these things also some something important specifically in the hill space There's also windows server because some of these hills ricks Are still using windows operating systems that we cannot get rid of us at least very fast, but this Still is very very good From an integration point using kubernetes also on the on the in the windows environment should be no big deal Either this is this runs physical virtual in the private cloud and the public cloud or any mix on the edge or in a collocated system does not really It does not Make any difference As long as we have the right software concept for that and as jill already said This is the way why container rising and container and operators is is the big thing And this is also what we see here here in this game We also see it of course aws and azure being the dominant cloud players here and if we're using the elastic kubernetes servers or the azure kubernetes service We can also very easily intermix and play Depending on the costing structure Last use case and I think this is tremendously active at the moment and still it's it's it's very It's kind of new and the people are the market is still is still involving which means the simulation Testing on the road is very very expensive and specifically in the last year and the or the Two last years where the test please had to be shut down and there was a lack of Of getting around in the world people starting saying well, how can we do the simulation here? can we use sometimes gaming technology virtual worlds And add the physics and the reality and the and the model lay on the and the sensors which are on these cars If we have a software model, which is good enough and can we combine this In order to verify if the testing that we did on the road Is correctly and can we match it and can we extend it and can we do variations? and I think this Graphics, which I stole from the NTU project in In in Singapore puts it very nicely together. You need a virtual test orchestrator. Typically We have to have the vehicle dynamics. We have to have the right interfaces Then we need scenarios the narios now open scenario comes out of the asem consortium I think this is very very good work open drive open scenario Open road I think this is the right way to go in order to have a unique common understanding and language which the Which the engineers understand we need traffic models. We need environment model This is typically done with co-simulation And also sensor models which are close to the real thing and which has to be provided by the real sense of sensor people there and we put everything together and what we see in this area is using the GRPC standard from from google in order to get everything together in a in a very very close and and and like a shared memory space or working together as a as a good model which Which is working on not a single executable but multiple different Containers which have to be Scheduled in combination on the same cluster very very interconnected And then of course this workload then will have to be Scaled out which means if it runs for a single engineer Typically they look at the virtuals or they look at the screen But when everything is okay, then the screen gets detached and they run the simulation a million times or even Even more in order to find out things which are edge cases which They change the weather they change the conditions. They change the cars They is et cetera, but it has to be matched to the reality. I think and this is still very very critical because We also have Two now two different ways of thinking. Yeah, the real car guys They think only driving is the real thing and people in the computer. They think this is Sometimes it's gaming. Yeah, and this is not gaming. This is reality modulation and simulation And it has to be accurate. That's the problem Okay, I hope I think of course For us, what do we need there? I think Kubernetes platform is the platform of choice for the autonomous vehicle development It has everything that we need The integration also to the nvidia playground the nvidia ngc container registry where there is lots of software available from the open source Which is gpu enabled and ready to go and ready to be consumed can be very very easily Constructed and I think there's still some way to go because The poor car engineers are quite new to this modern way of Computing, but I think this is I think there's no alternative any any longer Most of the people have heard something. They know something about doger But well doger is a little bit old now and you need of course much more But I think time will tell and I and I think we are we are on the right track here and Everything which has been said before at this conference fits very nicely in this picture Okay, well last picture just to give you some example and we're happy to show you The in-car recording from the left side with partner Siemens and b-plus We can show you the right side, which is an equinix data center And we have set up POCs and we are openly working together and share our experience as As long as it's possible from the end customer point of view there So thanks for that frank. We're we're Hoping that the back of my car does not look like the backseat of that car I saw that and I'm like, oh, no, please not that so This is a it's a it's a demonstration vehicle. Yeah, but they are they can record with With a high very very high bandwidth. Yeah, it's not really needed in it. It's more than you need more or less