Okay, so I learned that once the music is turned off the stage is mine. So this means the stage is mine now It's a huge stage actually Welcome to my afternoon presentation about approaches to ultra long term system maintenance At the embedded Linux conference 2016. My name is Wolfgang Maurer. I work for Siemens corporate technology In Munich for their corporate competent center embedded Linux Of course, I guess in Germany everyone knows what kind of devices Siemens is building medical stuff infrastructure stuff Power distribution transmission and so on so All kinds of industrial devices that are supposed to last for a very long while and that's Of course one of the reasons why we got interested in maintaining systems for ultra long Tom time ranges so not just five years not just ten years, but 20 30 maybe 40 years a while ago Let me use this opportunity to also mention the civil infrastructure platform a Recently founded initiative of the Linux foundation that deals with precisely these problems and we have a booth in the first floor where we showcase Some of these ultra long-term Devices and we'll happily tell you more about the initiative in case anyone wants to join our efforts So what's my talk going to be about? I will speak about architectural aspects of long-term software maintenance. We've heard already some talks about The more component centric questions of long-term software maintenance About how to provide Linux kernels in the long run About how to fix security issues in Packages that have been deployed for decades, but I'm going to take a little wider view of the whole issue and Consider not just the component-based level, but really the whole system design and architectural Questions which are in my opinion crucial When I was preparing this talk and that's so I felt obliged to put up a disclaimer Right at the start and I was preparing this talk. I Thought yeah, so We've given consideration to these issues now for more than a decade that I've been or roughly a decade that I've been working with Siemens we've spent a lot of time thinking about these issues when we founded the civil infrastructure platform and So I prepared the slides and I was reading through the slides and then I was thinking okay Hey all the stuff that is on the slides is really obvious actually So there must be something wrong with my slides. I didn't think the topic through enough I'm going to board the audience for like half an hour with stuff They maybe already know but then again. I thought when we look at industrial systems Most of these topics that are fairly obvious to the Linux community are completely Non-obvious to traditional system builders So now depending on where you'll come from what your background is if you've been doing a Linux kernel development for the last two decades and If you speak more to GCC and To get and to patch stacks then you do to external customers Then stuff maybe will be obvious to you But on the other high on the other hand if you come from the traditional businesses from the traditional Infrastructure businesses, then I guess or I hope that most things will be quite quite non-obvious to you and provide some Some ideas on how to improve your development processes your development your architectures for long-term system maintenance So what are we talking about when we think of? Software-centric appliances these day, of course we have on the one hand the traditional Consumer electronics devices we've already heard in a number of talk that mobile phones notebooks tablets and so on Are not something that you of course would consider To be relevant for long-term Maintenance and we all know two years is more than enough Time for the lifespan of a mobile telephone Of course if you're as stupid as me and drop yours into the Pacific Ocean, then it's even less than two years But that's an exception We also have entertainment systems that are quite softer centric these days We have things like ovens washing machines home control devices home automation devices That become more and more popular and that of course also rely on software that are very user-centric, but See much increased a greatly increased lifespans compared to traditional consumer electronics devices because if you're building a house At least if you're building a house in Germany, then you plan for the house to last for maybe 2030 40 years and in these 40 years you don't necessarily want to replace your home automation every two years Like you do with your mobile phone on the other hand we have Industrial systems and the classes of systems that we're looking at are fairly different It's maybe medical devices computed to mography x-ray imaging ultrasound devices with infrastructure devices gas wind Gas power voices apply with power stations with energy distributions But we also have things like park space management that requires more and more software. We have mobility We have planes trains automobiles, of course Mars rovers space stations. I was thinking of Removing this space station thing although everyone knows that space stations need a lot of long-term software, but then I just received this image which reminded me that actually the Next generation that may live in space stations, maybe your own offspring So we definitely should care about the software quality in these appliances When you think about long-term maintenance, there's from my point of view two fundamental questions The first question is when you look at your appliance does it really make sense? to Make this to provide long-term support for this appliance because there's of course always Alternatives to supporting a device for 20 30 40 years, which I'll talk about and that is The first question that you should be asking yourself Is it really not just doable to support your device over such a time frame, but is it also reasonable? Are there no? Alternatives and if you come to the conclusion, yes, we want to support our device over decades then a Second question you need to address is what is the most appropriate system architecture for long-term maintenance So you shouldn't think on a component level. You shouldn't think just on a kernel Or what not level but you really should you really need to be thinking about the system about the overall system architecture The types of lifespans that we consider just to get the picture for consumer devices They are not for long-term maintenance is about two to five years mobility Devices like cars already have a much greater life expectancy. Maybe five years is even a little short unless Um Yeah, you're running very special course, but 10 years 20 years Is more the expected lifetime here industrial systems range between 10 and maybe 30 years if you think of manufacturing plants and so on And infrastructure systems so the area where the civil infrastructure platform, of course, it's most interested in Take this to very extreme levels 30 years is certainly Not a very long Time of existence for a power stations and when you think of water supply Power lines and so on then 80 years is also Maybe even on the not so long side end of the spectrum What unifies all these domains these days is that of course all or many of the devices run Linux That goes without saying on an embedded Linux conference But the thing the interesting thing that has happened during the last years is that the long-life requirements That we face with such devices are these days not only Seen in the industrial environment, but I've mentioned examples of home automation and Smart cars that are Linux. They are also they pervade more and more into the consumer electronics sector and in my opinion if you look at If you look at developments like the internet of things smart home and so on then you Maybe it can start to think that the short lifespans that you face in traditional consumer electronics Are becoming more and more of the exception to the rule and maybe not the rule itself So long life spans longevity of devices will more and more become a case that people will need to deal with So the fundamental questions that are Associated with long-term maintenance so when or let me let me phrase that different then you look at systems that you need to maintain for decades then of course You need to update these systems that is substantially different to traditional engineering if you think and the true traditional engineering that used to be Current not so long ago if you think for instance of photographic cameras a Camera that you purchased 30 years ago will these days work exactly in the same way as it did 30 years ago You maybe pick up difficulties With getting films for the camera But you can take the camera out of the box and it works same thing for a camera that you purchased 90 years ago So okay, it's black and white, but the basic operational principle is still the same um Today's devices are all based on software and that means for some reason we are not yet able to create the software with the same quality with the same longevity that Traditional devices used to be built in software as you all know requires updates it requires changes it requires continuous modifications of the device over its lifetime because if you look at the modern camera with Wi-Fi support with Support for digitally exporting images the thing for sure won't run in 30 40 or even 80 years time so Updating devices changing devices in field has become a necessity That cannot be avoided and with every change with every every engineering change to devices There's of course risks involved in updates There's also a benefits involved, but there's risks involved and if you want to minimize these risks then there's basically two very simple Questions that you need to address the first question is how can we restrict updates? To very specific areas of the device so in case and if something goes wrong only this particular area Is being concerned but not the rest of the device and even more important question is how can we avoid? updates There has been this growing belief in the industry that and with consumers that every problem that a device Absorbs can be fixed with updates, but I think that's one of the major bogus or wrong assumptions that underlie this whole updates everything updates in very short time frames thing because Updates don't tend to fix more problems than they create in fact If you look at the statistics updates are really a major problem a major source of problems these days The second same assumption that is wrong is that all components can actually be upgraded That's of course for consumer devices, but the more you go to the traditional industries the more your devices Will become very very involved to be updated another bogus assumption is that We've heard a lot of times now or or we we keep hearing a lot of times that upstream integration of changes that you do to systems Reduces your maintenance if for effort leads to better products and so on which is of course right in a very large Set of circumstances, but it's not always right and especially when you come to the more arcane corners of Of the industries like the VME bus for instance that is used in many industrial Controller appliances, but if you are the only remaining User on this planet, maybe except particle physics, but they also don't count very much In these communities if you're the only remaining user and nobody else then you benefit from improvements in the field then bringing stuff upstream also won't automatically lead to less problems in your own appliances so What's important when you design long-term maintenance systems as I said It's not about components, but it's really about the whole architecture So you cannot Address all questions of long-term maintenance by simply installing long-term support versions of your components and be done That just doesn't work. It would be nice if it were, but it just isn't what long-term maintenance is All about is two things. It's for one architectural issues You need to design your systems from the ground up in a way that it supports long-term maintenance And you also need the proper mindset of the developers of the people designing the system to be able to bridge the gap between the long-term requirements of the system and the school of thoughts that The components the software components that make up the systems Intel I said in the beginning that it's not Always appropriate to go with a long-term maintenance strategy And of course rest there is an alternative to long-term maintenance And that is to periodically rebuild your devices or parts of your devices Just take them out and replace them with a new appliance that may be Cheaper than going for a full long-term approach. Just think of a power station that has an information display somewhere The traditional German way of engineering such displays is to make them last for 30 years to make them so sturdy that 27 North Korean tanks can roll across the appliance and it still will work But is that really always a reasonable choice? No, it isn't because if you take a standard display that you purchase for 20 euros Then the North Korean tanks come rolling Destroy the thing and then you just replace it and the same goes for the software if the software Experiences falls if you get problems with the software, then you just replace the component That's way cheaper than designing the whole component to last for 30 years in the first place The decision criteria that I like to use to distinguish between long-term maintain systems and Systems that are periodically or a periodically rebuilt I've summarized up here and basically it's these four Questions that you need to think about with the customers with the system designers the first and The most important one is what kind of software runs on your device is a device running a fixed set of software That you can trust that you provide by yourself that you are in full control of or is your device able to run basically arbitrary payload software of course with mobile phones the whole point is to run arbitrary payload software like apps when you're building a nuclear reactor or a Power station then there is a certain trend in the industry to allow apps to be run on these devices Or to allow apps to be run on medical devices So that doctors don't become too bored when they have to wait during surgery or something like that, but I Guess that's a question for the software architect that really needs to be addressed by them. Does it make any? Does the device get any benefits from being allowed to run arbitrary payload software? That is quite often not the case and if you restrict the software load on your device to fixed and trusted software then Things get a lot easier to maintain because you reduce your attack surface by quite a bit and that is in my opinion One of the most important preconditions for building successful Long-term systems to only or mostly run fixed and trusted software The second question That needs to be addressed is how is the device accessible? Of course in the Internet of Things everything is universally accessible by some By a network connection in some way or another and the trend in industrial appliances is also to make these devices more and more network-elabel and able just think of cooking ovens that these days have an Ethernet connections or eatings or Yeah, he does that have an internet connection The question again here to ask that's that needs to be asked is does it really bring? Substantial benefits to put devices on the network and is it really a smart idea to put my MRI machine on the network So what benefits do I get and what problems do I buy and as you all know? Most of the security issues most of the problems with devices come with network access So by isolating these devices, which doesn't mean you cannot access them You can provide ports that are physically accessible you can provide local network connectivity and so on but just think three times if it's a really good idea to put your precious appliance into the internet or not and then most of the times you will end up on this end of the scale for Really long-lasting devices and that again simplifies things a great bit The third question is about hardware stability versus variants. What I mean with that is How long will the hardware that your appliance is based on be available from the manufacturers and For how long will you be able to build up stocks to build up supply stocks that you can use to replace? 40 components in the field Again, it's quite astonishing how much The problems come from a hardware that is supposed to be compatible with previous releases So you make our contracts with hardware vendors They promise you to deliver the very same hardware for 20 years for 25 years But then after 10 years or so you realize some minor detail that they didn't really bother Even informing you about has changed which may require major Modifications of your software stack which again in turn is a very involved thing because say you need to backport a device driver over 10 years worth of kernel updates with device model changes with core kernel changes and so on good luck with that So really try to ensure hardware stability in the first case that is That is one of the preconditions for building extremely long-lasting systems The third point is somewhat related sometimes people think a lot about how to ensure that they can Resupply hardware for 25 years, but then they just build a USB port onto their device and that basically eliminates all these efforts because you can Plug in new hardware hardware that the device has never seen before into the system and that's the same as a changing hardware basis so Make sure that really both of these conditions are fulfilled Even if you are on the left-hand side of all these questions, they're still So if you are on the left-hand side of these questions, then basically things are fine or Are a good basis to build long-term maintain maintainable architectures But there's two more points that need to be considered one is the verification and safety requirements and The other thing is the cost sensitivity of your devices say if you are not Or if your cost pressure is not all too high if you can afford to you to Use hardware that offers more computational power than the task actually provides then this relaxes the This relaxes the assumptions or the requirements for long-term maintenance a bit because You get quite interesting architectural alternatives. You can run your core payload inside virtual environments I said that hardware stability and That's a stable hardware basis is essential In my experience But if you manage to move your appliance into virtual machine then basically you don't care about what your real hardware basis is You just let the appliance run in your VM. The VM always provides a Simulated or a virtualized set of hardware and then you just use the best commodity Product that's available the cheapest one to run your virtualized hardware Of course, this needs to be This needs to be brought in sync with any certification requirements that you have with any safety Requirements that you have this is a non-trivial process when you're speaking of virtual systems versus real physical systems With the certification authorities at the moment But the problem is one that can be addressed on the social level and is not really a core technical issue moving on another question that Needs to be addressed when you think about two long-term maintainable systems Is what types of risk? Should your long-term strategy? Actually prevent so why do you want your system to stay more or less unchanged on the market for quite a long while and There's one fairly obvious reason you want to prevent your device That it stops working So you want to be able even in when the device produces a fault and the device When the software on the device Contains bugs that you didn't fix before delivering the device you want to make sure that you can Fix these bugs and eliminate these problems 10 15 20 years after delivery. That's of course a Thing that goes without saying the same is or more or less the same is that You want to prevent that faults in the device cannot be repaired anymore or that device Can be influenced from the outside so these are all So these these are all these are all things that call for a changeability of The device after decades of operation, maybe the influencing Oh, yeah, no, no, sorry. Sorry so that that call that call for changeability after after decades one reason And that in my opinion is actually not worth going for with a long-term maintenance approach is if you want to Prepare the device for changed expectations on the customer side on the user side. Say you build an appliance that works perfectly fine On the day you deliver it to your customers and then 15 20 years later customers Have developed different expectations to devices say you're using an information screen that Displaced information as it's intended to do but then customers want to have touch screens Customers want to have nicer your eyes customers want to have 3d graphics and your Your salespeople obviously will always want to fulfill these customer needs but It does not make any sense to prepare for such Commodity changes or convenience Convenience changes in the design of a system. That's more a case when you rip out a Component of the system of the long-term system and replace it by something different of course when a Failure or a problem of the type I said here occurs You have a number of responses how a company how a manufacturer can react to such problems Yeah Replacing the device With a different hardware software combination or replacing a whole combina the whole component is of course one Possible way to go another possible way to go that we often see is you just ignore the issues Which is of course usually not what you want It can be reasonable under very specific circumstances, especially if you have contracts with the customer that states that The device is not supposed to operate and say beyond Beyond a certain time frame then it's not your problem, but that's more a legal thing the the interesting case is of course the interesting option is of course when you want to modify the software to get the Device in working shape again, and that then requires changes to the system requires backporting requires component changes That's where it becomes interesting if you don't do that after five years or after ten years But after 15 years when the complete landscape is different than it used to be when you build the device coming to the Technical aspects or to the software architectural aspects that you need to take into account during device design phase that will enable you 15 years later to still change the software in a way that you can fix the aforementioned issues and Actually, it's quite interesting when you look into the software engineering literature and See what the researchers and what what guidelines people have come up with a producing maintainable software Then it's mostly common places That's in the textbook. It's like Yeah, you properly structure your software into modules you make sure that build Systems are used you make sure that everything is well documented, but that has more or less That is not that is just General good software engineering practice and it's not so much related to long-term Maintaining systems what I find is most crucial for the long-term maintenance of systems is three points The most important of them being to minimize any cross-cutting issues in the software design Itself so avoid issues that concern multiple subsystems multiple components and so on where you need very Introduce some coupling of knowledge because even if in the original development teams people with the appropriate skillset with the combined skill sets for these Issues have been available then This will be completely different a decade or two after you've created the device and especially such cross-cutting issues cause couplings cause unintended Because unintended connections in the software architecture that are very hard to reconstruct Once the original Designers and the original implementers are not available anymore Which is a thing that you of course need to assume when you deal with time spans that we're talking about This can be done by harmonizing the technical and social organization. That's quite a big word But actually that's more or less common sense What you do if you have people with clear Responsibilities for clear components when you won't be introducing these cross-cutting issues into your software architecture at all You may call that Conway's law if you like that's an Four decades old wisdom software engineering, but again, it's mostly mostly common sense And what's also very important is to make sure that you keep a meaningful and reproducible history of how you Develop your system again. That's a common place for kernel developers That's a common place for a most base layer software packages that you find these days in Linux system people take real good care about making proper a proper a patch stacks about properly documenting their contributions to software, but that's far from that's far from universally done in commercial industrial software development, so Yeah adopting the best practices from the Linux kernel community from other communities in the OSS movement really helps here because as you all know Architecture documentation tends to get lost over time Design documentations tend to get lost over time But if you manage to keep your development histories your proper development histories Then it becomes actually feasible to reconstruct these original architectural Considerations that you want to preserve one major points in a for the software architecture I already mentioned a number of times and that's the connectivity issue You should really really really really really really think three times before your Equip your device with a connection to the internet or some other network and once you finished thinking three times About whether you should do that or not. You should maybe again think three more times If that really pays off or not And if you do this thinking properly, you will realize that in almost all cases Irregardless of what the internet of things people tell you and what update over the air people tell you and What the proponents of web-based solutions tell you the benefits that you get for a product by introducing it by introducing network connectivity to it usually very much outweigh the costs that are associated with it in terms of security in terms of In terms of undesired access and so on Another thing that's very important when you design software architectures for long-term maintenance systems based on open source components is that you have a Translator in form of a person so someone who knows both the arc the That domain that you are building your product for who knows the custom requirements who knows What the device? Is required to deliver in terms of features and so on but that also knows how the software communities work that So to speak deliver the components that go into the product because that usually That usually helps to correct some wrong assumptions about the guarantees that components make about the guarantees That long-term releases make About the way how you use these components in the first place It's still it's still very easy or it's still still very common To employ to see products where software products where I'm sorry Where OSS base components are deployed in a way that were never really intended by their original communities are mostly related to To web things to stuff like node.js that was never intended for instance to be used in anything that's Updated less than every two weeks But still you find that used in many systems that are supposed to last for years because the designers of the architectures We're not really aware of the mindset of the communities of the Yeah, of the communities that they That they base their components on and What also more or less goes without saying is yeah, you should of course make components runtime replaceable if you run Into an issue with a certain component make sure you can replace it without Touching the kernel without touching the base layers of your system and one really good way or one One advice that I find Really worth giving in that respect is that you should these days prefer user land solutions to in kernel solutions whenever you can This is these days possible with more and more devices with more and more classes of devices of course Using user lands device drivers using user land access mechanisms to systems hardware will not always deliver the best performance results will not always Delive a hundred percent of the capabilities that these devices Can provide then they are driven by a a proper kernel device driver, but If you are intending to build a system that is supposed to last for decades Then it's maybe anyway the wrong choice to drive hardware up to its maximum capability And leave a little a gap a little safety gap in there another thing that provides a endless line of Problems of fun depending on how you want to call it To systems that are on the market for a long while is to rebuild the systems from source code After 10 after 15 after 20 years they've last been built Building of course is something That people usually don't take much interest in It's hard enough to do and once it works once you produce binaries out of it people are satisfied, but when you think about How hard it is how hard it often is to produce binaries At development time then it of course becomes clear that this will be even harder 20 years after the first launch of your product because In build systems, there's often a lot of experts or a lot of experts knowledge Is implicitly required to build systems that is that is obvious to all developers who originally work on the system but that is basically unrecoverable after 20 years of bitrot that has been observed by the system so you need to spend special efforts to ensure that the payload application plus all modifiable system components that Often gets forgotten because this comes from distributions and that is just some binary that you install But of course all these distribution installation mechanisms will have cease to work after 20 years So you need to make sure that both the payload application and the modified modifiable system components are in a buildable state which again is quite simple to achieve but often overlooked so you preserve I'm your document documentation is really the key including documentation of seemingly trivial Details and you also need to make sure that the documentation stays available for decades and Why of course nobody really likes to pay work with paper these days anymore having hard copies of The respective documentation can really help you because if you go back 20 years in time then maybe some Genius may have come up with the idea of providing the documentation in Windows help files or in In army word or in some format that you just cannot Open reliably these days anymore So the simpler the format that provides the documentation the better that is a speaking of Building systems another point that is relevant is the availability of source code It's of course very simple to just produce a Tarble of the state of your application at the stage when it was delivered to the market But as you all know the application is not just made up of the application code itself But also contains lots of libraries lots of dependencies and when you look into build systems, they have dependency specifications especially with You have the more modern scripting based languages that can specify constraints on the revisions that you need to use But these constraints in my experience often fail to live up the The promises they deliver so just stating that you need a revision higher than blah blah dot something of one package Does not really guarantee that this will also work with really all higher revisions so and For many libraries that are not exactly in the spotlight that are used less frequently the availability of the source code even if It has been on the internet in some publicly accessible Repository can be quite troublesome even after five or only after ten years. So It's it's quite essential To make sure that you have copies of every library of every Dependency that you need to build your system And that of course also includes the build infrastructure In general when it comes when it comes to using or when it comes to dependencies in software products You need to consider when you use libraries that live So the textbook knowledge about libraries is of course libraries are fine libraries to solve things libraries Make it possible to share development efforts for common tasks in software engineering But the point that is quite often overlooked Is that libraries come at a cost libraries come at a massive cost because libraries? Require other libraries libraries require other dependencies libraries Libraries often silently change if so of course that doesn't hold for the For the well-tended system libraries like the libc or things like that But the more Arcane and the more obscure libraries get the less people are interested in really providing proper Version compatibility. So the amount of experiences you face with libraries And with a slight compatibility with problems with libraries can become really tremendous Now when you build a system that's supposed to last for decades You expect you You need to You need to very precisely distinguish between the prototype that you are Actually the the Development Version of the system that you're doing when you do experiments when you say you try different algorithms you try different Approaches to solving problems you play with libraries you have 17 different ways of solving one problem and the State that you ship to a product So consider you've used 17 machine learning algorithms to solve one particular type of problem Then you will in the end of the day Arrive with one that you use in your product and you should really consider Once you find the code that fits your needs to rewrite this code and get rid of library dependencies if you're using say Knew are to solve your machine learning problems then experience shows you install one library that pulls in 25 other system libraries That pull in even more dependencies and off the 500 lines of code that you actually need You produce maybe 50,000 lines of code that become a maintenance burden for your system Okay, so coming To the conclude coming to the end When it comes to We've we've already heard in the previous sessions a lot about the qualities and lesser qualities of board-to-port packages And that is one point that I just can reiterate so when you Develop your your products even if there are supposed to last for three or four decades Make sure that during development you develop you always develop against the latest Head revisions or the latest available revisions of your software packages develop against the mainline state rebase It changes that you do against the mainline state and try to stay Contemporary as much as possible during development Especially you want to you want to do that with the Linux kernel when it comes to embedded devices We all know that many embedded devices require vendor board support packages But that's actually what the exact opposite of what you want You want a working device and you want your working device not Provided by this one point something million line changes that SOC vendors often supply to you You don't want both support packages. You want board support. You want support in the upstream Linux kernel and Why I'm stressing this point so much is that there is Actually one chance to get this board support instead of board support packages and that is prior to purchasing the 10 to the something units of the hardware that you want to ship With your device so if you don't make the efforts of convincing your Your SOC manufacturers to provide you an appropriate and the base system in that appropriate form during development Then you will have lost this chance forever and then Anyone anyone who has supported board support packages or who has tried to port a board support package To a new kernel revision that was maybe even three or five years also far away from The time range time ranges we are talking about knows that this is basically the end to all long-term long-term support Efforts good. I'm Going to skip a few slides and yeah, I'm going to okay. No time has progressed even further than I had Expected it to be let me come right right to the end of my talk. I would have had to say a few more things about Designing software for base About working with with upstream component providers, but you'll see that in the Slides that I'll put online. Let me come to the wish list to my wish list a list of things that could improve the situation for Long-term maintainable system providers when it comes to picking Changes to back porting changes that occur in the active development branches of the underlying projects and to bring them back to these long-term maintain systems the major complication that we face when When we need to update systems that have been in the field for a long while is the problem of how to select patches from the from the action from the the ongoing development and that's a problem that's Far from solved. So what would be the what would be the desirable thing? Of course is if you could automatically track the upstream development that's going on In the base project and then find some way some some magic way of getting a list of Commits out that need to be back ported that would be Possible in theory, but and there are there is actually some research going on To achieve this goal, but I doubt that this will in any realistic time frame deliver Mechanisms that actually work in practice So the the human factor of the human The expert selection the expert They have the expert selection of patches that need to be back ported Is basically Unreplaceable so the best thing one can do is to avoid Duplicated efforts to have to avoid having experts in different companies doing the same work over and over again to do the same work of selecting patches of interest for different projects and the situation I guess could be improved quite a bit if Patches were By the upstream projects already Pre-characterized into different maintenance classes into classes that into so if so basically the patches would need to be annotated if they are If there are candidates for back porting some of the larger projects Of course already do that but most of the projects don't and it would be quite helpful to also have applicability ranges in terms of the previous releases that the patches would Would apply to Would apply to because of course when the when some development has been going on then there are patches that only Concern subsystems that have not been available in all the releases and so you also don't you don't need to check if You also you can you can In the first place Avoid looking at these patches because they surely won't apply anyway okay, so Yeah summarizing I Think that the long-term maintenance best practices for development are very similar to when you do proper open source Style development anyway, I think that's obvious to open source communities But that's not obvious to the traditional engineering providers And so traditional engineering can learn quite a lot From the open source communities in that respect Providing long-term maintainable systems is definitely not rocket science, but it's still more of an art the science You've heard me saying a lot in my opinion. I think that and that is good It's actually really hard to come up with quantifiable numbers But it's definitely a topic that needs to be addressed in the future and we will need to find ways to Quantify to quantify these decisions and to make these decisions more based on objective criteria Then on gut feeling and experience because we will be seeing a lot more long-term Relevant devices in the future. So with that I'd like to thank you for your interest and your attention And I don't know if there's room for questions Do we have questions none then yeah, thanks again


Okay, welcome back, everyone, to theCUBE's coverage of AWS Remars here in Las Vegas. I'm John Furrier, host of theCUBE. I've got Alexi Serkov, partner at Deloitte, joining me today. We're going to talk about AI, bias AI trust, trust in the AI for the, to save the planet, to save us from the technology. Alexi, thanks for coming on. Thank you for having me. So you had a line before we came on camera that described the show and I want to, I want you to say it if you don't mind because it was the best line for me, at least for my generation. That describes the show and then your role at Deloitte in it. Sure, listen, I mean, I may sound a little corny, but to me, like I look at this entire show at this whole building, really, and like everybody here is trying to build a better Skynet, you know, better, faster, stronger, more potent, you know, and it's like, we're the only ones, like we're in this corner of like Deloitte trust for the AI, we're trying to make sure that it doesn't take over the world, you know? So that's the, that's, you know, that's a gist of it. How do you make sure that AI serves the good and not evil? How do you make sure that it doesn't have the risk? It doesn't, you know, it's well-controlled that it does what we're asking it to do. And of course, for all the young folks out there, The Terminator is the movie and it's highly referenced in the nerd circles, Skynet's evil and helps humanity goes away and lives underground and fights for justice and I think wins at the end. The Terminator three, I don't get it. I can't remember what happened there, but anyway. I thought the good guys win, but I think they do win at the end. So that brings up the whole point because what we're seeing here is a lot of futuristic positive messages. I mean, three areas solve a lot of problems in the daily lives, you know, machine learning, day-to-day hard problems. Then you have this new kind of economy emerging, you know, machine learning, driving new economic models, new industrial capabilities. And then you have this whole space save the world vibe. You know, like we discovered the moon, new water sources, maybe save climate change. So very positive future vibe here at Remars. Absolutely, yeah. Yeah, no, it's really exciting just watching, you know, watching the speakers talk about the future and conquering space and mining on the moon like it's happening already. It's really exciting and amazing, yeah. All right, let's talk about what you guys are working on Deloitte because I think it's fascinating you starting to see the digital transformation. Get to the edge and when I say edge, I mean, back office is done with cloud and you still have the old, you know, stuff that the old models are people who will use, but now new innovative things are happening. Pushing software out there that's driving with a fintech, these verticals. And the trust is a huge factor. Not only do the consumers have a trust issues, who owns my data, there's also trust in the actual algorithms. Exactly, exactly. You guys are in the middle of this. What's your advice to clients? Because they want to push the envelope hard, be cutting edge. Right.  Absolutely, well, I mean, the simple truth is that with great power comes great responsibility, right? So AI brings a lot of promise, but there are a lot of risks. You want to make sure that it's fair, that it's not biased. You want to make sure that it's explainable, that you can figure out and tell others what it's doing. You want to make sure that it's well controlled, that it's responsible, that it's robust, if somebody feeds it bad data, it doesn't produce results that don't make sense. If somebody's trying to provoke it to do something wrong, that it's robust to those types of interactions. You want to make sure that it preserves privacy. You want to make sure that it's secure, that nobody can hack into it. And so all those risks are somewhat new. Not all of them are entirely new. As you said, the concept of model risk management has existed for many years. We want to make sure that each black box does what it's supposed to do. Just AI, machine learning, just raises it to the next level. And we're just trying to keep up with that and make sure that we develop processes, controls, that we look at technology that can orchestrate all this de-risking of transition to AI. The load's a big firm. You guys saw you in the US Open sponsorship was all over the TV, saw that. You're here at Remar's show that's all about building out this next infrastructure and space and machine learning. What's the role you have with AWS and this Remar's? And what's that in context to your overall relationship to the cloud players? Well, we're one of the largest strategic alliances for AWS and AWS is one of the largest ones for Deloitte. We do a ton of work with AWS related to cloud, related to AI, machine learning, a lot of these new areas. We did a presentation here just the other day on conversational AI, really cutting edge stuff. So we do all of that. So in some ways we participate in that part of the, the part of the room that I mentioned that is trying to kind of push the envelope and get the new technologists out there. But at the same time, Deloitte is a brand that carries a lot of history of trust and responsibility and controls and compliance. And all of that comes- We get a lot of clients and we get big names. Get a lot of big name enterprises that rely on you. They rely on you now. Exactly, yeah. And so it is natural for us to be in the marketplace, not only with a message of, let's get to the better mouse trap in AI and machine learning, but also let's make sure that it's safe and secure and robust and reliable and trustworthy at the end of the day. And so this trustworthy message is intertwined with everything that we do in AI. We encourage companies to consider trustworthiness from the start. It shouldn't be an afterthought. Like I always say, if you have deployed a bot and it's been deciding whether to issue loans to people, you don't want to find out that it was like, biased against a certain type of population six months down. I can just see in the boardroom, the bot went rogue. Right, yeah. And you do all those loans. And you don't want to find out about it like six months later, right? That's too late, right? So you want to build in these controls from the beginning, right? You want to make sure that you know, you're encouraging innovation, you're not stifling any development and allowing your- There's a lot of security challenges too. I mean, it's like, this is the digital transformation sweet spot you're in right now. So I have to ask you, what's the use case? I was in call centers, obvious in bots and having self-service capabilities. Where is the customers at right now on psychology and their appetite to push the envelope? And what do you guys see as areas that are most important for your customers to pay attention to? And then where do you guys ultimately deliver the value? Sure. Well, our clients are, I think, are aware of the risks of AI. They are not, that's not the first thing they're thinking about for the most part. So when we come to them with this message, they listen, they're very interested. And a lot of them have begun this journey of putting in kind of governance, compliance, controls to make sure that as they are proceeding down this path of building out AI, that they're doing it responsibly. So it is in a nascent stage. What defines responsibility? Well, you want to be, okay, so responsibility is really having governance. You know, like you have a, you build a robot dog, right? So, but you want to make sure that it has a leash, right, that it doesn't hurt anybody, right? That you have processes in place that at the end of the day humans are in control, right? I don't want to go back to the Skynet analogy, right? But humans should always be in control. There should always be somebody responsible for the functioning of the algorithm that can throw the switch at the right time, that can tweak it at the right time, that can make sure that you nudge it in the right direction, that at no point should somebody be able to say, oh, well, it's not my fault. The algorithm did it, and that's why we're in the papers today, right? So that's the piece that's really complex. And what we try to do for our clients, as Deloitte always does, is kind of demystify that, right? So what does it actually mean from procedures, policies, tools, technology, people? Yeah, I mean, this is the classic operationalizing, so new technology, managing it, making sure it doesn't get out of control, if you will, stay on the leash, if you will. And I guess one piece that I always like to mention is that it's not to put brakes on these new technologies, right? It's not to try to kind of slow people down in developing new things. I actually think that making it trustworthy is enabling the development of these technologies, right? The way to think about it is that we have seed belts and ABS brakes and airbags today. And those are all things that didn't exist like a hundred years ago, but our cars go a lot faster and we're a lot safer driving them.  But first of all, there are some safety technologies that you don't even notice, which is how a lot of AI controls work. They blend into the background. And more importantly, the idea is for you to go faster, not slower. And that's what we're trying to enable our clients to do. Well, Leslie, great to have you on the queue. We love Deloitte, come on, share their expertise. Final question for you is, where do you see this show going? Where do you guys actually hear you're participating? You got a big booth here. Where's this going and what's next? Where's the next dots that connect? Share your vision for this show and kind of how the ego system and this ego system and where you're going to intersect that. Wow, I mean, this show is already kind of pushing the boundaries, you know? We're talking about machine learning, artificial intelligence, robotics, space. You know, I guess the next thing I think, you know, we'll be probably spending a lot of time in the metaverse, right, so I can see like next time we come here, you know, half of us are wearing VR headsets and walking around in meta worlds, but you know, it's been an exciting adventure and you know, I'm really excited to partner and spend, you know, spend time with AWS folks and everybody here because they're really pushing the envelope on the future and I look forward to the next year. The show's small, so it feels very intimate, which is actually a good feeling. I think the other thing in metaverse, I heard that too, I heard quantum. I said next, I've heard both those, next year, quantum and metaverse. Okay, well, why not? Why not, exactly, yeah. Thanks for coming on theCUBE, appreciate it. Thank you. All right, it's theCUBE coverage here on the ground, very casual theCUBE, two days of live coverage. It's not as hot and heavy as re-invent, but it's a great show, bringing all the best smart people together, really to figure out the future, you know, solving problems, day-to-day problems and setting the new economy, the new industrial economy and of course, a lot of the world problems are going to be helped and solved, very positive message space, among other things. Here at Remarch, I'm John Furrier. Stay with us for more coverage after this short break.


the campsite. The only ones here right now this will be girls camp and then if you walk this way this will be honeybee's campsite and a few boys tents and then there's another pad through there where someone will go. I don't know honeybee might go down there and she might put all the boys tents up here since I'll be right there and I can kind of listen for both and I think there's one more adult sleeping over at night so should be good but that's the bathrooms and then down here we hung the slack lines and the hammocks and we're getting whistles for the kids in case of emergencies put whistles on all of them so yeah there's some hammocks down there there's two down there there's two more over there one night yeah there's two there then there's a big hammock over there with the slack lines and then the zip lines over there too the zip line kind of goes like from up here down to here so yeah it's gonna be fun