Hello, everyone. That's me on Twitter. Feel free to follow me and tweets and take photos, stuff like that. No problem. Before I start, I want to say big thank you to the organisers for putting on a really good show and really looking after us today. Thanks very much for that. I'm here in Belgium and I'm all on my own. If you see me around, I'm quite shy. You'll probably see me in the corner hiding somewhere. Feel free to come and say hi. Ask some questions, that kind of stuff. This is my first time in Belgium. Before I came, I knew Belgium was famous for a few things. Obviously, you're famous for your waffles and your beer and your chocolate. I wanted to do a bit more research first before I came. Did you know that Belgium was the first country in the world to have a national lottery? Congrats. You also invented the Smurfs, which I've kept my children quiet on many a weekend. Thank you for that. You've also got more castles than any other country. I'm definitely going to come back and visit and try and take a look at these spectacles. I'm from the UK, as you might have guessed, and we're famous for a few things as well. Our British cuisine, for example, our affordable housing. You can buy this garage, I mean, studio flat, for a mere 136,000 euros. Also, our work-life balance says a guy with a baby and a laptop and a music stand. I don't know what he's doing. Anyway, I'm here to talk about changing software and how to make software easier to change. It's said that the one constant in software is change. How can we make our software easier to change? That's what I'm going to try and talk about today. In particular, one thing, this acronym, OCP, not omniconsumer products, because they make some horrible things. Instead, I want to talk about the open-close principle. What is it? This guy is called Bertrand Mayer. He invented the Eiffel programming language and also the concept of design by contract. In his book, Object-Oriented Software Construction, in 1988, he coined the term of OCP. OCP states that your classes should be open for extension but closed for modification. In this talk, we're going to explore what that means and show you some examples of how you can actually do that in your code. More specifically, Mayer says that classes can only be modified to correct errors. If you need to change or add new features, you need to create a new class to do that. Shortly after, Mayer and his principle came solid. Solid is a set of guiding principles that complement each other that help us have better object design. Solid was created by this guy, Uncle Bob, and it's made for five principles, and they are single responsibility, open-close, Lisk-off substitution, interface segregation, and dependency inversion. Obviously, I memorised it with that and didn't read it at all. So why is the Eiffel code principle important? So it's risk and cost. So it said that if you're changing stable working code, you've got a relatively high probability of introducing bugs into that. So it's less risky if you can just create new code without changing existing code. And also on the cost front, it takes more time to change existing code because you have to look at it, understand it, try to look at what's going on, make those changes without bugs, and then fix those bugs. So it's this risk and cost idea. And when we're changing code, we kind of have these two options, simple versus easy. So simple on the left-hand side, this is where you make your code as simple to understand and simple to change as possible. And it has this kind of graph, so it's cost up on the side and time along the bottom. So in the beginning, it takes a bit more work upfront to make your code nice and changeable. And then as time goes on, you reap those rewards because your code is much easier to change if you have that upfront time invested. Versus easy, and this is where lots of startups tend to fall down, is they just need to hack in, make the easiest change they can possible. And it's really fast at the beginning, your shipping features. But then at some point in the future, you'll get to a point where change is really hard and even the simplest change seems like it takes forever. So in my opinion, you should be gambling on success. So whatever product you're working on or project, you're gambling that it's going to succeed and it's going to succeed for the long term. So you should really be thinking about the simple side of things because in the long term, you don't want pain. So Dave Thomas gave a talk at Lundstar Ruby this year called My Dog Talked Me To Code. It's a really good talk, so you should definitely go and watch that. And in that talk, amongst other things, he says, forget all the rules. So forget solid, forget Sandy Max's five rules. This is the only rule you need. A good design is easier to change than a bad design. And as developers, we need to embrace change. And just like you, I know what it's like. You work on a feature, it gets shipped, and almost immediately they want to change it. And you're like, man, I just worked on that feature. But this is our job.  Let's embrace change. Let's make our code easier to change and in turn make our lives easier. So a slight sidetrack from OCP for a second. So what do you think is the number one reason that makes our programs harder to change according to, well, me? It's this. Conditionals. And the reason I say that is because conditionals are magnets for business rules if such and such do a thing. And at the beginning, when these things are simple, it's fine. But what tends to happen is they tend to be a magnet for your business rules and your conditionals get more and more complex. You end up with more and more branches. And then what happens is they end up being duplicated around your system and this is when change becomes really hard. So if you can, try and avoid ifs. I'm not saying never use an if today, but just try and avoid them. There are other techniques you can use and some of those we'll see surely. If you can't avoid using a conditional, at least name the concept. So at the top here we have an example where we have a book that's on loan and its due date is in the past and it hasn't been returned yet. Instead of having that, try and name that concept. So in this case, in terms of a book loan, this is describing a book being overdue. So name the concept and encapsulate that logic somewhere. It just makes it easier to change, easier to understand. So now I'm going to go through a few examples of how we can use the open-closed principle in a few different ways. And remember that the open-closed principle is about creating new classes without changing existing classes. So first of all, let's talk about inheritance. And this is the original idea that Mayor had of how you would implement the open-closed principle. So it's very simple to start with.  And if I wanted to create a new functionality like a convertible, for example, I can create a new class called divertible, derive off of car, because a convertible is a type of car, add the new functionality that I want, and we're done. So very simple. So, as I said, inheritance was the basis of the original open-closed thinking, but there are some trade-offs. Inheritance is relatively inflexible. It's harder to compose things if you want to mix and match stuff. So we try and avoid inheritance if we can. Now, let's look at some other ways that we can do it. So first of all, dependency injection. So this is where you can pass collaborators into a class and then use them internally. So what does that mean? So here we have an example of a library and it has a method called loan that takes a book. And it does a bunch of stuff, but the thing we're talking about here is it's going to log to file whenever we take a loan. Now, this is not particularly open-closed because let's say in the future you want to change how things are logged. So at the moment we're logging to file we maybe want to log to a database or some third-party service. If we wanted to do that, we'd have to go into this class, change it for that to happen, but we want it to be open-closed. So what we can do is we can extract out the login part into its own class and call it a file logger, for example. And then we can pass that in to the library. So here we've got the constructor that takes the logger. You can give it a default so it maintains its existing behaviour. And then we use that logger internally. So in the future, if you want to change how login is done, we don't need to touch this class. This class stays closed, but we can extend it by creating a new class and injecting that instead. And this is how it would look if you wanted to call it. So using the default, the API that Zach has sent us was before, but if you want to have the option of having different loggers, you just pass them in. And it gives you more flexibility because then, for example, if you wanted to log to lots of different places, you can create one logger that logs everywhere and internally it would just call all of your other loggers. So the pin injection is for when you have a role that needs to be changed for a class. So Sandy Metz did a really good talk at RailsConf this year. So if you want a more in-depth explanation and some more examples of how to do this, then definitely check that out. So the next one is the decorator pattern. So this is just around wrapping an existing object to extend its behaviour. So here we have an example. So we have a method for importing products. It takes a file path, and it reads it as a CSV file, loops through all the rows. It then does some validation. So if the name is present, then it does some stuff. So for this example, there's a couple of problems with it.  So we could use dependence injection to deal with that. We could support other ways of reading files. So I'm not going to talk about that part for now. Instead I'm going to focus on the validation and the conditional. So as I said before, ifs and conditionals are where your business rules, it tends to be a magnet for your business rules. So what's highly likely that's going to happen is that these rules will change. So at the moment it's just the name that has to be present. But in the future you can imagine that someone will say, actually the product ID has to be there, or the description, that kind of thing. So what I really want to do is extract this out and make this class or this method open-close. So how to do that is let's create another class. Let's call it product row, for example. And this can take the CSV row into its constructor and we store it. And then if we add this valid method, we can encapsulate that bit of validation in there. And then we can use that in our class. So as a quicker side, you can dry this up a bit. So if there's a thing called simple delegator, and if you're not aware of what it is, so you drive off of it, it's part of the standard library. And whichever object you pass into the constructor, it will then store that. And then any messages you try and send that your class doesn't know about, it will get delegated to the object passed into the constructor. So this is much cleaner, nicer. So if we now use that in our original example, so I've just split this up into two methods. So the top method now just deals with reading the file from CSV. And then for each row, we just wrap each of those CSV rows with our new product row class. And then in the method underneath, we can get each of those, loop through them, and if a row is valid, we do some work. So that's a bit easier to read, and we've extracted out our validation. But we still have a conditional, so I'd like to remove that. And we can do that by using select, which is on innumerable. So now in our top method, we're just reading the CSV, each of the rows, we then wrap with our new product row, and we use select to just get out the valid ones. And then inside our method that does the actual business, the interesting stuff, it's just a simple loop. We just, for each of the valid rows, we can do some stuff. Now we don't have any conditionals, and we've extracted out the logic around validation. So a decorator pattern is useful when you're separating concerns, for one thing, and also when you don't have control of when an object is created. So in this example, the CSV row objects we were in control, that was created for us by the framework, but we can extend its behavior by wrapping it in a decorator. So the next example is the command pattern. And the command pattern is for encapsulating behavior in a class, and it allows you then to pass that behavior around. So in this example, we have an image class, and an image can open a file, and it will store it. And you could probably have some other things in here, like what dimensions the image has, or what color a particular pixel is, that kind of stuff. But we've also started to add some methods around effects. So we've got a blur method and a grayscale method. And when you see this kind of stuff, it's not a big stretch to think that in the future there will be more effects you want to add. And if you did, you'd have to edit this image class. So what I want to do now is make this image class more open-close and take out these effects. And this is how you'd call it at the moment. So you'd create an image, and whenever you call an effect, it would take the current image, apply the effect, and return a copy of that image. And then you can chain those together. So the API is quite nice, but the class itself isn't open-close. So what we can do is we can create some extra classes, we can extract out those effects, and then if we make a slight change to image, we can add this apply method that takes a list of commands, and then we can use reduce to go through and apply all of the commands. So if you're not familiar with reduce, basically it takes a list of things, in our case commands. It will iterate through all of them, and then it will end up with one single result at the end. And how it works is it has this idea of an accumulator, and you can set the initial value of the accumulator by passing it in to reduce. So in this case, we pass in self, which is the current image in its current state. And then for each time we iterate through, we pass the accumulator as it currently is and the current command. We can apply that command and whatever the block returns, which will be the new image that becomes the accumulator. And then whatever the accumulator is at the end we pass it in to reduce to go through. So now we can use it like this. So instead of having a chain of commands like that, we can have just image.apply and you pass in the class, the command that you want for the particular effect. So the key thing here is that in the future, when there's a new effect, let's say you want, I don't know, some polarizing or something like that, I don't need to change image. Image stays stable, doesn't need to be changed. I create a new class that does that effect and then I can just pass it in here. So the command pattern is all around encapsulating a behavior and then allowing you to pass that behavior around and use it. Okay, so this next example is slightly longer. So hopefully you're ready for this. Okay, so service locator pattern. So this is around looking up functionality at runtime, depending on what your needs are. So here we have an example. So we have a class notification center and it has a method that sends a welcome message to a user and at the moment it sends it via email. So we have this internal method called send email, text an email address, and it takes some kind of content flag so it can go and look up the content for this particular message. And this is fine, no problem. But in what happens when a new requirement comes up, your boss says, right, users love getting emails all the time from us, but some of them want to get their notifications through SMS instead. So we want to add the option for them to pick SMS in their settings. So obviously we do the easy thing. We add a conditional. We say, right, user, look at what their notification method is, their preferred notification method. If it's SMS, we'll then call this internal method called send SMS, which takes a mobile number and again this content flag, and then we'll send them an SMS. Okay, great, fantastic ships. All goes well. Your boss is super happy. And he says, a few weeks later, he says, right, ask on so well, we want to add another option. We've got some super users that want to allow webhooks. So they want to give us a URL and they want to integrate with their Slack or something like that for other notifications. And you say, sure boss, no problem. Let's add another conditional. We'll check for webhooks, we'll do the same thing. So this is code rot to me. And what you see here is that humans follow patterns.  That's the easy path. And this is one of the reasons why I've tried to avoid conditionals because you just see this stuff grow over time. So when a new requirement comes, no, no seriously, right, we're not going to add any more conditionals. This is not the right thing to do. Instead, we're going to create a class. So for each of our notification methods, so for email, SMS and webhooks, we're going to create a new notify class and we're all going to have this deliver method that takes a user, and from that, they can get the information they need to be able to send and they'll take this content flag to get their content. However that works is not important for now. Once we've created these classes, let's do some small refactorings over some time now to implement this and try to make this more open-close and a bit easier. So the first thing is, we'll get rid of all those internal methods and we'll just use these new classes we've created. One. Now step two is let's just clean that up a little bit. So we had a big if statement. Let's just turn it into a case statement. We haven't changed the logic. We still have this conditional. But now it's a bit easier to see some duplication. So you can see we have this new deliver user welcome on all of those cases. So what I'd like to do now is just refactor that out and just remove that duplication. So if we just split the looking up or at the bottom, it takes a user and then depending on the notification method we'll return the class that can deliver notifications in that method, in that way. And then above it in our null, in our existing method we'll just call the method that looks up the class and then call deliver like that. So what I want to do now is just extract that lookup into a separate class because I think that's a separate concern. And so now what I've done now is I've just created a new class called Notifier Registry and that has a Notifier 4 method and you just pass in the notification method. So you pass in email or SMS and that will return the class that can fulfil that. So now this class is now open and closed because in the future when I add more ways to notify users I don't need to change this class. This class will just look it up from the registry. And our registry just looks like this. It's just the big case statement we had before. Now this is not particularly open and closed because as I said in the future when I create a new class that does some stuff I need to come in here and change this. So let's look at how we can make this open and closed. So if you look at this closely you'll see that it's actually just a mapping so we have a symbol equals a particular class. A symbol is a particular class. And in Ruby and other languages we have a construct for this we have a hash. So now we've removed the conditional all together and in our registry we have this hash that maps our methods to our classes and then we can pick out the appropriate class from the registry using fetch. So fetch is a method on hash and if you're not familiar with that you give it the key you want to look up and if that value doesn't exist in the hash then it will return a default. So we're defaulting to email notify here. So the rest of this now is how can we change our hash to be more open close? Again still at the moment whenever I create a new class I have to add that class to the hash. So we'll come back to this in a second. One of the benefits now of having all of these things in a registry means that you can power other parts of your system from the same list. So if we had let's say a profile screen for our user that allows them to pick how they want their notifications we can drive it from our registry. So here we just have the notification registry we pick out all the methods and we generate a drop down from it. So you can actually power other things from your system from the same place. So now the problem we have is how do you populate our registry in an open close way? And there's a couple of options. So first is to treat it as just configuration. And this might look something like this. So maybe in your initializer if you're in Rails or some part of the load loading of your application you just say well I'm just going to have this as configuration. And this is okay. There is a downside though. It means that you have to create a class and you have to remember to add it to this configuration. And you can forget. You can have typos. And there's a way to this to go wrong. But it is super simple and super easy. So there's always trade-offs. Another option is to maybe have some auto registry. So let's get classes to register themselves with the registry and then be available. So one way to do that is to by convention. So to do this first we have to change our registry slightly. So remove the hash entirely and just have an empty hash. And then we have a new method at the top called register. And that will take a method. So that will be email or sms. And the second parameter is the class that fulfills that method. And then we just store that in the hash. So the method is the key and the class is the value. And then we have our same look up and we can do that by adding a new method called load. And this will be called at the start-up of your application. And for a particular folder in your app it can require all of the classes in that folder. We then look at all the constants in your system. Loop through each one. And look for all of the ones that end in Notifier. We can then strip off the Notifier bit and keep the first bit and make that the method. So if it was email Notifier you should strip off the Notifier and just have email as the method. And then we can get the actual class. And then we can call register that takes the method and Notifier. So this would work, but there are some downsides. So first of all your classes have to be named in a particular way. And the other thing that's potentially wrong with this is that you can have classes register themselves where they're not actually Notifiers. Maybe they just happen to have the same name. So you end up with some classes in your registry if you're not careful that shouldn't actually be there. Another way is to allow classes to self-register. So let's see how that might work. So if we make a slight tweak to our registry class and our register method now just takes a class and a class only. And then internally it asks the class what method it satisfies. So if in the case of our email Notifier it would just respond and say it supplies email. We use that as the key in our hash and then the class is the value and then our lookup stays the same. So I've created this module here so it's a bit complex but don't worry about the detail. The important part here is this send notifications via a few lines down. So this would add a class method to your Notifier classes if we included it and it allows you just to supply the key for how your how your classes get delivered. It will store that and then it calls register internally and it passes in self and self at this stage is the class itself. So how this would actually look. So this is an email Notifier. We'd include the module and it gives us this macro. So send notifications via email because this is our email Notifier and then we have our standard deliver method. So here we can call our class whatever we like but the class says what it is, what it fulfills. Now this totally works. So if you were to do this what happens is whenever this class is required it would register itself with the registry and if you power your UI from the registry it would then appear as a new option in a drop down and when you actually go to the Notifier we'd look up the Notifier from the hash and use that to deliver. So it totally works except it doesn't work. If you're in Rails that won't actually work. There's not entirely true. In production it would work because all your classes are required and this would work when that happens but in development and test your classes aren't required all in one go. They're only required when they're referenced and because our classes are never referenced they never get loaded and so they never get into our registry. So you'd always end up with a default and it's due to this setting here. So how can we fix that? So how we can fix it is relatively simple whenever we go to look up a Notifier from our registry we can just check to see whether it's already been loaded and if it hasn't we can just go to our particular directory that has our Notifiers make sure they're all required and then return. We can set this flag to make sure we don't do this all the time. So if you were to do this this would actually work and I've got a small demo to prove it so I'm going to do this totally live coding. So here we have an example so here we have the drop-down where the user can pick the notification method, it saves it and when we click Notifier it's going to send the notification by the selected method and you see here in the log just in red this is the method that was used so that was sent via SMS. So now what would happen if I wanted to add a new way? So here we have an existing Notifier just looks like I've just seen before or I should be typing. So let's say I want to add a new method that sends a notification to all existing using all existing Notifiers so all I need to do is to create a new class so and I can call it whatever I like so we're going to notify all the things we'll include our module that does does our magic stuff we create we add our macro that says how we deliver things we add our deliver method just like all the other classes do and I'll come back to this in a second so I'm just going to create a quick help method here this is going to give us a list of all the existing Notifiers in the registry and we're just going to do a quick check we're going to reject ourselves otherwise we end up with some infinite loop nightmare and then inside Deliver we just loop through all of these Notifiers in the registry we're going to call Deliver on each one of them and so if this method is selected we should be able to notify via all existing methods so I just created a class I didn't restart the app now when I go to the UI because it is driven by the registry I have a new option that I can save it saves it I want to click Notifier it sends a notification to all existing notifications thank you so that was a big long example and hopefully I was able to explain the way that was followable but the point here is that this was to kind of take it open close to the nth degree I wanted to extend our application without changing any code and I was able to do that by just creating a new class put it in the right place and we had new functionality and just talking about how recapping how we populated our registry so there was three different ways and each one has got pros and cons so the first way we looked at was configuration the pro to that is it's super simple and really easy you get a side benefit of that which means you can register different things in different environments so let's say you had something called an external service and in staging you wanted to do something slightly different you didn't want to call the external service for some reason you could use that for your configuration but con to that is there's two steps to it you have to create the class and remember to add it to your configuration there's also the convention thing we looked at that's really nice because you just create a class of a certain name the downside is it has to be named in that convention which sometimes you don't want and also you get the thing where classes that just happen to have the same name might register try and be registered and then the last thing we looked at was self registry where the class kind of broadcast what it supports which is really nice because you just create a class you call it whatever you want and it just works the downside is it can be a bit too magical so maybe you have new people on the team and they just don't know how this thing wires up so they're pros and cons to all these things so the service locator pattern which is what we're looking at at moment this is where you look up behaviour at run time and then when change comes you just need to create a new class so for me this is all about balance and trade-offs and I've gone for a whole bunch of patterns there now this is not a license for you to go out and go pattern crazy and just invent all these patterns and just apply them all to your code so I've kind of used them where it makes sense and just good object design really helps with making your code more changeable so here's a quote from a really famous guy so it may seem like overkill in the beginning but as your code base ages you're going to rid the rewards of good object design and if you want to tweak that it's totally fine and yes I did just quote myself but changing software is more than just a bunch of design patterns what you need to remember is that code is much easier to change if it's easily understood and that means naming things well making things small that kind of stuff also remember that code is red many more times than it's written think of your co-workers think of your future self I've written some code and I thought it was good I come back to it six months and it's like what the hell does this do? I don't even know so just think really carefully when you're writing code make it easier for your future self in particular and also as we saw in one of those examples we follow existing patterns in our code existing styles so what happens is if you start off badly you come back to existing code and you want to change it you follow what's currently there you don't necessarily change into better ways of thinking so you may have spotted a theme through this talk and I want to just say it's really okay to create more classes right make your classes small well-focused, named well and whenever I say this to Rails developers in particular they kind of freak out and they're like where do I put my classes? they're not models or views, I don't know where they go and I would say first of all it doesn't matter, just put them somewhere don't use that as an excuse not to do that but what I tend to do is group classes by functionality so maybe have a folder that's around security or a folder around authentication or whatever it might be and group by functionality rather than pattern that's probably bad I would say so just to finish up some things to consider create classes that are focused on on small things go look at solid if you haven't looked at it already and remember this kind of trade-off between simple and easy immediate short-term short-term wins and I know what it's like if you've got an investor demo in an hour you just have to get stuff done, that's fine but kind of think about it, if you have a bit more time try and think about the future and it's all about balance and trade-offs and some guidelines that I try and follow myself is name things well make things small avoid conditionals and also try and avoid nails so I haven't covered that in this talk but it was really cool so check that out so I work for loyalty line we're currently hiring I have stickers and t-shirts if anyone wants one, come and say hi ask a question, I'll give you a t-shirt, no problem this has been on Twitter feel free to tweet me questions and stuff that's it, thank you


I'm Anthony Charakara, I lead the Computational Extra Science Group at the APS. And today I'm gonna talk about some of the ways that we're using high performance computing and AI to enable essentially new kinds of techniques and new kinds of science at the APS. So before I start, I wanna emphasize that this is the work of a great many people. I just happened to be the person presenting today from the Computational Extra Science Group. So we're a diverse group of about 15 PhDs with expertise ranging from applied math and signal processing to high performance computing, workflows, water non-crystallographers like Brian Toby and Bob Van Dril. And more recently, we've been hiring a lot of early career stuff and postdocs who come to us with an expertise in deep learning and automatic differentiation. I particularly want to highlight the work that I'm gonna talk about today, particularly highlights the work of Taken Bizer and Saugat Kandel and Yudong Yaw and Anaka Babu who were former members of the group but have now moved on to other opportunities. Outside of the CX6 group, I also wanna acknowledge a lot of people at the APS, the Math and Computer Science Division, the Data Science and Learning Division, Center for Nanoscale Materials, folks at Lawrence Berkeley and in India as we start pushing more towards the hardware side of computing. And again, in particular, I want to highlight Tarjo who's a staff scientist at CNN and Henry Chan also a staff scientist at the Center for Nanoscale Materials. And funding for this game from Lab Directed Research at Argon and from a department of energy project called AI at scientific facilities, digital to inference and eco-experiments. So I'm gonna start out by talking about numbers basically, essentially laying out kind of a business case for why AI and talk about some of the data and compute needs that we have today, what we expect in the near future. And I suspect a lot of this will be familiar to folks on Syncotron and X-Files from around the world. And then I'll talk about a couple of examples of how we're using AI to address this. The first for analysis, we'll show how we can use AI to make our analysis workflows hundreds of times faster and sometimes even more accurate. And I'll show how this enables for the first time in a real time analysis on really high data streams, like gigabit per second data streams.  Essentially the idea being, how do you maximize information gain from an experiment in a minimal amount of time? And then more longer term, we are interested in extracting knowledge autonomously from our scattering data. So essentially, how do you directly learn the physics without having to go through a multi-stage process of data inversion, some understanding of what that entails and how that relates to the physical questions that we're trying to answer. So just as a quick overview, if in case you're not intimately familiar with the APS, we are pre-pandemic, where we welcomed users from around the world, academia, industry and government, and we used to see about 6,000 users every year. We operate nearly 70 beamlines, all capable of independent operation, each providing a unique take on materials characterization and many of them are multimodal in nature. And kind of that in turn implies that we need unique data workflows and compute solutions for each of these beamlines because each of them are unique, they're bespoke and they're often more complex instruments than just one particular technique. So the primary motivation is data volumes. So for AI and for the strategy and how we're doing computing at APS. So this is a plot of the data produced by the APS over the last few years. So you're reaching about 10 petabytes or so.  So we're going to see massive increase in data. And if you look across at different facilities, LCLS or across the department of energy facilities in the US in total, we're going to see 10 to 100 X increase in data rates. Another way of looking at this is by looking at the sort of historical capabilities of instruments like this. So the curve in red shows the photon flux and you can consider photon fluxes like a proxy for data rates. The curve in red shows the photon flux over the last 60 years or so. And the curve in green is Moore's law. And Moore's law has been slowing down and sort of flatlining here. And so what you can see is over the last 60 years or so consistently the rate at which we're generating data at synchrotron facilities has been outpacing Moore's law. And this gap is going to continue to increase. And so we need essentially a new way of thinking about how we analyze data at our facilities. Just plugging in a bigger, bigger computer is not sustainable because of this increasing gap. And we do this, we now have a supercomputer plugged into some of our beam lines and I'll talk briefly about it, but that's still not going to be sufficient for our needs. And so AI will become a necessity to fully exploit the instrumentation and the facility itself. And as you're all very well aware, materials characterization very often involves a solution of some form of inverse problem. Whether you're doing imaging or spectroscopy or diffraction or some combination of them, you typically have to solve an inverse problem and inverse problems are computationally expensive to solve. And then the third motivation is how we want to do science going forward. We want to solve problems. We want to study science that we can't do today. So for example, we're interested in why materials fail catastrophically. And being able to study this means being able to image a material in action at multiple length scales, starting from the nano scale to the macro scale to see the origin of failure in the material and how that evolves, grows and leads to a massive failure on the component scale. If you want to do this, you need to have some form of automated steering of your experiment. And you need essentially any argument capable of making these decisions on where to acquire data, what modality to acquire, et cetera, et cetera. And if you want to do this, of course, then you need real time data inversion, right? Even if the data takes 15 minutes or half an hour to be analyzed, that's too slow. You essentially need to be able to do data inversion on the order of seconds or less. So because of all of these challenges over the last couple of years or so, we have been essentially integrating our performance computing resources at our leadership computing facility directly with beam lines and instruments at the APS. So Polaris is a top 20 supercomputer and a portion of that machine, about four petaflops of that computers processing power is now being made available directly for use at the APS. And so we're building up software solutions that tackle a lot of inverse problems. So things like tachography, XPCS, et cetera, high energy diffraction microscopy, and many others are now directly utilizing these machines to do the analysis. So data is moved directly from the beam line to the instrument. You do some processing, you move that back for visualization analysis at the beam line. And this also includes deep learning training at scale and deploying at the edge, which I'm gonna talk about specifically in the next few slides. So the examples that I'm going to talk a bit more in detail are in current imaging, because that's my background and I'm more personally involved with in some of these projects. So current imaging is going to be a big part of the APSU. There are going to be a large number of tachography beam lines. In addition to that, you're going to have beam lines to things like core and surface capturing, atomic, which we're going to do 3D Bragg CDI, polar, doing polarization modulation spectroscopy, various checks beam lines, then combinations of Bragg CDI, XPCS, et cetera, high energy 3D Bragg CDI, and sector nanoprobe which we'll be doing 3D Bragg tachography. So a great number of coherent imaging beam lines at the APSU and all of these are going to essentially rely on us being able to do the data analysis in the case of current imaging phase retrieval very quickly and accurately. So again, just to give you kind of an idea of how much of a jump in capability this is, I think most of the community here is familiar with tachographies, I'm not going to describe that. Today you can measure in the order of a millimeter square at 10 nanometer resolution in a few hours after the upgrade that's expected to be on the order of a centimeter square at approximately 10 nanometer resolution in a day or two. I like to make the analogy, this is the equivalent of mapping out the entire land area of the United States at few meter resolution, except that instead of taking a simple image at every spot, you have to solve this complex inverse problem which is phase retrieval. And so phase retrieval, as many of you know, is an iterative process, right? You apply certain constraints in real space, you apply some contracts reciprocals space and you essentially do a lot of Fourier transforms 2D or 3D to recover the real space image. And the challenge with this is that you do have to do a lot of FFTs, complex to complex FFTs. You need hundreds of thousands of iterations and you typically also need to have many different starts to get high fidelity data because you could get trapped in local minima, it's a non-smooth surface that you're trying to converge to. And it's often very sensitive to the choice of hyperparameters that tune your phase retrieval algorithms like which algorithms, how many iterations of each algorithm, how to apply the constraints and so on. And this is kind of like an expert knowledge, right? It takes a lot of time to build up the expertise required to do successful phase retrieval. So our approach has been to essentially replace iterative phase retrieval with neural network methods that essentially learn to solve this inverse problem of diffraction data to sample amplitude and image in a single shot. So in the case of dichography, for example, you acquire a lot of overlapping data that provides you the oversampling required to recover the lost phase information. You run this through a nitrate of engine of some sort and you get sample amplitude and phase when you're done. In our approach, we essentially learn a mapping from one single diffraction pattern that you've measured from one scan location to what that sample amplitude and phase would look like. So it's a single shot mapping from this to this. And so we don't explicitly use the overlap information in any way but it's used implicitly in the generation of training data and so on. So because this is not no longer a nitrate of process it's just one single shot. You just give the, once you've trained the network that is you give the diffraction data to the neural network and you essentially get out a prediction of what the amplitude and phase look like. This approach is hundreds of times faster than traditional iterative methods and you also require significantly less data to do this. So this kind of process has been, this approach to addressing the problems of phase retrieval has been widely studied by a lot of different people and here's just a few examples from literature. So this example is from electron microscopy from Jean-Marie Miao and others. This is from optical microscopy, I think. Here's another example from single shot dichography. This is from Esther and Giorging and others at Brookhaven doing this for x-rays and all these examples basically you will see different approaches to replacing iterative phase retrieval with neural networks that learn to solve the inverse problem in a single shot. And so what I'm gonna talk about in the next few slides is how to implement these sort of approaches on high rate instruments that we have at the APS and how we can get essentially real time instrument images from these instruments. Now talk about specifically how we need to develop instrument specific or technique specific solutions to some of these challenges. So workload APS looks something like this and this animation gonna show us how it works. So it is a bit complicated so it takes some time to break this down. So this portion of the animation represents the acquisition of the instrument. So you have a focus zone plate scanning a sample you measure the diffraction in the the current diffraction in the far field you acquire a sequence of these images, right? Let's just scan through the sample. We move these images to a high performance cluster Polaris or anything else and we then do phase retrieval and I'll come back to why we still do phase retrieval. When you're done with doing phase retrieval you're essentially left with some training data that you can now do supervised training on. What I mean by that is you have the diffraction data. You also have the images that you have obtained by doing conventional phase retrieval. So you can use that to essentially do distributed training on a high performance computing resource. So you take the diffraction data you take the phase retrieval images you give it to neural network and say learn the mapping from this input diffraction to what the sample looks like. When you do that you're left with essentially a trained model which we then push to an edge computing device that sits at the beam line. This edge computing device is responsible for the live inference that the user sees. So essentially the acquired data is split into two streams. One goes to the high performance computer one goes to this edge computing device and this edge computing device that is running a copy of the trained neural network makes a prediction makes a near instantaneous prediction of what the sample image looks like, right? And so the reason we're still doing phase retrieval is because we're constantly updating the copy of the neural network, right? Samples change the instrument drifts over time and so we want the network to have the latest information as part of its training. But this kind of breaks the dependence on phase retrieval for real time imaging. So right now if phase retrieval takes half an hour or one hour that's okay because how we get the images to the user is through these live, the live inference is through the neural network running on the edge computing device. And so when phase retrieval is done a new copy of the network comes to this device but if that doesn't happen you could still use the old copy of the network for some time at least. And so using this approach you've essentially been able to do live inference on two kilohertz data stream on 128 by 128 detector images. And we were essentially bandwidth limited to one gigabyte per second. It's actually a detected computer's network card couldn't handle more than this and which is why we tapped out of this. In principle we could go even faster and for larger images. So here's a recording of the beam line computer while this is happening. So this is the data that you acquired. The, you can see the diffraction data and some of the underlying structure from the zoom plate. This is the live inference on the neural network. So the neural network gets this image and then immediately predicts what this should look like. And then image on the bottom is a stitching of this as a scan progresses. So this is a scan location and it's going across it's doing it in a format spiral. And this is the cumulative image obtained by stitching all of these scan locations and the predictions of the neural network together. So to kind of summarize this portion of the talk without high performance computing we're talking about days or weeks to do the phase retrieval and image reconstruction and you kind of need the full data set with high performance computing and essentially scaling out this phase retrieval problem for typography to many, many GPUs. We can cut the reconstruction time to minutes to hours but with this hybrid approach where we have both high performance computing and a train neural network running on an edge instrument you can cut the reconstruction time to the order of milliseconds. Okay, so I'm gonna switch gears a little bit to a different current imaging technique for in this case, Bragg CDI kind of to make a contrast between how we approached typography and how we approached BCDI and why we did it. And I also highlight something that we're doing more and more which is to incorporate prior physics knowledge into our neural network architecture and our machine learning solutions. So in 3D BCDI, similar to typography we're solving the phase retrieval problem except in this case, we directly measure a small volume of reciprocal space around the Bragg peak and so we want to do essentially then reconstruct the full 3D image of a nanoparticle or a single grain in a polyprosline material. So we essentially want to solve a 3D to 3D inverse problem as opposed to typography which you're doing 2D images and switching them late. So in this case again, we have a network that in this case a 3D convolution neural network that learns this mapping from the reciprocal space information, reciprocal space data directly to sample amplitude and phase. And again, once trained you do this in a single shot. The difference with this network is that unlike in typography where you have to do supervised training that is where you need images of both the diffraction data and the sample images to train the neural network. In this case, by incorporating essentially known physics that is the forward scattering model we can train this neural network in a completely unsupervised manner. And so the way that works is you have some input diffraction data. You pass it through this untrained neural network it makes a prediction of what the image should look like. Initially this is a terrible image but that's okay. What we do is then we take the object amplitude and phase generate the complex number and pass it through a forward model. So essentially do some support constraint. We do a 3D FFT do some normalization and get what the estimated diffraction data should look like. You then compare that to the input diffraction image compute a loss function and then use that to update the network weights. So in this entire sample loop you never have to actually show the network images of the sample of what the actual particle looks like. All you need is the data that you've already collected at the instrument. All the 3D BCDI data that you've collected over 10 years, 15 years or so on. And you can imagine how attractive this is to us because this just means that by collecting more experimental data so just as the instrument keeps operating we can acquire more and more experimental data which we can use to train more and more accurate neural networks in the future in a completely autonomous fashion. So the network gets better and better and it's done in a completely unsupervised manner. And essentially to kind of reiterate we eliminate the need for ground truth images and training. So in principle, you will never have to do phase retrieval again. So here's some results from this network. So the top row is phase retrieval and the middle row is the raw prediction from the neural network. And the bottom row is what happens when you take the prediction from the neural network and you want the few iterations of phase retrieval. So initially if you had to run a hundred iterations of phase retrieval now you just run like 10 iterations of phase retrieval. And the Chi-square error is listed here in this case lower Chi-square error is better. So although the images in the neural from the raw neural network prediction look very good the Chi-square error is a little higher but if we do this kind of hybrid approach where we take the neural network prediction run a few iterations of phase retrieval you would not only end up being faster but you also end up with lower Chi-square error in the images look as you can see in some cases even superior to phase retrieval which often ends up with some artifacts in there. Okay. And of course we are a user facility and one of the things that we really care about is making this available to our users. So in this case the auto phase and end model is now incorporated into production software that the users use while they're operating the instrument and also can take home with them for their data analysis. So this is a package called cohere that Barbara Frost it can cross harder I've been developing over some years and essentially you know you do there's options to a lot of data preparations preprocessing of the data display and so on but you also have the option now of choosing this neural network prediction instead of a random guess or a flat guess when you start your phase retrieval. And so when you do that essentially you can get in this case you know this was phase retrieval run for 600 iterations it took so much amount of time. Essentially you'll see your dog who recorded this video she cuts you know she just run 50 iterations of ER when you start with a neural network prediction and in a fraction of the time that you take to run phase retrieval from the beginning you have a good reconstruction. And if the video plays out a little more you can see a comparison between the different reconstruction images starting with phase retrieval run as in the past versus phase retrieval run we starting with the neural network guess. So the three things that you see here are essentially phase retrieval the raw neural network guess and this one which is a combination that we start the neural network on a few iterations of phase retrieval and you can see this produces a very pleasing image which we can also quantify with the Keiser error and see that it's the best possible reconstruction even better than phase retrieval while being faster. Okay so I showed two examples of networks that we're deploying at the APS one is TycoNN and one was auto phase NN so this was for typography this was for 3D BCDI and I showed two different approaches essentially to how we train these networks right with TycoNN we do this in a supervised manner we did it in line with the experiment with auto phase NN it's done on supervised essentially learns by itself how to do phase retrieval and it's trained in an offline manner and the reason we approach this is because we're essentially governed by engineering criteria for typography you're acquiring a lot of data sequentially very quickly right you're running a detector very fast with BCDI it typically takes a few minutes because you have to get a full 3D data set which means involving physically rocking the sample stage so inference times for BCDI don't have to be so high if you can do it in the order of a second or so you know that's plenty whereas for typography you basically have to be able to get inference from a neural network at the same speed that you're running a detector right otherwise you just keep ending up with a backlog of images coming in from a detector so this means that our network sizes in turn are also governed by this requirement on inference time so for TycoNN that means having parameter having a very small network we essentially quantize, prune and make the network as small as possible so that we can keep inference time really low with AutoPhaseNN since this is not a criteria we can train a large network we can train a network with more than 10 million parameters and this in turn has consequences on how these networks generalize with TycoNN because it's so small it is basically only good for this instrument and this sample with AutoPhaseNN this is a generalizable network it's large, it gets to learn more things and so it basically is good for convex objects with weak phase and now we're working on extending this basically to more complex objects with like defects in them so they have a lot of strong phase in them and in principle this would mean increasing the size of the network to make it capable essentially of learning broader space so I talk a lot about current imaging but that's not the only kind of technique where we are applying the sort of philosophy of training on high performance computing and deploying on edge computing devices this is a network of Bragg and N that's ancient Louis who's now at Amazon Hemansharma who's in the CXX group and many others at the APS have developed and this is a neural network that essentially does very accurate peak fitting for high energy diffraction microscopy and so it's actually a convolutional neural network that takes patches of the detector and then predicts to have very high subpixel accuracy what the peak location is and this network is 200 times faster and more accurate than current pseudo white approaches that have been used to do peak fitting and just like with dichography essentially they've deployed this network on an edge computing device to process streaming data so you read the data right off the detector so she has a video of that happening and identify the peaks and then fit those to high peak high accuracy okay so I'm gonna switch gears a little bit and talk about how this real time analysis that AI enables in turn lets us do more exciting things with instruments in particular how we can start steering these experiments to target features that we're interested in and this solves a more general problem right so you put a sample in and you don't know what's in the sample and the question that we typically face when you start an experiment is how should you acquire data so as to maximize the information game in a minimal amount of time and so in the approach that I'm gonna show what we do is we sample a few points initially randomly and then we pre train a network that essentially kind of learns where to sample from given a set of measurements where to sample from so as to maximize this information game and I'll show again we put this in the loop in the experiment so that these decisions are made very fast and the instrument is tiered very quickly to target locations and essentially we can use this approach to reconstruct images with far fewer points than you could if you were to do this in a high fidelity raster scan or other kind of fixed scan so the particular sample is a tungsten diselenite film and we measured the signal at the OO8 brag peak in a scanning diffraction modality at the nano probe beam line at the ABS so this is kind of like how our edge setup looks like this is for tachography as well I guess I didn't show the image earlier so this is looking upstream at one of our beam lines so we put we use this NVIDIA Jetson mini computer so sort of like a think of it like a Raspberry Pi except it comes with you know CUDA cores and the ability to run very fast neural network inference it's inexpensive it's low power and it's sufficient to keep up with a lot of our compute or edge computing needs when it's running a train neural network and our workflow goes something like this so you know your sample a few points randomly you we have this train neural network that essentially then suggests the next 15 most important points to sample from we run some root optimization through Google's OR tools to optimize the scan path over 50 points require those points we compute a change in the image and then loop backwards to figure out again the next set of points from that the neural network suggests to sample so all of this is in a completely automated fashion so you know your user will start the scan here and then this cycle will continue until the change in the image is minimal and you're done with the scan the cool thing about what we did was we trained this neural network on an utterly unrelated image it's this famous image of a cameraman and essentially it was trained on 100 different masks at different sampling rates so you know we block out different portions of the of the image the neural network has to learn where to sample from so as to get closest to the to the actual ground truth image and this is an approach that we have essentially adapted from this paper from people at Argonne and from Charlie Bowman at Purdue and these are the results from that so the on the left you have the full resolution image that you would have acquired if you were to measure every single scan location and this is an image that is taken at a 100 nanometer stamp set the results on the right show the corresponding image if you were to if you were to use this AI guided acquisition approach the plot on the bottom right shows the locations that were chosen by this neural network to scan so you kind of see it starts focusing on edges of features where things are changing the most and it ignores portions of the sample that are relatively flat but there's not a lot going on and so this is the image that's acquired with you know like 20-25% of the number of scan points as this image and you can see it acquires in all of the features in the image so again here's a video of this running at a beam line so this is the 26th beam line at the APS and this is a live recording of the computer so the bottom row is the points that are measured by the neural network the image here is the interpolated image I think the text will walk you through the rest of the video so this little red dot is the live location of where the beam is and this path in white is an optimized route through all of the points that the neural network suggests is the most important to sample from this is the live detector image so you see it discovered a new feature and so spends more time measuring around it just to make sure it gets all of the aspects of that feature we can also do this sort of automation in experimental and intelligence sharing of instruments not just for the experimentation side but also for the instrumentation side and so Luca Rabafi in the optics group and Sagat Khandle and others have been working on this thing called autofocus essentially the idea of being how can you automate the process of you know alignment and focusing of mirrors and they're doing this by essentially working with a digital twin of the beam line in this modeling software called Oasis so they develop a lot of these models in simulation and then you can deploy these instruments and they've been getting very good results for this other related work people like Yinsan, Nikita Kuklev, Ihar Lobak and others at the APS they're looking at AI based sharing of the accelerator itself and they're looking at AI applied to increasing the efficiency of the operation and also being able to predict things like power trips ahead of time okay so with that I raised the end of my talk I hope I've convinced you that AI would be an integral part of APSU beam lines we talked about AI for analysis and I showed some examples of how we can accelerate and essentially allow real-time imaging for the first time showed an example of autonomously steering the microscope to features of interest that we're interested in and this one I've not talked about today in the interest of time but you know more long-term we're interested in learning physics directly from the data we measure and again putting these into production is integral to what we do writing the papers only part of it this is going and some of our AI for analysis techniques already going into production like we saw in the next few years we expect some of our steering models to go into production as well and a lot more long-term we hope to see some of our AI for knowledge models in production all of our code data train models everything is freely available and always will be and we welcome to try these out and let us know if you have any issues thank you and that'll take any questions