Hello everybody. Okay, so this lecture in materials is covered under a Creative Commons license that allows it to be freely shared and redistributed, so do with it whatever you like. So I'm going to give you a follow up on your lectures from yesterday. We're going to talk a little bit at a deeper depth on pathway and network analysis. Most of the examples I'll show are drawn from cancer, but everything I talk about is applicable to any model system or disease process. So the objectives for this module are to understand the landscape of pathway and network analysis. But I'm only going to touch very briefly on individual tools and try to talk about the principles underlying them. We're going to talk about where pathway and network data comes from, how it's collected, what the reliability of the various sources are, the analytic approaches to data analysis how you visualize and integrate it, and talk about some applications of pathway enrichment analysis. So, you probably know why we're interested in pathway analysis and that's why you're here. It's because high throughput data is very multi dimensional and hard to grasp. When you're sequencing thousands of genomes and tens of thousands of genes. It can be very difficult to interpret the results of experiments such as a perturbation experiments by projecting this multi dimensional data onto a smaller number of biological pathways. So you've increased the statistical power of your analysis by reducing the number of hypotheses from the thousands to the, the dozens. And you're able to tell biological stories, such as identifying hidden patterns in gene lists, or creating predictive models to explain your observations. And you can do such things as predict the function of annotated genes, based on where they associate in, in the pathway. So, this is a very broad type category pathway network analysis is is broadly defined as any analytic technique that makes use of bio of biological pathway or molecular network information to gain insights into a biological system. And so very large and rapidly evolving field with many different many different competing approaches. But basically to do any of these tests, you need two ingredients you need your data, high throughput biological data, such as altered genes proteins RNA is in a derived from an experiment, and you need a information source of known pathways and networks.  So here we're looking at a simplified version of the first steps of the EGFR pathway with the its ligands, the receptor, and the first dimerization step, and it's laid out in a biologically intuitive manner, which reflects the way we think about biological pathways. In contrast, a network consists of less well understood relationships among among the various actors and that the main feature of this is that it's not semantically or ordered. We can't as easily look at this and try to figure out where the start and end of it is the network is actually probably more like what the cell sees. It doesn't it doesn't recognize sequences of events, but the pathway is an easier way to understand it for our limited minds. So there are pathway databases and network databases and some which have both. The databases have the advantage of usually being curated. Typically they're created by a team of curators and authors who scour the literature and reduce the experimental findings into a textbook style view of biological processes. They have the advantage of offering a human interpretable visualization of the pathway and tools that make use of the cause and effect along the pathway disadvantages of these is that they don't cover the genome very well. The best pathway databases maybe cover half of the genome. They are always a little bit out of date, because they're waiting for the review article that comes in with a big summary of the pathway. And so they're going to be missing latest knowledge. And a bigger problem is that each database has a different idea of where a pathway starts and ends. So they'll have the pathways will have different names, and they'll have different different constituents so it can be hard to relate the results from one pathway database to another. A typical pathway database typical pathway databases are reactome and keg. I'll be talking about reactome a lot because I'm one of the PIs for the reactome and initially conceived of it many years ago. These both these databases explicitly scribe a biological processes as a series of ordered biochemical reactions. And here is the basic reactome model can use as a similar model, where the core of every step is a reaction. It has a series of inputs, and the reaction transforms them into a series of outputs. And along the way, there can be regulatory molecules or enzymes and catalytic activities, which affect the effect the rate of the reaction. And this basic model can cover a whole variety of things proteins, small molecules complexes, and even such things as changes in the topological location of a macro model. If there's a transporter step in which a small molecules moved from the cytosol into the interstitial space, they're going to be an input, which is the small molecule inside the cytosol, and an output, which is a small molecule in the interstitial space, and each of them is treated as a separate related object. So, an example, a very popular example is keg the Kyoto encyclopedia of genes and genomes this is a huge library of information on hundreds of species contains the genome fully sequence genomes of many species proteins pathways chemical compounds diseases. And part of keg is its pathway collection, and that's a collection of manually drawn pathway maps, representing knowledge on spanning metabolism cellular processes, human diseases, and drug interactions. And keg is partially open, you can browse it for free. But if you require want to do data analysis on it or download it you need to pay an annual subscription fee for it. And his should be kind of a familiar picture they'll see a lot of these. This is the keg version of the cell cycle. One of the unique features of keg is that each of its options. Sorry, I'm getting a message here. I don't know if that was meaningful. The one of the key features of it is that each of its genes is identified by its enzymatic enzymatic activity using its EC number. Reactome is complete open source and completely open access there. It is free to it is free to use it's free to download you can set up mirrors of it you can create your own reactome. It is a 20 year old project that was I initiated with you and Bernie from the European bioinformatics Institute. And it is a pathway database encompassing metabolism signaling and many other biological processes. It has a team of about a half dozen curators who work with authors in the field to curate paper literature reports in a machine readable format. Every single statement is supported by a supported by the primary literature and across references many other informatics databases. It has a series of data visualization and analysis tools built into it with all as well as being available as our and Python libraries and the series desktop tools. It provides a Google map style interactive reaction diagram. It has textbook style illustrations with clickable overlays, and it lets you do things like find the pathways, give it a gene list you could find the pathways which can, which your genes are participating in. You can calculate the over representation of genes or proteins in a pathway, and you can do mapping from one species to another. And this is a typical reactome page. I've searched searched here for cell cycle. I get a little picture that was drawn by one of the curators or an author and a, and a human readable summary of that pathway underneath that I can dig into it, and I can start getting a detailed pathway diagram. Again, this is interactive and allows you to perform analysis directly on the diagram and a lot of database fields. The, so we'll go into reactome a little bit more later, but now I'm going to switch to networks. So in contrast to a pathway networks are much less attractive they they but very very information dense. Whereas pathways capture only the well understood portion of biology things that could go into a review paper. Networks cover high throughput data and less well understood relationships, including things like genetic interactions. You knock down a gene in yeast, and something and the expression level of another gene changes. It's not stated in some way but we have no idea what the mechanism is, but we know there's an interaction between physical interactions such as experiments in which you do a combino precipitation to develop to get lists of complexes. We know that all the gene, all the proteins in that complex interact with each other. We don't exactly know what, what their, what their role is. Co expression data gene ontology term sharing their adjacency and pathways, anything that describes a bind molecular relationship can be put into a network. So network databases can be built by built via curation and many are they can also be built automatically from high throughput experiments. They have extensive coverage of biological systems they may cover the vast, the majority of human genes. But the relationships and the underlying evidence between the underneath these interactions is much more tentative, and one of the problems with networks is they tend to have a lot of false positives. There are a series of curated networks that that in which some of the false positives have been removed by applying strict standards for evidence gathering. One of the ones that I tend to use our bio grid, which is a has interactions from 80 different species and has 2.6 million interactions among 87,000 genes intact which is one of the year, which is a European bioinformatics project that react to and collaborates with quite, quite strongly. This has 9,000 species and 1.2 million interactions. So, locally, Jean mania, which is one of Gary Bader's projects is a compendium of 2,800 gene association networks, representing 660 million interactions across nine species. And I'll give you a little look at that. So here's typical search on intact. I searched for everything that interacts with TP 53. And it found something like 9,000 different interactions, which I then filtered down to a smaller number of physical interactions that I could visualize. And it's showing the interaction graph here. I don't know how well that shows up. You can't see the lines very well on that screenshot unfortunately. And then with this, you can do things like given a given a gene list that you've come up with, identify pathways that contain them or networks that contain them both and see how far they are far away they are from each other. Jean mania is has a much better visualization than either intact or bio grid so it's the one I tend to use when I can. It has 2800 networks that have been brought into a framework in which you can do a variety of things but the most interesting thing is if you have a gene list of say a dozen genes. It will, it will create a network for you that contains those genes in your list, as well as other genes that are necessary to link them together. So it allows you very rapidly to search through it and find potential potential pathways among things that relate your genes. And you can select the type of network to interrogate, you can look at only genetic networks or genetic class physical networks, or things that are related by gene ontology terms or co expression terms. So very flexible and very useful tool for doing data exploration. So I'm going to talk a bit about the what the visualization analysis tool landscape is for biological networks. There are a series of a series of desktop tools that are very useful. Disont, which is shown on the left is designed, particularly for metabolic interaction networks, and it builds up color coded color coded by function on networks of that contain a gene set of interest. Navigator is is similar but it gives you a 3D view, you can rotate a 3D representation of your of the network that you've built up. And then side escape which I think you're all familiar with does pretty much everything that you that you might want and has a very rich ecosystem of plugins that provide both simple to a very advanced network analysis. I'm going to talk a little bit about the different types of pathway network analysis that you may see there are basically three types of modeling tools. So one is the most basic one is gene set enrichment analysis in this. In this type of analysis you have a set of genes that came from a high throughput experiment such as an RNA expression experiment, where you've got some genes which are in what were up regulated and other genes which were down regulated, for example. And what these tools do is it divides the divides the biology into a series of bags of pathway bags all the genes that are involved in cell cycle, all the genes that are involved in apoptosis all the genes that are involved in immune signaling. And identify uses their number of algorithms which are able to identify genes which are over represented, or pathway pathways which are over represented in your gene set to give you a sense of what biological processes have been changed in your experiment. And so this is this tools are essential for taking a large, large data set and identifying the identifying the biological processes, which may be altered in that, in that experiment. The second one is is used more for discovery. And these are sub network construction and clustering modules where you have a, you have a gene set of interest, and you want to construct a network, which relates them. And this is useful for finding new pathways that are altered in in cancer or my disease, or identifying a clinically relevance tumor sub types, which differ by which pathways or which sub networks are activated in that in that disease process. And finally, the most sophisticated and hardest to use of these tools are ones which do pathway based modeling. And these actually construct use the information underlying pathway or network database to construct a model of your bio of biological process, such that you can make predict you can predict on it. So having analyzed, for example, a single cell perturbation experiment which a series of genes in a pathway were knocked up or knocked down, you see what the effect downstream of the downstream effects of that perturbation. These systems will build a model, which then allows you to predict new biology, what happens when you knock up or knock down a gene, which was not in your experimental data set. So these are used are for things like prediction precision medicine prediction in patients. If I give this patient a drug, given their, their mutation profile, do I expect that drug to be effective. Okay, so let's talk a little, we'll talk a little bit about these these techniques. So, enrichment of fixed gene sets I believe you have covered that. Yes, yesterday, but just as a reminder, this is the single most popular form of pathway network analysis. There are two bifurcations in this is simple ever representation analysis, in which you have to choose a threshold, when between threshold in your data set. So for example, in a expression data set you have to set a threshold for which genes are up regulated versus which genes are down regulated to bins. And then it'll tell you it'll find pathways or pathways which are over represented in the up regulated set. The problem with over representation analysis is you have to choose a threshold. And it's not always obvious how to do that functional class scoring, which the exemplar of that is a GS EA, which I think you use yesterday is that correct. Okay. ranks pathway, ranks the pathways by the degree of over representation in the data set and provides a kind of a the ability to form a continuous function across them doesn't require you to find threshold and gives you a series of p values and false discovery rates for the for the pathways of interest. The advantages of both these techniques are they're easy to perform and they have lots of good end user tools, and the statistical models are very well worked out. And you accurate, pretty much accurate false discovery rates and p values disadvantages of any of these systems is there's a fundamental assumption that you can take the take human biology and partition it into non overlapping sets of pathways and so we know that isn't so that you have a series of bags. And we know that isn't true pathways are heavily overlapping. And in fact, when you use any of these tools, you usually get a series of overlapping enriched pathways which contain some of the same genes and you have to apply additional tools in order to try to figure out what is an independent observation and what is. What is not. Nevertheless, this is where most people start and it's adequate for the vast majority of uses. So, here is an example of pathway enrichment done with reactomes online tools. Go to the reactome website and click on analysis you get a page like this which offers a variety of online tools. The simplest one is the gene set enrichment analysis in which you cut and paste a list of genes and optionally a list of values associated with them such as expression levels. And here I've cut and paste a list of genes that were taken out of cosmic, which is a cancer pathway database, which is a list of cancer associated genes to see where all the cosmic genes fall on a pathway map. And this is one of the visualization this results that you get from this analysis. This is a Voronoi plot of the also known as a bubble plot foam a plot of all the pathways known to reactome. We've highlighted the ones which are over represented in that gene set, and we're more or less getting what we expect for seeing gene expression signal transaction developmental biology DNA repair cell cycle and the immune system, among other things, which are over represented cancer data set, which is exactly what we expect. And then we can zoom in on this and go down to the pathway level and actually see which genes were in our gene set and how they're related to each other. And then there's a large table underneath this of the p value and false discovery rate, and a lot of underlying information you can browse through. Okay, so that's, that's gene set enrichment analysis. Okay, now we'll talk about de novo subnetwork construction and clustering tools. So these are tools in which you start with a network, typically, and you apply a list of all your genes and proteins to it to identify topologically unlikely configurations that is in your genes, or protein or RNA set. Is there a subset which are closer to each other on the network, then you would expect by chance, because those are likely to be interacting with interacting with each other and shed light on your biological question. Then these algorithms will extract the clusters of those unlikely topologically unlikely configurations, and annotate them with a variety of biological annotation sources such as gene ontology terms. So here's an example of doing this. I'm afraid this is a really old example from 20 from 2010, but still a good one. One of the things that reactome has done is to take its path curated pathway data, turn it into a network, and then add high throughput data from the other net other interaction network database sources, and to do a little bit of machine learning in order to remove false positives. And so this gave us a what we call a functional interaction network that has 11,000 proteins and 270,000 interactions in it. It's relatively small by reaction network standards, but because we filtered it, it has a relatively low rate of false positives. Okay, and then, from this we applied a set of genes which are were up regulated in in triple negative breast cancer identified the topologically unlikely configurations of those genes, and it split the, the big network into a smaller network consisting of a series of genes which are interacting with each other frequently, and have cross connections to other genes so we clustered this into a breast cancer disease module of 10 to 30 genes. Actually, I'm going to give you, yeah. Alright, so this is just a information on the functional interaction network, and our calculated false positive rate is less than 1% in this are false rate negative rate is very high 80% because we're only covering a fraction of the gene. Okay, so here's something that you can do with it. We've looked 52 pancreatic cancers and identified their somatic mutations. And then if you just look at the, at the distribution of the mutations, the K, the KRAS gene, which is the very first one and those little crowded. You know, is there in over 90% of the cancers, and then there's a very, very long tail of less frequently under less frequently mutated genes. Can we use the information from this tail to to understand more about the biology of pancreatic cancer. So what we apply that mutated gene list to the the reactive functional interaction network. Sure enough, we get a series of clusters out of this. There's a big one that has KRAS right in the right in the middle. And it is, we annotated it with gene ontology terms and we get or be signaling and FGFR signaling and axon guidance is a big signaling module. And then we get smaller modules with hedgehog signaling, calcium block binding. The spliceosome axon guidance. These are all things which are most of these are known from the literature. Some of them were some of them were new, new processes. And then we can take the modules and we can cluster the patient samples by which modules are active in that particular patient. And sure enough that clusters actually pretty well the patients are here on the y axis and the modules are there on the x axis. And we identified four different types of pancreatic pancreatic cancer, which are distinguished by which modules are active. And then we looked to see if any of these tumor types were predictive of survival or response to drugs, and they weren't, but when we applied the same method to breast cancer. That's what I was referring to before, and did the same analysis. We found a module which is actually expressed. This is a patient whose expression is correlated with survival. This is a Kaplan-Meyer curve. These are months in which the patient was disease free. And patients that have a low expression of genes in this module do better than those that have a high expression. And when you look at what's actually in here, it's cell cycle and aurora bees and aurora bees signaling just associated with mitotic rate. So this is actually a pretty good prognostic marker for breast cancer discovered through Dinovo subnetwork cluster. So here are some of the clustering algorithms that you can use. Genemania is the easiest to use and really, really useful. And then there are two site escape apps, which I can recommend. One is called Hyper Modules, and the other is the ReactomeFI network, which was used to generate those examples before. And then there is another very popular algorithm called Hotnet from Ben Rafael's lab at Princeton, which uses an interesting model in which the network is represented as a metallic lattice. And it puts mutated or your genes of interest and makes them hot, and then it uses a heat diffusion model to find neighbors. I don't know why it works, but it works quite well. Okay. All right, now we're going to talk about pathway based modeling. We have a list of your alter genes, and you apply them directly to biological pathways, not to networks, in an attempt to try to discern the biological relationships. And this basically shades into systems biology. We're trying to make mathematical models based on the prior information from the pathway to explain the empirical observations. So these are, these start to get into a, into heavy math, but so for metabolomics and biochemical systems, there are tools that represent the pathway as a series of partial differential equations. One of the more popular tools is CellNet Analyzer, and it's used for metabolic relationships. For kinase cascades, people use the tools net forest and network kin. These are network flow models which model the flow of information through a kinase cascade and can be used to construct models of kinase signaling. And there's some tools which are attempt to reconstruct the transcriptional regulatory network, based on high throughput RNA seek or micro array data to identify key, key nodes in the network, such as transcriptional master regulate master regulators, and the oldest and the most of these is from Arachne, from the Colofano lab at Columbia University. And then there are probabilistic graph models, such as the paradigm tool, which is from David Housler's lab at UCSC, which represents the pathway as a series of influence nodes, where information comes into one reaction and goes out to several others, and using a Bayesian probabilistic network attempts to infer how the, how each node will influence will influence the next the next one. So it propagates activation through the network. So here is a little look at how paradigm works. You start with a very, here's an example of very simple pathway with just two genes and an output apoptosis, one's a negative inhibitor, and the other is an activator. It then expands this into all the steps of the, the central dog. So MDM two actually has a gene, and it makes an RNA, and it makes a protein, and the protein becomes active and it makes an active protein and TP 53 is a gene that makes an RNA makes a protein. And it makes an active, it makes the active TP 53 protein, which is inhibited by the MDM two active protein. And so then it breaks this, it expands this graph into all the macro molecular things and says okay, if I have introduced a mutation into MDM two, then this is decreasing the activity of the gene which decreases the RNA which decreases the protein, less active protein, and therefore it's going to inhibit apoptosis to the very complicated way of see of modeling something that you can, you can see very clearly, but when you get to large pathways then becomes useful. Okay, and this allows you to model multiple changes at different levels, including mutations copy number changes, changes detected by proteomics changes that are that are detected by post transcriptional post translational processes and paradigm has been very, very successful. And here's an example in understanding but interpreting biological data is an old publication from TC GA in which paradigm was applied to glioblastoma multiforme, and it very rapidly identified for different GBM sub types which had clinical clinical correlates. And here's a question I end with a with an experiment designed to get at a core question which has not actually been addressed, which is pathway modeling sounds great people use it. But can you actually use pathway curated pathways to predict biology. A couple of years ago, I went to literature to see if anybody had actually done empirical experiments in which they made a they took a pathway from a curated pathway database, stared at it, and used it in a systematic way to make predictions about what was going when you knocked out or knocked up different genes in the pathway and I didn't find any systematic studies in it so the react with the reactome staff we did this experiment. And so we asked two things. If you have a, if you have a curated pathway. So can a expert gazing at the pathway diagram predict what happens when you knock out some of the genes that are at the top of the pathway and can and can an algorithm do the if that's successful can algorithm do the same thing. I know it seems obvious and you think it's going to work but we've never tested. And the step one of this experiment is that we selected 10 cancer related pathways that were well annotated. And from each of them we gathered key input and output pairs. So the curators decided using a series of rules that we put together, which are the key inputs that we wanted to test, and which were the key outputs from the pathway which were the key inputs from the pathway activation. And overall we collected 4,968 pairs from those 10 pathways. Next, the curators went out to the literature to identify functional genomics experiments in which the key input was perturbed, one of the key and key inputs were perturbed, and they measured the effect on key outputs. And that was by CRISPR, or RNA experiments, and we were careful not to use papers that had already been curated into reactome because that would be cheating. And of those nearly 5000 test pairs. We found 531 papers, which reported empirical results on 847 of them. Next, for those 847 cases, the curators stared at the pathway diagrams and tried to make predictions. And we had a rulebook for how they would do this we had three curators doing it. It was an experiment in which independently, they, they made predictions we found high correlation among the three curators, and so on the basis that we divvied the 847 up among three different people to make their predictions. At the same time, a talented computer science scientist in the laboratory created a simple graph based analytic tool called MP Biopath, which basically does Boolean modeling of the pathway to make predictions to make predictions automatically. And it has a very simple algorithm at its core. All it does is it counts between the, the source, the input and the output, it counts the number of positive arcs, and the number of negative arcs, and figures out from that whether the overall effect is going to be up, down or no change. And then it has some rules for what happens when when you have things combining and forms. Okay. So the results were actually, I think pretty, pretty good. So the curators accuracy was 81%. So overall 81% of the time when a gene was knocked down, and it was predicted that the activity of the target would go down. They were, they were correct. The algorithm MP Biopath was also not bad. It wasn't as good as a curators but it was correct 75% of the time. If you just randomly guessed it would be 33%. So these results were significant. The largest. So what we're looking at here is a summary of that the greens are true positives. The oranges are false positives and the science are false negatives. The outer ring are curator, sorry, the inner the outer ring is MP Biopath the algorithm, and the inner ring are the curator predict your predictions, showing that the curators are doing a little bit better than the, than the algorithm. When we looked at where the errors were, the largest source of errors were here the false negatives. And when we looked at those in detail, it turns out that those are cases in which a key step was missing from the curated pathway, we hadn't kept up with the literature. So there's biology missing. The largest source of false positives was one in which the curators or the algorithm predicted that the, that there would be a change in that it would be up. But in fact, the empirical results were that it was changed but it was down. And these also turned out to be usually related to missing genes in the path in in the pathway where there was a feedback loop of some sort, which was changing the direction of the relationship. There was excellent core concordance between the algorithms prediction and the humans prediction. This is an ROC curve. How many people here are not familiar with ROC curves. This is the rate of false positives. That's the rate of false negatives where we're treating the humans predictions as being ground truth. In this case, random guessing would give you this curve. Perfect guessing would give you a curve that goes up. You have zero false positives and 100% accuracy. So it would look like this. We're getting a fairly high steep curve here, which is can be interpreted as being 91% concordant. So the conclusion from this was that, in fact, pathway databases are useful for modeling biology can be used even very simply for making predictions, but we need more funding to, to do more pathway curation. That's my, my pitch when I write grants. So conclusions from and takeaways. So pathway analysis allows you to discover biological processes hidden in large scale data. You have a plethora of databases and tools to choose from the curated pathway databases after many years are now reaching levels of completeness that allow for accurate prediction of perturbations. And the field is really right for machine learning approaches because we've got all these great curated data, curated data sets and interesting relations. And so I've left you with links to the various tools that I mentioned. I'm happy to answer any questions.


Good afternoon everyone, not a good afternoon. He's the only one person who would like a response to me All right. Well, good afternoon. I'm Eckhard Groll in case you don't know me I'm the William E. and Florence E. Perry ahead of the school of mechanical engineering as well as the rally professor of mechanical engineering I'm excited to see so many of you here today for our Purdue Engineering Distinguished Lecture It's great to see you would like to give you a little bit of an introduction to what this is all about but beginning in 2018 the Purdue Engineering Distinguished Lecture series invites world-renowned faculty and professionals to Purdue Engineering to encourage sort-provoking conversations and ideas with faculty and students regarding the grand challenges that we're facing and the opportunities in their fields Beside presenting a lecture to a broad audience of faculty and graduate and undergraduate students They engage in an interactive panel with faculty and students as well. So after the lecture We will actually have a panel discussion and I invite all of you for the panel discussion as well So with that the formal introduction of today's speaker will be done by Arvin Rahman He is the dean of engineering and he said I should keep the introduction very Short, but I would like to mention that Arvin and I have been colleagues for many years And I've known well and I'm so grateful for him to lead the college So with that Arvin come on up and do the introduction. Thank you Good afternoon warm welcome to everyone who's here in person and to everyone watching this online as we stream this event Yeah, it is my honor to introduce today's distinguished lecturer dr. Karen Wilcox dr. Wilcox is the director of the tinsley Odin Institute for computational engineering and sciences at UT Austin where she also serves as the associate vice president for research and She's also concurrently professor of aerospace engineering and engineering mechanics there at UT Austin While she does that she also finds time To be an external faculty at the Santa Fe Institute Now dr. Wilcox is Internationally renowned for her work on reduced-order modeling and scalable computational methods and her research has had tremendous impact on the world in particular It has been incorporated into aircraft systems design and into environmental policy decision-making She has been the recipient of numerous recognitions She's a fellow of AI double a fellow of Siam fellow of the US Association of Computational Mechanics and most recently member of the National Academy of Engineering Would you please join me in welcoming dr. Karen Wilcox to the stage All right, thank you very much for the kind introduction I Did not think to unlock my computer before all that happens and there we go All right, it's really wonderful to be here to see some old friends Sing up some younger friends and also to meet new new people new faculty And to talk with some of the some of the students I was trying to remember bill when I was last at Purdue was so long ago that I can't even remember But it's been at least a decade. I think so all right, so today. I'm really happy to Share with you some thoughts and some glimpse of some of the work that we've been doing over the past years With the long title here from reduced-order modeling to scientific machine learning how computational science is enabling the design of next-generation engineering systems and I really want to through the talk touch on these four things I'm going to start off really motivating the work that we do through the lens of digital twins Which I'm sure is a phrase as a topic that many of you have been hearing Thinking about what the excitement the potential is around digital twins But also what some of the challenges are around digital twins Particularly when it comes to the complex systems that we as engineers think about then I'm going to dive in and Tell you just a little bit about some of the reduced-order modeling work that we're doing and particularly the method that we call Operator inference and I do have to warn you there is going to be a little bit of linear algebra ahead But really just try to convey you to you some of those building blocks that are so and so essential to Realizing the potential of of digital twins Okay, so let's let's start out. So I have to ask is there anyone here who has not heard the phrase digital twins? Yes, I've started to ask the question in the negative to get the more meaningful data collection Of course, this is a hot topic. It's something that we're hearing about in a number a number of Venues when I think about digital twins and particularly when I communicate what is a digital twin to an audience of non-experts I often like to draw on the Apollo example and for those of you haven't seen it There's a blog from somebody at Siemens that kind of recounts the story of the Apollo 13 crisis and it's kind of it's I guess fitting that we're sitting here in the Armstrong building to talk about this So Apollo 13, I think we're all familiar with that. Thanks to Hollywood where the spacecraft suffered a malfunction up in space became damaged became crippled and Of course, the challenge was trying to figure out what to do and how to bring the astronauts back safely to earth So the story goes that during the Apollo program NASA when they launched a spacecraft up in space would also keep a physical simulator on the ground in Houston And so during this Apollo 13 crisis again The story goes that NASA were able to take the data From the physical spacecraft stuck up in space Feed it into the simulators and these were physical simulators These weren't digital simulators back back then 1970 but physical feed the data and adapt the simulators so now the simulator is mimicking what's going on and it's physical counterpart and Run the what-if scenarios that ultimately inform the decision-making that brought the astronauts back home safely to earth So I really like this example Not because this is a digital twin again It was a physical simulator because this example shows the power of what happens when you take models and Today we have very powerful mathematical models that are encoded as digital simulators You take models and you combine that with data from a physical counterpart and not just data That's collected once but data that is collected on an ongoing basis That is reflecting the changing nature nature of a physical system You put those together and then what that gives you is a very powerful way to drive decision-making So if we fast-forward today and we think about this the sort of paradigm of models and data and Driving better decisions. Let me give you another completely different example To think about and this is this is something that I have become Very interested in through some of our collaborative work at the Odin Institute So radiotherapy regimens for high-grade gliomas. So this is planning treatment for cancer patients cancer patients and in particular in this example these high-grade gliomas As a form of brain cancer So what we're looking at here and I always and I know I'm being recorded give the caveat You're going to hear an aerospace engineers explanation of oncology, but I'm speaking to mechanical and aerospace engineers, so that's okay What we're looking at here is what's called the standard of care Which is sort of the standards treatment that is given to patients today So on day zero the blue dot that you see there. That's The MRI the imaging that's taken The the tumor you can see they're depicted in the image then on day 20 the radiotherapy starts and See do I have a pointer on this thing? Maybe not So you see there with the purple bars that represents two grays per day of radiotherapy delivered and we Delivered two days on Monday Tuesday Wednesday Thursday Friday five days a week and then nothing on Saturday and Sunday And then two grays per day for the next week And so then you can see weeks one two three four five and six for a total of 60 grays That's the end of the standard of care regimen and Then at some later time There's an assessment for what is called the time for progression Which is a prediction of the time at which the tumor will come back at a certain level And then a follow-up evaluation and so again, this is the standard of care This is what is delivered to all patients So now I want you to sort of think about that Apollo example think about the notion of a digital twin and Think about how one could move from this one size fits all approach to treating cancer to To now bringing together models data this decision-making and moving towards patient-specific care So my group my group of aerospace engineers and computational scientists have been collaborating with the oncologist at UT Austin and at MD Anderson to To really think about how digital twin technology could make a difference in this kind of application and what you see now depicted on the second line here is You know a notion of how a digital twin Could could work in the situation. So again day zero. We have an image at day 20 We have another image for the first week We start with the standard of care where we're delivering two grays per day and then at the end of that That first week there's another image taken Which is now data that can be fed into the mathematical models that are describing The tumor in its evolution and being used to calibrate those mathematical models To a patient-specific model to think about this idea of models And we need the mathematical models that in this case and by the way and bill you may be interested to know this the models of tumors Reaction diffusion equations. So they're actually very sort of familiar although the physics the biology is very different. They're the kinds of Partial differential equations that we see in other aspects of engineering Combining that together with patient-specific data and really putting it all together and driving towards This notion of a digital twin that can drive decision-making So whether you're thinking about aerospace systems aircraft spacecraft whether you're thinking about civil systems buildings bridges and infrastructure Or whether you're thinking about medicine again That's idea of combining models and data Through the process of data simulation, which is mathematically how we referred to putting the two together to Issue predictions and drive decisions. I think is very very powerful and that really is is at the heart of what is a digital twin and You know, I believe strongly that a digital twin builds on modeling and simulation, but really goes beyond it And goes beyond it in the way. I've described where we have models We have data and we have this bi-directional interaction between the two So that was one of the next graphics that that was coming up And then the the next one switch I'll skip over in the interest of time was really really just to highlight and show you some of the work That's going on at the Odin Institute the incredible interests and digital twins across engineering across Climate sustainability the natural sciences and across medicine and while all of these areas are really exciting When I look at the medicine field, this is where I see just incredible opportunities To bring the kinds of simulation and computational techniques that engineers have been using To to really influence things in medicine. All right, so we're going to go over this one a little quickly The elements of a digital twin and again, this was fleshing up some of the the things that we have going on at the Odin Institute and manufacturing aircraft spacecraft Thinking about climate change really at a global scale with high fidelity simulations in this case This is Antarctica and that's Greenland and the interaction with the Atlantic currents and then in Medicine where we have two big centers within the Institute one focused on computational oncology and the Wilson Center Which focuses on cardiovascular? work so Okay, so the potential of digital twins very very exciting and Sometimes I think that if you were to open the newspaper or to pick up the brochure from a simulation company You might think that digital twins are a done deal that that we really can create them That's not the case and I want to spend just the next say five minutes Talking about some of the challenges of digital twins despite this great promise We're still not there yet, and I want to be very clear this cancer example I gave it's really Aspirational to think about having a digital twin for every cancer patient that could provide these optimized treatments in the way That I described this is a vision But we're not there yet. So why are we not there yet? Well, so the first thing is if I were to ask you and when I talk to people and they we start talking about digital twins to Think about conceptualizing a digital twin. What would you put in the middle of it? anyone What word would come to mind? What would you? Model oh good bill Bill got the right answer so Bill said model It's probably because of his background, but for many people they immediately go to the data they think about the digital twin with the data being core and You know really having piles and piles of data and then Probably having some kind of a numerical model wrapped around it probably a machine learning model that will go to work on all these on all the data that we have with some kind of analytics and drive decision-making and this may be a Good model for a digital twin and in some applications And in fact maybe in some of the most successful applications that we've seen of digital twins particularly in the aerospace industry Where we've seen digital twins driving predictive maintenance for let's say aircraft engines This is the situation we have where there is tons and tons of data being generated by the engine as it's flying and the Decisions to be made are the decisions about when to make maintenance. These are relatively simple decisions That fall well within the realm of the data that that are being collected across the fleet So that that way of thinking about a digital twin is absolutely valid for some problems But there are many many problems for which that is not That is not enough and that's because as many of you I am sure know it is really difficult to do predictive modeling for complex systems And what do I mean? We discussed this at breakfast this word predictive a predictive model is one that can predict Can predict the future can predict conditions that I haven't been seen but not just predict them predict them and get them right or predict them and Have some knowledge of just how valid are those predictions how good are those predictions? How much confidence do we have in those predictions as we we go to make decisions and again? It's really really hard to do predictive modeling for complex systems for all the reasons you see here these systems are Multiple cross multiple scales Despite the fact that we now have exascale computing. We cannot even begin to resolve All the scales in a human or even in a complex material in an engineering system Multiscale multi physics. We're talking about high-dimensional parameter spaces There are computational constraints Sometimes we need to do these sort of the data assimilation in in real time There are limited data. This is a really really important one to emphasize and actually if you take one or two things Home with you today after my talk. This is one of them. Yes We live in the age of big data But as engineers and also in the medical world we can almost never measure Exactly what it is. We want to know We are almost always taking indirect measurements of something else and then trying to infer what it is We want to know and not only are those measurements indirect They're usually sparse on an engineering system. They're expensive. They add weight to your aircraft They add costs. They consume power. They do not come for free and yes The data may be big as measured in gigabytes, but it is almost always Limited especially when you're talking about predicting the future or predicting conditions that you haven't seen so yes, the data are big but the data are limited and then finally Verification validation on certainty quantification are so important if we really want our modeling to be predictive so these are all challenges and You know, I really land on the statement that big data alone is not enough in some cases It's enough but for many of the problems that we as engineers care about big data are not enough so we really have to think about our digital twins as Bringing in something more and that something more is the models the models that encode physical principles Conservation of mass momentum and energy and all the other pieces that that we know to hold So let me give you another way to think about conceptualizing a digital twin, which is at the core Bilstall the thunder it's a mathematical model and that mathematical model encodes the domain knowledge even if it's imperfect domain knowledge it encodes what we know about these systems and Again, I really want to emphasize that this notion of a mathematical model This is what has let engineers predict the future as and make predictions about the structure in this building and What loads it can take and how it needs to be designed to have the levels of safety and reliability that we have in the built world around us Those are mathematical models encoded in in computation So this is not a new concept a mathematical model encoded in Numerics of course with data data fed in through the lens of models And all of that wrapped around with with decision-making Okay, so two ways to think about a digital twin and again I don't want to suggest that digital twins cannot be data-centric they absolutely can but I do want to emphasize that for many of maybe the the High-impact applications of digital twins and engineering and a medicine of in particular We need the models to be at the center of it all Okay, so that's wonderful. We say go off and do it, but there's a big catch and That big catch is that these mathematical models which by the way in many situations come to us in the form of partial differential equations What are partial differential equations? They're equations that describe quantities that evolve in space and time and of course that's not all of it There are other kinds of mathematical models Discrete phenomena that are that are important, but just as one example These physics-based models are powerful They let us predict the future but the catch is that they are really really expensive to solve because by the time you have discretized this mathematical model and turned it into a simulation capability you might have something like this example of simulation a combustion simulation of a Rocket engine a simulation that at full-scale the scale of a full engine might take weeks or even months on a supercomputer and You don't want to do just one simulation You want to do many many many simulations and you want to have a digital twin That is assimilating data and updating these simulations And so the expense of these physics-based models is a is a real barrier to Having them at the core of of a digital twin Okay, so that leads us into reduced-order modeling which really is a critical enabler for accelerating predictive computations We need to make those predictive computations. These are the mathematical models that we know are imperfect But we trust them to give us predictions that can drive decisions that can be validated and can be drive decisions We need to make them faster So that these mathematical models can be the core of a digital twin or even can be just the core of of engineering design So what is what is reduced-order modeling? reduced-order modeling is an old is an old topic it's a field that's been around for several decades and The general idea is shown here, and I'm showing to the students here. I hope you Sort of recognize your state space form a good old x dot equal a x plus BU right? This is sort of a schematic of that on the left is a high fidelity physics-based simulation by the time you've taken your governing Equations whether their partial differential equations or whatever they are and discretize them You're going to end up with some kind of a large system to solve and that system will have millions or maybe even billions of degrees of freedom because again, we're talking about discretization in space and ultimately in time and again that That simulation might take minutes hours days Maybe weeks maybe months to really solve at the scale of a full system where we're targeting for digital twins So the idea is can we come up with a reduced-order model there on the right? The reduced-order model is first of all much smaller in dimension So instead of having millions of degrees of freedom it has maybe tens or hundreds and importantly instead of taking hours or days to solve it it takes seconds maybe fractions of of seconds and Of course we can come up with reduced-order models But the big question is can we say something about those reduced-order models and how good they are? How predictive they are so that we could actually start to use them with confidence in our design or in our design task or our digital twins start task A broader class of problems is surrogate modeling right so surrogate modeling is the process of coming up with approximate models Of which I would say machine learning approaches are one way to come up with surrogates reduced-order models tend to follow a particular Process that process is first of all to generate training data Meaning that we have to solve or somehow query this large high-fidelity system to generate some kind of representative data in the reduced-order modeling world We refer to those as snapshots So these are going to be solutions pressure fields temperature fields velocity fields that represent the the system at high-fidelity The second step is to identify some kind of structure And the structure that we're looking for is some kind of structure that is going to be amenable To this compact low-dimensional representation the reduced model Many of the methods not all of them But many of the methods will look for a low-dimensional basis that Describes a low-dimensional subspace so the geometric picture in your head is here's that rocket Combustion simulation there are millions of degrees of freedom. That's a million dimensional state space What we're looking for is a low-dimensional subspace Maybe a hundred dimensional subspace that cuts through that vast High-dimensional state space and as a subspace on which the dynamic the the dominant dynamics really evolve So identifying structure and then the third step is the reduction step where we go back to the physics And I wrote here partial differential equations because that's what we deal with we go back to the physics we take the physics physics governing equations and Project them mathematically onto the low-dimensional subspace and I'll show that and a little bit of math in a second Again, I just want to pause here and note that if I had stopped at step two We could have been talking about a machine learning method right generate training data and then identify Structure sort of fit some kind of an input output map Where I think reduced order modeling and black box machine learning really deviate is in this third step Where we go back to the physics and even though now we are talking about an approximate model We are injecting the governing equations back in through their projection onto this low-dimensional subspace And by the way when you do that if you do it in Responsible ways and you are working with nice classes of systems particularly linear systems You can come up with reduced order models with rigorous guarantees Error bounds in some cases error estimators and other cases that again give you a sense when you use the reduced order model to make Predictions for conditions that you didn't see in training just how good is it going to be now? We can't do that for all systems particularly in the nonlinear case But in some cases we can get those rigorous guarantees and why are there guarantees? It's again because the physics is is injected Well, there is no network and there is not necessarily a network Let's let's do the linear algebra then maybe we can see if your question is answered. Okay, so Again start with a physics based model do not try to read the partial differential equations and this let your name is professor Anderson So just again to make it more concrete the governing equations for that reacting flow That you saw It's the Navier stokes equations conservation of mass for minimum energy and then the governing conservation laws for the chemical species and again People who are expert in these things work their magic and turn this mathematical model Into a numerical model that is solved with high performance computing And just to make things kind of clean and a little bit more concrete To the to the question that was just asked I'm going to work with the form that you maybe can see there at the bottom of the slide Which is x dot equal ax plus bu plus f of x and you so sort of the standard linear state space form with all the nonlinear terms just just grouped there and in in f and What is x here? These are now the discretized state So we're solving for the pressures the temperatures the velocities the chemical species But now instead of being infinite dimensional fields. These are discretized Over some kind of grid and so there are millions of degrees of freedom in this the system We can write it down nicely on this PowerPoint slide But this is a massive system and of course there are all kinds of complexities that go into that discretization Okay, but what's what's the kind of the point of showing you these equations? So I want you to think first of all just think about a linear system So a big linear system time time-dependent system x dot equal ax plus bu and again X is a very big. Let's think million dimensional state vector So let's think about those steps of reduced order modeling We said we would look for a low-dimensional subspace in other words We would approximate our very high-dimensional state x in a low-dimensional basis In a low-dimensional subspace represented by the basis V so V here is Got dimension millions of rows, but it's only got our comet columns It says it's got a hundred degrees of freedom and now the x-hats are the modal coordinates They're the expressions of my unknowns in this low-dimensional basis So here's that approximation we can substitute it into the governing equations Now we have millions of equations to satisfy We have only a handful of degrees of freedom So we are not going to to be able to satisfy all those equations. We have a residual This is the projection step and again just keeping things very simple for those who are not familiar with these methods what does projection mean it means imposing an orthogonality condition and It takes the form you see there The end of the story and the thing that is important is that the reduced-order model which you see at the bottom x hat dot equals a hat x hat plus B hat u Has the same form as the system we started with x dot equal x plus V u Now instead of solving for a million States we're going to solve for just r of them tens or hundreds and The matrices a hat and b hat are nothing but the projections You can see them over there on the right. They're the projections of the big matrices onto the low-dimensional subspace So that's a linear linear system And just hold that thought the reduced-order model if you derive the model through projection onto a linear subspace in this way The reduced model in the linear case has the same form as the one we started with Okay, the same is true if we were to take a quadratic model So now x dot equal x plus V u and now let's introduce a quadratic term What does it mean quadratic term think of Navier-Stokes u to u dx? It's quadratic in velocity Right or a term that involves pressure and velocity is quadratic So we're going to introduce quadratic terms which again show up because they are given to us by the governing laws of nature if you were to go through the same thought process Proximate the state in a low-dimensional subspace project you would get a reduced-order model that now is operating in this low-dimensional subspace, but it has the same form as a system we started with Okay, so why why do I emphasize this preservation of form? I'm writing it as linear algebra Because that's the way I like to think of it But the fact that we have an a matrix we have a linear term the fact that we may have that H Operator the quadratic term. That's not a coincidence. That's coming because of the governing laws of nature, right? When you write down conservation of mass You have a linear equation when you write down conservation of momentum Depending on the system that you're working with you'll go to have linear terms. You might have linear diffusion terms new D by D second derivative of velocity. It's linear in velocity You might have quadratic terms. You might have cubic Yes, you may have non-linear terms and that's a topic of another talk But the structure in the model is there because of the physics and so if your approximation preserves the structure Your approximation is doing something that is aligned with the physics And I think this is a really central idea as we start to move towards surrogate models that embed physical structure embed physical constraints And and move towards being being predicted Okay, so that's all all great and you'd say oh, this is wonderful Why is the whole world not using reduced sort of models because Karen mentioned that this is an old field that's been around for decades and You know the answer is that reduced sort of models have had some impact in some Some areas and some areas of industry But they have not seen anything close to the adoption that we're seeing with things like machine learning and you could ask Why is that and I think the main reason is they're just too difficult to implement right? I'm talking about a matrices and h's and higher-order terms and now I'm talking about going into a code And by the way, if you're bowing that code is a Fortran code that has been started to be written decades ago That is thousands and thousands of lines of code and someone's now telling you you need to find these matrices and these operators And then you need to project them. It's just just too complicated And you know this well because we battled with it with a with an Air Force code over over that so that then brings us to The work that we've been doing in operator inference, which is to say we said we're trying to come up with approximate models We have all this beautiful theory from classical model reduction That says if you start with a physics based model if you start with what you know and go from there There's a lot of structure you can embed and by embedding structure you can then Have theory and guarantees and predictive power that comes along with it But on the other side you have machine learning methods that are so attractive Because all you have to do is run those giant codes and collect data and then set the machine learning to work on it You maybe need to know how to how to run the code But you certainly don't need to go into the source code and extract all these operators that Non-intrusive nature of machine learning. I think is one of the reasons that has seen so much so much adoption So we asked the question. Can we can we try to get the best of both worlds? Can we bring the theory in the perspective of model reduction and put that together with a non-intrusive view? that comes from machine learning and Learn predictive reduce models and so the answer is yes, we can do that and that's the method that we call Call operator inference Where again, we go back to this observation that says if you are going to get the reduced models through projection and Today I'm talking about projection on a linear subspace. It preserves the structure So said another way if I know the governing equations I'm solving and I also should make a comment There are classes of problems where we don't know the governing equations and the job is to discover the physics And that is a really important class of problems, especially in biological applications I as an aerospace engineer are talking about problems Well, I don't need to discover the physics because I know what it is I might models may be imperfect, but I believe in conservation of mass momentum and energy I have to say I sometimes tell my children reminds me of Homer Simpson saying in this law We obey the second law law of thermodynamics if you guys have seen that episode, okay So in situations where we have a physical model that we trust enough and it's just too expensive We know what the physics are. We don't have to discover them and through this lens of Classical model reduction. We know the form of the reduced models. We should be looking for so Again, I really want to emphasize this point the form of the model that we're looking for is going to be inspired by the theory But we are not going to derive the models by the theory by that sort of Intrusive projection based approach instead what we're going to do is grab the data and machine learning style We are going to learn we're going to solve an inverse problem to learn the operators Are they structured form of a reduced model where the structured form of the model was presented to us by the by the physics? Okay, so that's the operator inference. You're going to start with a physics based model You're going to use this lens of projection. You're going to write down on a piece of paper I am looking for a reduced model that has this form the form dictated by the physics and then you're going to pick up a Machine learning viewpoint and say can I take data now from training data from these simulations? Or by the way, we could also be taking data from experimental simulations throwing it in the pot and learning The reduced sort of model which is describing the evolution of the coordinates on this reduced sort of subspace And as with all great things in life, it's all just linear algebra. There's something else I tell my kids all the time my kids think I think I'm a little crazy That's right, they're required to love me. That's right It's all just linear algebra this notion of learning the operators amounts to the Optimization problem that you see here. So the first thing and for the example here we are learning a reduced sort of model that has quadratic form and Again, I said it was a topic for another talk. It turns out there are many many non-linear Partial differential equations that you can actually write in quadratic form through clever manipulations of the variables But let's think of this example of quadratic form So we've got to solve an optimization problem to find the operators the matrices a hat b hat and h hat those are the matrices the operators of the reduced sort of model we are Driving it with data that comes from the simulation and you see these these x hat here these x hats here This is the data from the high-fidelity model, but projected onto the the low-dimensional man of the low-dimensional Subspace which is a very easy matrix vector multiply operation And what are we doing here? We are minimizing the residual Which is that in other words is we're taking the data from high-fidelity code with Substituting it into the form of the reduced model and then we're saying minimize the the sort of the fit to that model form and by Posing it in this minimum residual sense You can see that the things in blue are the things we know those are the data and the things we are solving for The a hat the h hat and the b hat. They're all linear So this is nothing but a linear least least squares problem It's a linearly squares problem at the at the end of the day now, of course There are many important tales that I'm glossing over in a talk like this regularization is really important this question of ill-posedness of this problem Benjamin Peer stopper my former postdoc and actually the original paper there was in 2016 with Benjamin had some really nice More recent theoretical work where he shows provable recovery Of the intrusive reduced model. Why is that important? Remember I said that those intrusive methods come with With a bunch of theory and so if you can recover the intrusive reduced model you can Get a window into some of those those guarantees Okay, a few things to emphasize. It's entirely non-intrusive All you have to do is generate the snapshots from high-fidelity simulation or from an experiment So start with the data set compute the proper orthogonal decomposition basis That's the way we identified this low-dimensional subspace.  That's nothing but a matrix vector multiply and then solve the linear least squares problem To come up with the reduced operators You never have to touch anything about the high-fidelity model or the experiment that you are That you're trying to approximate So really is trying again to get the best of both worlds now, of course, we are giving up some of the theory To go non-intrusive like this, but what's I think really important is that we are keeping the physics We are we are we are embedding embedding the physics So to the young people out there before you reach for your neural network, you need to ask yourself Do you know more about the system you're trying to approximate? Is there a structure that may be more amenable to the physics and may give you more guarantees? Then the last note to mention is that everything because it's all linear algebra everything here is very very scalable and my former postdoc unit Farcast has been working on a distributed Algorithm of this where you can take now very very large-scale snapshots distribute them across processes and Do all of these computations at really really large scale so we really could start to learn reduce sort of models At the at the at the largest scale and you'll see a little bit of this in the results Okay, so in the last the last few minutes I Want to show you just one example and this is actually work that we've been doing under an Air Force Center of Excellence where Bill is a collaborator and also under another contract working closely with Air Force Research Lab These results in this work is unit Farcast's work unit has been a postdoc in my group And he has just started as an assistant professor at Virginia Tech So you can see the the setup here. So we're in this case. We're modeling a rotating detonation rocket engine So the high-fidelity model here is alias simulations of the reactive viscous 3d Navier-Stokes equations And these are horribly horribly horribly expensive simulations And I really want to emphasize that we UT Austin we don't even touch the code AFRL is doing all the simulations all the CFD simulations and Generating the data and we are interacting with them at the level of the data So it really is truly non-intrusive and I also want to emphasize that as much as you know We would love to do intrusive reduce models. It's just not practical practical to create reduce sort of models with this kind of complexity of of Code especially in commercial and legacy codes Okay, so there are those those governing equations again So let's walk through the steps. So the first step again is generate the training data and again We're not even doing this part of it the AFRL is doing this part of it Again, just to give you a sense of the scale of this problem the alias simulation We can simulate what they are simulating for two milliseconds And by the way, the desire is to be able to simulate for much longer, but the computations are just too expensive the full Engine has a hundred thirty six million degrees of freedom and the spatial Discretization and then you can see there just some of the the CPU hours That it takes to generate two milliseconds two milliseconds worth of data Turns out the and this by the way is a another sort of really interesting thing I've come to appreciate all this effort and energy goes into running these huge Simulations these huge simulations But then the data are so big that people can't store them and so they don't even store all the data that comes out the data that is actually available are dramatically down sampled both in space and in time and I think this in itself is a really sort of interesting question for those of us who are Working in these areas, so we end up having 501 Snapshots, so that's 501 time instance And it's also down sampled in space to a four million degree of freedom grid right there are 18 We use transformations 18 state variables in our model. So pressure temperature velocity is chemical species, so it's 18 times four 72 million degrees of freedom even in the down sampled and we have 500 data points and Yes, you could fit a neural network to this data But you really would have to stop and think about how many degrees of freedom you have to fit and How much information you really have to fit it and whether you would have anything that would be Predictive in any sense. Maybe by the time you've done training your hyper parameters Okay, so we have our training data snapshot matrix is 76 million by We use 375 Snapshots for training. We're going to keep the rest back to see how the the method goes We now come in compute the POD basis again at this point. It's linear algebra Again, there's a really important details about scaling Especially in a problem with so many different physical quantities you have to be very careful We compute the singular value that define the singular values and the singular vectors define that low dimensional subspace The singular values are an indication to you of just how reduced your reduced order model can be I always tell my students don't ever walk into my office without a plot of singular values in your hand The singular values tell you so much about the complexity of the problem the complexity of the data set You're working with and just how much you could expect to to reduce it You know, it has been working all of this in parallel and front era, which is the NSF leadership machine It's the the plus a supercomputer on an academic campus in the country at UT Austin. He's scaled all this up to To more than a thousand processing units and it's very scalable Okay, so we've computed the low dimensional subspace now we can infer the reduced model again at this point It's nothing but but linear algebra. So let me show you just a few Sort of snapshots of what it is we can do and Again, if you take anything away from this talk the next sentence I'm gonna say you should take away which is there is no magic Despite what you might read in the newspapers or even some academic papers. There is no magic you do not get to go from 76 million degrees of freedom down to 24 degrees of freedom and get perfect solutions And if anybody tells you they can do that you should become very very skeptical The reduced sort of model has 24 degrees of freedom. It is completely decoupled from the CFD It is going to run in a fraction of a second Really really fast the CFD even down sampled 76 million degrees of freedom. You saw how how expensive that is You are not going to get everything. There is no magic. What you're looking at here are three different implications in this engine one close to the injectors one within the detonation region and one downstream of the The detonation region. We're looking at pressure traces with the CFD in black and the reduced sort of model in orange And what you see is the reduced sort of model kind of gets it, right? It gets the right frequency it sort of gets the right amplitude it gets the bulk the course behavior Really pretty amazingly. Wow But it does not get the details Does not get the high frequency either in time or space as you would expect going from 76 million down to 24 degrees of freedom And I should say the top row there is at the end of the training horizon and the bottom three rows are Now moving out into the simulations well beyond the training regime It's kind of hard to visualize these kinds of results But I'll say I think this problem is really at a scale size and complexity of physics That goes beyond what people have been able to do with reduced sort of modeling and getting this kind of Predictive performance just a few more ways again. It's very very hard to visualize This is another really important area of research is Visualization of all the incredible stuff that we are computing What you're looking at here now Time is going from left to right so training in the first column beyond training in the last three columns on the top the CFD sort of full colors solutions of pressure and then Picking out a narrow band of pressure with those red contours the full CFD and the reduced sort of model And again, this is to emphasize there is no magic the reduced sort of model is not going to get it perfectly But what you're seeing is that the reduced sort of model is doing a really good job of predicting Kind of the the large-scale the general dynamics of what's going on and again We're working very closely with the Air Force to understand what is a designer looking for what would a designer want to see? Being predicted well in a reduced sort of model so that this reduced sort of model could be useful in running many many simulations and supportive design As opposed to one high-fidelity simulation that would take would take weeks So we're feeling really happy happy with these these results Okay, let me oh, I've got one minute just to just to close up So again just to summarize our take on operator inference again given a physical Natural system with non governing equations and a set of data and fur reduced model the end of the day It is all linear algebra. It is all linear algebra. There's some of course non linearity in the physics We use the physics to define the structured form of the model We use the theory and the lens of projection base model reduction to cast the inference in reduced coordinates We use inverse theory to analyze the structure of the inverse problem to understand What writes how big of a reduced sort of model can we solve? What guarantees can we make about this inference problem? and then we use numerical linear algebra to us to achieve efficient scalable algorithms and I would encourage you all to not take away Operator inference, but take away this general principle that as we start talking about Machine learning and how we combine physics-based modeling and machine learning to make sure that we're not just slapping physics into a loss function as a penalty term and hoping that somehow the neural net is Matching the physics but to really think about the physics and what structure that defines and bringing the physics in as a central part of the models that we bring Of course, there are a range of use cases There are many cases and to go back to what I said earlier where you have tons of data And we're black box machine learning is going to get you to what you need There are other cases where you can really afford to go a more mathematical route to go reduce water modeling We're really been asking the questions as to how we can can land in the middle And then lastly, I just want to conclude and just pop up a couple references So Boris and Benjamin who are now both faculty members both former postdocs and worked on a number of the work that I showed here. We recently just authored a paper an annual review of fluid mechanics that talks about these operator inference problems It just came out last month Unit there. I mentioned him in his work with the combustion Is in the final stages of submitting a journal paper as soon as his his advisor gets around to the final round of edits? I didn't have time to talk about it today, but I talked I emphasize the structure that's preserved through the linear subspace We have some I think really exciting work together with Rudy Healan who's a postdoc in my group and Looking at quadratic manifold so that you have a richer approximation Space and it turns out that many of the things that I said about preserving structure and knowing the form of the reduced model Carry over if you work into a if you move from a linear subspace into a quadratic manifold So again before you reach for the neural network. There's so much fertile ground In the middle there with more structured yet powerful approximations and then lastly again. It was hard to know what to talk about I'm really excited about the work that we're doing in digital twins We've been working with probabilistic graphical models really emphasizing the role of uncertainty quantification in digital twins and Anubhan in particular has been leading the charge on our collaboration with the oncologists where we have taken the foundational methods that we developed for Structural health monitoring digital twin for an unmanned aerial vehicle and brought them over to that cancer setting that I motivated in the in the beginning All right, I'd also like to thank the funding sources and then a very last plea which is If you haven't seen this my son is on a campaign You must feel sorry for my son because you've already heard that he is subjected to all these that's I don't know Conditioning about linear algebra. We are almost at a million views on the TED talk So if you haven't watched the TED talk, please watch it But in all in all seriousness again, I think a really important thing that we as engineers as computational scientists as people Who come from a background of physics based modeling have to do is to make what we do accessible to the public and talk about the importance of Models together with data so that it's not just a I'm machine learning all the time But it's a I am machine learning together with the incredibly powerful rich base We have of physics based simulation optimization design and control and I really tried to do that in that in that TED talk Okay, so with that I want to thank you all for coming and for listening to me talk when there were no slides And I guess we'll have time just for a couple questions. Thank you Hello, thank you for answering my question I just had a question about the advantages of using digital data versus the physical experimental data to actually base your The reduced order modeling on and then eventually the digital twin because I noticed when we were talking about the rotational Detonation engine. We only got a couple of microseconds of data versus is it certain like the The amount of like you would get more time using the physical simulation Versus less fidelity on the You know, I think I think that the surface of all is a real opportunity space which is simulation data and experimental data and Maybe you have a sense at least in the way we're approaching the problem there is no reason why we couldn't put the two together and I think this is this is a very good question to ask for this example the simulations are so expensive and you have such a Tiny window in through the simulation data, but if you were to talk to an experimentalist you would Hear from them that in this particular example the experiments are expensive and you know the kinds of data that you can collect are extremely limited as well and You know, one of the nice things about the simulation data is that you get the full full state, right? We get pressure and temperature everywhere and again talk to an experimentalist. Yes, you can have things like PID Particle image velocity Techniques that give you fields, but they're not going to give you sort of this clean data everywhere like you'll get from a simulation Now, of course the experiment is the real world except that it also isn't the real world It's also a surrogate of the real world the simulation is probably even less the real world So there are real strengths pros and cons to each and I think moving towards figuring out How you can leverage both simulation and experimental data and maybe even close the loop so that you're Understanding of these very expensive simulations and these very expensive Experiments, which are the ones that you should run to best inform The kinds of models or ultimately the kinds of decisions you want to make Yeah, we're very we're very interested in also using experimental data But again, I guess this is my naivety as a computational person I just sort of assume that they measured pressure and temperature and then you start reading the papers and they're measuring things that I have never heard of And then I'll ask, you know how and I don't even want to start saying them because I'm probably going to pronounce the words wrong But these luminescence How like how does that relate to temperature and then it turns out it's like a very complicated relationship? And so you would have to build that in so We often think of experiments of being measurements of just what we want they almost never are But this in itself is a really interesting opportunity And now the experimentalist can correct me I Thank you for the talk again So in the example that you gave the full-order model had 76 million degrees of freedom and the reduced one had 24 and I think that's sort of an extreme case, right? But I was wondering how the accuracy of the reduced order models scale with the order yeah, so One thing I sort of skipped over I was going fast at that point is Because we only had we had 500 snapshots and we use 375 snapshots for training we and Again, I'm sorry. I'm sorry if they're machine learning people here and you're feeling attacked but We don't believe that you can fit more parameters than you have data points because that's an undetermined system And yes, you can do it and you can get an answer. That's not real and So our size of the reduced model is actually limited by the amount of data that we have We could fit bigger reduced models, but again, it's not real You don't have the data to inform and we're forgetting we're fitting quadratic reduced models So we need you know 24 degrees of freedom for the linear term and then something that scales with 24 squared For the quadratic term, which is why we come up against that limit of 24 So we are in the low data regime for this kind of application Of course if you had more data you could fit a bigger reduced model and you sort of asking the question How does the accuracy? scale You can you can get more and more, but even though it's an extreme example I think it's a really good example of what you get with reduced modeling Which is you do not get the fine-scale detail You're not going to get all the and if you were to dive into those CFD pictures and see the very fine Scale detail of what's going on and the pressure and the temperature fields You don't you're not going to get that and if you do get it You probably have a reduced model that's overfit and will not predict what goes on later But what you can get are these sort of lower Larger spatial structures. You can get the frequencies. You can get the amplitudes You can get enough for it to be useful in Some kind of a setting particularly when combined maybe with real-time data in a digital twin So I know I'm not really exactly answering your question But I think it is more than just making the reduced sort of models big and again We could we could probably fit the data exactly But you would be looking at fits by hyper parameters You wouldn't actually be looking at a reduced sort of model And I think this is something to be very very careful with with neural networks is What you're looking at maybe one more question I guess I'm going to ask about the linear algebra problem, right? So you when you solve for the a heart b hat you assume you have a unique solution because it's linear or quadratic What happens in general in solid mechanics is that this a and b matrix are going to depend on x2 Right there. You don't know if you have a unique solution, right or not Yeah, so so I didn't I didn't really talk today at all about non-linear problems other than the ones that have the particular quadratic form So, you know first to say something I said earlier Which is actually turns out that there are clever tricks you can play with the variables that you use in the reduced model That allow you to write many many systems in quadratic form Even though the partial like the the governing equations are not and that's because we can we can Sort of play games with what we're representing in the reduced model state So there are many cases transformations that can get around that And then there are other cases where you cannot get around it or with or getting around it Causes you to end up with algebraic equations and all the things that create a mess And these are some of the things that we are looking at You can also sort of go through the notional approach that I Layed out, but then you don't get this nice clean linearly squares problem at the end you end up with a non-linear problem and then you could still Solve an optimization problem to find a reduced model. You might not have a unique solution You might not be able to guarantee So yes absolutely and for sure there are classes of and I should also say there are classes of problems That are not amenable to approximation in a low-dimensional subspace or a low-dimensional manifold like I've described today and Those are and there are classes of problems for which a neural network Approximation that maps many to many in a very sort of compositional and hierarchical way is the right approximation Technique, but there are also many classes of problems that are amenable to these kinds of approaches And so we shouldn't we shouldn't reach for the neural network before we have thought through linear approximation quadratic metaphors Wonderful. Thank you very much again for giving us a great presentation and being here today. So help me. Please. Thank you