So, I'm the infrastructure lead and the thing that I want you to understand by the end of this talk is what you can do with our infrastructure. So, when we look at how people set up and these simulations. We're pretty familiar with starting by finding a PDB you like or docking a drug you like, and we can't help you there. But where we can help you is at parameterization. So, you can put a drug through gaff or CGEN FF depending on your favorite MD engine. You can ignore the warnings about how you should benchmark the generated parameters and not just blindly use them. So, you can do your all your black magic to the PDB file like I don't know adding caps adding waters keeping waters setting pro nation states. However you however you feel like it should be done.  Then you do everything again with your new force field, hopefully the same way, but your notes weren't that great. So make sure to change your non bonded cut off because that's what that's the way you're supposed to use a different force field is you need to be changing your non bonded cut off in your simulation settings, otherwise you're doing it wrong. But then you realize your MD engine doesn't support the enhanced sampling method you wanted to use. So you try to get plume at working and like some numbers come out and you're like 60% sure that they're good numbers. And then you just give up and redo everything with a different MD engine. The tools and experience help, but preparation tools are permissive and failure is often detected late tools tend to move from chemistry worlds to model world as early as possible. Protein and ligand force fields use completely different tool sets and you're kind of left to figure out how to put them together at the end if you want to mix and match. The tools often need user oversight and are difficult to run in batch and automated pipeline. The open force fields are often engine specific and you can't really interoperate easily. And the parameters that affect the energy are not all encapsulated in the force field. So at open force field we thought what if your tools remember the intended chemistry throughout the preparation even after the parameters were assigned the chemistry could be known and referenced in the appropriate times. What if the same tools and force fields worked for both ligands and proteins, just throw molecules. What if the same tools and force fields could be used for any MD engine, and you could prepare the whole system and then apply the force field and export to any engine. Where everything was just done in a Python interface not jumping between tools with different styles of arguments or different file formats. If the tools warned you when you were doing something that didn't seem to make chemical sense. And finally, what if all of the energetic parameters, everything that you need to figure out the energy of the system except for the coordinates was included in the force field file. So, advances in force fields are cool. And we built a lot of infrastructure for applying them to. So we can take in just about any chemical format from smiles to PDB and give you a parameter I system. You can prepare the system once and then parameterize it any number of times with any combination of force fields and MD engines. And I think our support for new force field technology for new functional forms is going to make this even better. So we want to take you through what's available now and get you excited about the changes that are coming up. We have an FF workflow system preparation happens between the chemical structure files section and the topology section. And that's where you're putting things together that's where you're salivating. And so you can bring us starting there you can bring us your prepared system in any, in any format. So if you have the open eye wrapper in particular, or if you have an open eye license, the list of formats that you can come with is ridiculous. And with our decade you can take quite a few can get basically all the common formats, except all two that's very tricky with our decade back end as well. But more or less anything that defines a chemical graph is in scope as a starting point for open FF. And especially one thing that we've been struggling with is the loading of PDBs because PDBs sometimes they have components or you know residues that meet a set template and so we can look them up from a library like oh is this supposed to be a double bond is that supposed to have a formal charge. But oftentimes it doesn't oftentimes you've just got a small molecule in a PDB file. And we are now building out we've we've had some rudimentary functionality sense very early on but we're really building out right now the ability to load unrecognized components from PDB files. So you can create a topology, which has a complete physio chemical description of the system. That's the protein your drug your water and salts and co factors membranes buffer chemicals, whatever, along with the starting positions and the box size and you can save that out to a disk if you like just as a topology with no parameters yet. And then do other. And then you can combine that with a force field to get an interchange. And so again the spirit of force field specifies everything you're cut off so you don't need to worry about saying that correctly so you're not misusing the force field that's already specified in a spirit of force field. How to apply the parameters the P&E settings whether constraints are used. It's all defined in the force field. And our force field format. I think we can safely say is more extensible than other formats. It's easier to debug problems with Smirnoff force fields like we saw this morning without getting tangled up in atom type relationships and things like that. And then we can use interchange to get output files suitable for simulation and a number of md engines. And so, right now, we can export to open a man grow Mac open a man grow Max we're pretty confident about that I'll show you an example soon. We have rudimentary support for amber and lamps, and we're experimenting with importing components from these systems as well. So, basically taking things that were parameterized elsewhere, and if the force fields are compatible and that's a big if we're building out experimental functionality so that these things can be combined. And so if you parameterize your system and something happens and you don't like it, and you wanted to use a different force field. That's fine. The topology object itself before it's ever met a force field is serializable. You can save that to a disk and load it up later switch out your force field. If you want to use a different engine later, you could save your interchange, totally serializable. Come back to it, export to a different format, must with whatever you want in the interchange. If you want more accurate parameters for some drug series that you've never seen before it's got some exotic chemistry that's not from any of our training sets previous talk we've got bespoke fit it slots in perfectly. And if you want to do anything weird, we've got documentation there all the codes open source so if you can't find your answer in the docs if you're comfortable with Python, you can probably find it in the code. And we have first class support for Jupiter and then Jupiter notebooks so this is a really easy way to record and communicate the workflow and what you did such that other people can use it modify it. So I wanted to announce that topologies can now be created directly from PDB it's this came out somewhat silently in a release about three weeks ago. But where it used to be very painful process of loading all the components that you want in your system from PDB, you know, one at a time. Here's a protein and let's go get some waters. No, now we can load a box with proteins waters salts small molecules supplied that they have connect records. Yeah, it's, it's very powerful. Yeah. Yeah, so the point was in this code block, we have unique molecules equals ligand, and there's more than one unique molecule in here there's water and ions and a protein. And we have built in recognition for the water ions and protein. It's just when you come in with an exotic new species. You have to know the chemical identity of that and help us load the PDB by telling us about it. So, when people try to sell you Legos, have you ever gone to the Lego store.  No, that's not how Lego sells Legos Lego they show you the dinosaur you can make the roller coaster and once you're interested in the things that other people have made, then you start asking about the blocks. And that's what we're going to try to do this. So, the remainder of this presentation is basically just a bunch of cool stuff that you can do with the toolkit, and to prove that you can do it we already did it. And if you go to this URL, Matt's put it in the discord. But you can open up a co lab for every one of the things that you're going to see in the remainder of this and do it. You can try it out with your own stuff you can install it locally. It won't help too much to pull it up on your phone they so tempted to put up a QR code, but your phone can't run co lab, but your computer can. So, these are our vignettes and the repository is 2023 workshop vignettes. Again, there's a link in our in the discord. And each one of these, we've given a color code so previously we've been very paranoid about sort of product management we don't want to release anything that the most picky of users would complain about. And so we keep things very under wraps for a very long time. And we found that's not helpful like you heard about this talk yesterday we showed some kind of wacky stuff you could do with open force field and she loved it and now now Jensen has been investing heavily that they've had a person doing this stuff for a while. And so we want to give you a measure of confidence in each one of the things that we're going to show. So we've got a stoplight system where green means it's production ready. You should expect this to work if your inputs are valid and if it doesn't work it's probably a bug that we need to fix. Yellow means it's something in the public API, but you should expect to be, you know, if you feed in a bunch of inputs, something weird's going to happen and we'd love to get your bug report we're going to try to fix that. And red means that this is something that's a prototype. It's something that if you're, it's part of a private API somewhere or you're going to have to acknowledge as some point in writing this code that you're using a prototype or something experimental the different packages do it in different ways but we show you how they how they work in notebooks. So for example, here's a vignette. We said, you know, hey, we want to show that open force field can do something useful that people haven't really tried before. And so we've made a myself self assembly simulation. It's actually not very big this isn't all of the code. This is some code showing that we've defined DLPC. We've used pac mole to make a box of them. And then we simulated that box. And sure enough, we get same dynamics we don't have a graph here I'm not sure if this is precise to, to a great extent but if you have a system like this that you want to try out. Go for it. This one's a yellow light only because the pac mole wrapper isn't something that's in the public API yet. It's something that's kind of a. You know, apparently, but if you want to go ahead and use it feel free, but if something weird happens at, you know, don't be surprised. Another thing I wanted to show off that we can do is, oh, yeah, and we didn't do it on that one. So what I want to show off we can do is modified functional forms.  And this is what our research team uses a lot. So here's an example of an octane and two metal purity and I think. So this is part of a hydration or a mixture salvation for you or something Josh that may help with recovery. But as you can clearly see here. This is using a double exponential non bonded for this isn't Leonard Jones. How incredible is that, and you can actually run this, you can, you can follow over to chocolate and repository here, and take a look at how you could get running in a non functional form. And so let me say a little bit about our docs. We've, we have comprehensive docs, we've now our documentation we've consolidated it under the URL docs dot open force field dot org so no longer do you need to go to every packages GitHub and scramble around for the documentation link docs dot open force field dot org is the landing page for all of our packages. And there's a search bar up here that will work to find you code from all of our packages API references theory documentation. If you don't know what package you're looking for, it'll search all of them. And we've heard a lot back from users and in fact mark Forrester has been really helpful in getting us to understand the new user mentality like, you know the wall that you hit and like the subsequent wall that you hit as you're getting started. And so we've added a page of what we're going to call wayfinding docs. And this is sort of like that diagram that I showed earlier but it's vertical. And this shows the information flow from chemical systems down into molecules and apologies and force fields out to an interchange. And so if you're wondering, Oh, what do I need to install what component do I need to be using for this thing I want to do. So you can come to this page and you can click through all of these, these go to the documentation pages for the different, the different projects and to the specific API reference for the component that you're looking at. We've also got consolidated lists of projects we've got theory documentation as well. We've got a number of example notebooks. And so you can you can go and check these out in the docs and run them. These are separate from the vignettes these are better than the vignettes and really high quality API references. So, yeah, I think Josh had a specific thing you wanted to walk through here, but I'm not going to do that. And beyond that we have video tutorials like I said we've done these advanced workshops and we record them we edit them we clean them up so you don't get the ums and the weights and the the room windows being cast. They're really kind of snappy I don't think any of them are longer than an hour, and they go through just very punchily the some of the advanced workflows that we show off. But docs are always going to be a work in progress, just like the software just like the force fields. And we want to gain a reputation for stellar documentation this was written before the feedback yesterday so Josh maybe should have updated the slide. But yeah, we want to hear the bad things about the docs. It's really hard for us to tell if they're doing their job because, you know, if we have bad docs, people ask us how to do things all the time but if we have good docs we don't hear anything. So please tell us the things that you're having trouble with please tell us if things are going well. We want to make sure that this is something that people like companies who don't want to get all mixed up in scheduling and IP security can just come to our docs and passively find the solutions to what they need.  Instead we're making a central examples page as well. That should make these a lot easier to find. Yeah, and for all of these will have links to co lab will have links to installation we really want people to get stuff installed liquidity split and start using our tools. Yeah, so bespoke to it I don't need to go over this again bespoke fit can save itself some work.  And once you've run bespoke it. This is literally what it looks like by the way I know people sale I did this amazing thing in a talk and you don't think that you'll ever find the code or be able to run it. So here's the code. This is literally a vignette and it's just called the bespoke vignette. If you're if you go to that link from the discord. And in a Jupiter notebook you just run bespoke fit and then you load the force field and then this is 20 more lines of code to your protein ligand simulation. So bespoke fit the product itself. So we've done a lot to improve the stability and the verbosity of, you know, bad things that happen as William mentioned yesterday, when running bespoke fit but there's still work to do on that. We can detect connectivity changes because when we do QM optimizations, they don't care what come informatics that the bond is protons can wander around you get these awful fits. And we don't prevent them, but we do prevent you from getting a horrible force field, we have it safely crash out when there's an activity change. We've done some routine maintenance to follow the biopolymer support the toolkit. And now there's better intermediate file preservation so when something does go wrong you can see, though it's still not perfect it's a complex beast and we're working to make it better. As William said yesterday, sometimes force balance runs for seven hours, and then you don't know what happened but you decide to kill it or maybe it kills itself. We've made a fork of force balance that we're intending to use as a dedicated bespoke fit back end. And so force balance is great as a research. It's got all these bits and bobs all these interfaces where you can plug things in and all of those slow it down when it acts as a back end for bespoke fit. And so we're looking at just stripping off a lot of the extras from the force balance fork and making it just a very liquidy split torsion fitting tool to be a back end. Oh cool and now over to me to talk about the rest of our infrastructure. So, like I mentioned before, we've made PDB loading way easier. We've had it for a year, but it was very painful you could only load one component for PDB. And now you can load an entire solvated box. So this is one of our vignettes this is a green vignette PDB files, and it's, it's got one more block below this to show the simulation and that is the entire Jupiter notebook. All of the open FF code to set up the simulation here, and the rest of it is boilerplate open MAM to set up the integrator. So, if you want to run a simulation, easy peasy. Open force field will can't function without RD kit and open eye. That's how we do our chemical substructure matching for parameter assignment. And as a consequence of this so we have pretty high quality to and from RD kit methods and to and from open eye methods. And just to show you something that you could do with that. So, in that protein ligand simulation I showed you a second ago this was our starting ligand. And if instead before starting the simulation, I take the ligand from the topology and I send it to RD kit and I run this, this reaction on it that's this taken aliphatic carbon hydrogen bomb, and turn it into a carbon fluorine bond, and then take every product that could come out from applying that reaction once put it back in the protein and simulate it. So here you can see these different aliphatic hydrogens have been substituted with chlorine here you get to kind of at the same position because of sclerochemistry. And you get one over there on the methyl but we do filtering to make sure because it's replacements around the methyl you don't need three of those you only need one, and then send it out to simulate. This whole thing with 70 lines of code from loading to the chemical modification to simulator. So, not quite alchemical free energy, but still very fun. And so the toolkits getting really exciting. Right now we're aiming to do a little bit more for polymer loading, we can load unique molecules, but we can't load unique new like residues from a protein. And we want to be able to do that and we're working on that with the shirts lab right now. And now I'm going to talk about interchange a little bit. I showed it on the diagram but in greater detail, interchange is the thing that comes out when you combine a force field in a molecule. It contains all of the physics, physics parameters for your simulation as well as positions, and I'm not sure about the losses maybe an old figure, but box vectors. And we can export that to open M and grow max for sure amber for kind of charm and lance eventually, I think our lance exporter works pretty well now. And so that's our big picture. This is, this is our spider that interfaces with a lot of the other members of the molecular dynamics ecosystem. So what's inside of an interchange. We've got a vignette to answer that question. So we can make an interchange from a simple small molecule like this. Here's the code to make it. And then we can say okay interchange what's in you tell me about your bombs. Tell me about all the force field parameters that get exercised in parameterizing this molecule. And it says well there's three unique force field parameters, and I'm going to vectorize these out so it's sort of a linear but here's the case and here's the length so these case are up in the hundreds of lengths or angstroms. And then we can say well, where are these unique force field parameters applied, and it can give us a number of more useful representations but for presentations. This is a great representation it will give us a sparse matrix, telling us where these different parameters get applied to the actual top logical bonds and system. And if we multiply these matrices, we get the system parameters, which are again two columns pay and links. But these are, and I've numbered the rows here so a carbon fluorine bond has these parameters number four is also carbon fluorine it should have the same parameters. And so if you were interested in doing some like ML optimization force field interface stuff. Interchange is there for you it's got these API it's got a few others, you can just pull physics parameters out, you can pull the assignment matrices out or the parameter assignment matrices out and get to work using ML tools. And here's a similar vignette. This is showing. We parameterized a molecule, and then we're just showing the bond length because we just wanted to pick a parameter. That's assigned to each bond in here. This is the smiles parameters vignette. Interchange can export to different MD engines. Here, for example, this is a box of DMS O ethanol and some sort of like sugar thing. And we've created it using pack mall, we've assigned parameters, we've created an interchange, and on the left, we exported it to open mm and this is a trajectory of the open mm simulation. On the right, we've exported Gromax and Gromax, I did an energy minimization and NPT thing so it looks a little bit more packed. But yeah, simulations run. They don't explode. Things are looking pretty good. We had a slight bug with water but I think I saw it got fixed this morning. Yeah, so go ahead play around with Gromax. Yellow light, we haven't exhaustively tested out the Gromax exporter but it doesn't explode so far the energy single point energies look good so far. And if you smell anything funny when you try it, open up a bug report for us. There's early support, and this is prototype so this is red. There's early support for creating interchanges from already existing open mm systems. So you can get open mm systems from a variety of places like the open mm force fields package can make you a charm lipid or something, and you could load that into an interchange for combination with other components. Force fields are hugely detailed. Not all things are compatible. So this is going to be a little bit rough going and there's going to be a lot of user experience improvement that we need to do in this field. But here, for example, we've taken a sample six host guest system, and we've loaded it into interchange using the from open mm method so here's all interchange from open mm. And indeed we've evaluated it the numbers are maybe a little bit funny, but we got it in there and it's to a place where we can re export it to open mm we have the data structures it's holding on to it. You can try to export it to grow max that would probably be even uglier. So this one, this one's not wiggling as an indication of how much we pressed it. This is a red vignette. And another red vignette is the plus operator and this is one of everybody love this about permit is you've got the plus operator you can load arbitrary components stick them together and watch them wiggle in the same box. Over here, we create a ligand interchange just using our standard ligand parameterization workflow. We make a protein interchange that's when it's using the FF 14 SP force field and then we combine them together. You don't need to do this to use this workflow by the way, the normal way to do it is to make a single force field and apply that to a topology with both the ligand and the protein. But if you wanted to do it this way you can. And we can use that. And here's the example of exporting it to different formats. There's a little bit of gobbledygook because of bond constraints everybody handles them differently they get a little bit mangled and the exporting so again this is red. But for the most part open mm and grow max are matching closely on a lot of numbers amber looks like we got something fun going on. But these are things that you can do and we're going to be working on improving the reliability of these we're not getting parsing errors anymore we're getting numerical errors. So that's good. Beyond interchange. I believe it was bill spoke mentioned yesterday he tried to tangle with QC archive, it was very difficult. And as scientists tangle with QC archive, we have thick rubber gloves that are called QC submit, which reached out to QC archive, but it's your guide as an mm practitioner in QC land. The open force fields data sets on QC archive have additional metadata saying which chemical graph they're trying to encode along with some other things. And if you use QC submit, we use QC submit to submit those but you're not going to be submitting data sets. If you want to pull down our large data sets you can put on those same gloves pull them down and get graph molecules with energies patch. Here for example this is one of the torsion drives from our protein fitting project which you'll hear about after the break. And this is Bailey and Alan in Bailey and I believe, and this is a 2d torsion drive so it's it's got like a bajillion points, but in the end, you get this nice energy plot, which I put in hard trees, because I didn't have time to convert infrastructure wise we're also building out Alabama scale. This is what David Dawson spoke about yesterday. This is our large scale orchestrator for big numbers of free energy calculations we're going to be using that routinely in four scale benchmarking. I'm not telling you anything wiggling here, not because nothing's wiggled thousands of CPU hours of things have wiggled, but it's a very large data structure in a database and it's not practical to go get wiggling out just for a top. But this is running. It's very exciting. Right now it's sort of a tool for internal use our deployment docs are pretty good we hope we have a lot of researchers at different institutions getting access to this, and I think that you could passively try spinning your own institution but if it has trouble, I wouldn't be too surprised. It's it's kind of a complex piece just by nature of its scale. And so one of our big goals in the next year is to make the user experience for going through the toolkit and getting to a simulation, a lot easier. And so for example, this is the non canonical amino acid vignette or workshop that got Rebecca on the track that brought her ultimately here nine months later. This took 40 seconds to run just execution time and had about 160 lines of code you had to chop out the new amino acid and put caps on it and calculate charges and it was a whole mess to load it to to load it from PDB it's just awful. And with topology dot from PDB, and I didn't use the same modified amino acid in this case just for a technical reason. But here I've made an amino acid it's like Alanine Alanine free me and Alanine but the way Alanine Alanine Alanine free me Alan Alan. But the three means really weird, I modified the free mean it's not just fossil free mean to make sure that this is not anything that might be recognized from a known set structure library it's like phosphorus, oxygen carbon nitrogen, or something. It's completely arbitrary modified amino acid. And instead of needing to do like 50 lines of loading this weird unrecognized thing from PDB. We just, we've added a new argument and it's read, because this is private, but you can feed in now an OFF more where some atoms are identified as leaving. And it will now recognize this as a substructure alongside all the other amino acids that can be loaded. This is not the final API, you can play around with this, have a lot of fun. But your code will eventually break when you remove this option this is just here to show the possibility of how easy things can be, if we can load modified amino acids. So this outputs to over here you can see this is from a PDB it doesn't know anything about bond orders the carbonyl oxygens or single bonds. So the output of this is a full chemical graph suitable for Smirnoff parameter assignment. And so we can go ahead and run the simulation with this using the naggle prototype instead of the AM1 BCC like semi empirical charge assignment. This has a runtime of 25 seconds and it's down to 90 lines of code. So you can go ahead and try running this in the custom sub structures and naggle. So our big goal for the coming year is to improve our infrastructure is usability we have amazing things coming out for the force fields and we want to make sure that we can get those force fields into as many hands as possible. So we want to expand interchange to have sort of vanilla importers for open and amber and grow max and by vanilla.  Nope, we can't import things that are already parameterized and have polarizability. We can't do exotic things but we can do Leonard Jones bonds either constrained or unconstrained sort of the manila stuff that's like the sage tile parameters, we want to be able to import those. And we want to make this really easy for protein ligand workflows like I showed you we can cut down the lines of code for modified amino acid by about 16. And we want to keep cutting that down we want all of these to be very snappy and not require user customization. We want to have expanded polymer loading like I said, and because we run out of PDB codes. We want to add support for PDB X and M M SIF, and we want to make a major push to improve the quality of our documentation. And one of the big things is that we want to centralize our documentation that was organized into all of these little packages that don't talk about each other so if you have a problem and you go to one package docs, you don't know that there's another package that will really squarely do what you need. So we want to go ahead and make it really easy for new users to figure out what tools they should be using and how to how to get started with them. And finally, I want to say, this is going to be the year of benchmarking. So you've seen so many exciting things with polarizability with alternate smirks typing this morning with with all the different things. So as an organization we're looking at this and we're saying, wow, which one of these should we put into the mainline force field first, we can't do all of them at the same time that would be a huge mess. And so we need some way to fairly compare the accuracy improvements provided by these different, these different suggested changes to the force field. And the way that we are standard benchmarking suite had run before was that it was carried over from the season one industry benchmarking project, which was a very tightly knit software stack that was made to work with exactly one version at a time. And the only goals for this were accuracy and deployment. We had to deploy this on pharma computers that we couldn't see that we couldn't debug and it needed to work on the first crime. So as a consequence, this workflow is totally inflexible. It does RMSD, DDE and towards and fingerprint deviation if you want anything else, and you will probably break the entire workflow silently with scientific error, and the numbers will come out gibberish but it won't tell you why. There's a huge demand for benchmarking keeps getting reused. And so we want to modernize it.  And that is where instead of being locks on all sides, there's plugins on all sides. So instead of loading, you know, a preset exact, you know, only QC archive data. We want to have different plugins for different types of data inputs so we can have local QC data, QC archive data or physical property data sets or other kinds of data defined by users. We want to have a number of analyses that can run and again just another set of plugins. The first generation of generalized benchmarking will do the season one benchmarks, along with the sage release benchmarks so these are condensed phase property evaluations. And we want to add support for alternative functional forms, because this is a big point we have really smart people at open force field suggesting changes to our force field that could make it much better. But we need to make sure that we have the benchmarking infrastructure to fairly compare them and do the exact same tests. So phase one is we're going to get this put together the plugin infrastructure and functionality equivalent to the season one benchmarking and the sage benchmarks. Phase two is we're going to roll this out to the entire internal team. And I think this has been a long time in coming, and I think everyone's going to be really excited when we can give talks and have you know the exact same plots saying how our methods are improving the quality of the force field. Depending on the state the state of outcome scale. We might be integrating that so when you have potential force field you want to test using the benchmarking harness it will also dispatch a number of calculations to pulling it home. And we may also do the protein observable benchmarks that you hear about from shape in a few minutes. After that, we're going to be discussing a scope for phase three, and phase three could include more sorts of benchmarks or data sources, or depending on the state of the stack at that time and what our boards think this could become a season two industry benchmarking which does everything that season one did with the program analysis, as well as condensed phase analysis if there's interesting things about that, as well as free energy calculations using the open free energy staff. But that's something that'll be for discussion. Listen for this and advisory board calls if you're on our board. And I think that's the end of my talk I'm a hair over time. So what I'll do is I'm going to release everyone or the coffee break, but feel free to approach me if you have any other questions. Thank you.


elivery of the food growing strategy not only monitoring it but actively participating in the delivery of the actions from that and that's made up of a network of stakeholders from allotment associations community gardens and other forms of growing within the city as well as the external officers group that we mentioned earlier on that brings in other partners from NHS local housing associations Health Improvement Coordination as Police Scotland and we also have our internal officers group that draw together internal officers so the conversation is across the whole council and not just specifically within the parts development side of things I was very heartening to hear that so many different departments and groups and organisations are involved in that plan I worked extensively in community food growing in New York City and at that time, 20 years ago there wasn't that level of awareness that the understanding that everybody needs to get behind the kind of work we're doing and the massive benefits that it can bring to the city Marie, you've got a few more questions you wanted to ask Thanks, convener You've been touched on by others how can local authorities help communities growing in allotments to thrive in areas of low volunteering It's a very difficult one to answer and the one I touched on there would be around the community lines practitioners in the unique position that they have within local GP surgeries I would understand there's a correlation between those who are termed as hard to reach and showing up at GP surgeries The community line practitioners are ideally placed to start to open that conversation to look at a person-centred approach to their health and wellbeing to have that conversation around have you ever considered food growing have you ever been part of an allotment have you went for a walk in a park or these other benefits that we get in directly from our open spaces that's certainly one way of doing it The other way is raising awareness continually talking with people advocating presentations to area partnership committees presentations to our elected members presentations to our community planning partnerships attending conferences raising awareness the areas of socio-economic disadvantage you will find there is grass root organisations there are chapels a great example of that our site at growchapo came about from the quality of life group that was chaired by Police Scotland and they were finding people were dropping between the gaps between our health and social care partnerships wasn't the criminal behaviour it was manifest in this criminal behaviour it was a mental health issue at the back of that it then led to an approach where we looked at an underutilised piece of open space within an area of socio-economic disadvantage we started an engagement process it was a massive response in favour of development into a growing space we've got 35 organisations I believe involved in growchapo each one of those organisations has access to the site and they bring in their user groups their user groups then get access to the site they might never have been aware of before so that's one way raising awareness is another way and the position in the community links practitioners within GP surgeries is also another way where we can get areas of low volunteering and encouraging people to take even if it's not just volunteering volunteering comes secondary and enthusing them with the benefits that growing can provide you tend to find if they've got a fair bit of time available to them they'll contribute that to the site we were in Springburn yesterday and it was amazing that we're just in the middle of the housing state but once you get into the allotment you just didn't realise where you could be anywhere actually I was quite child there to be honest I was going to move on Sinclair you've touched this earlier again the planning system would be better used to ensure adequate provision is made for new and existing community growers anything else you'd like to throw in there thank you yes if I can just touch on one thing from the previous point as well before I move on to that so in Seafine where they do food banks they actually engage with those people who are coming with food insecurity see if there's an opportunity for them to get involved in volunteering and growing their own foods in terms of planning a very interesting point and certainly colleagues of mine have made clear that planning can probably do more so historically growing sites in green spaces are protected in local planning policy probably not strongly enough now we are using mapping much more partly through the section 9 of this act to provide a map for food growing potential but we're now linking that up with the open space audits and strategies that come through the planning system as well so I think there was a suggested link to bring together the mapping for food growing with the mapping for general green space across local authority areas and we're doing that there's a delay from the original food growing strategy to now because our open space audit wasn't done at the same time but we've tweaked the way that we're doing that and we're now looking at all of the land, publicly accessible land across Aberdeen over a certain size and asking the question what is the potential for that land to be used for food growing once we have that data that we can then start drilling down and mapping that out a bit more building that into our local development plan the next time around it's not an immediate solution but it is starting to build it much more into the planning process it's been suggested by colleagues that possibly the planning act could do more than that and that while there's some things in there open space as a whole is much stronger play is much stronger other things are much stronger than food itself so perhaps there's an opportunity to look under the new framework as to how that could be strengthened historically we've not had a huge success in actually bringing forward food growing sites through planning which is disappointing to say the least in a period of significant growth for Aberdeen over the last 10-15 years to not have those sites is a missed opportunity but like I've said if we can look again at how we manage that planning process and how we can manage things like local place plans and engage with communities to understand that there are opportunities that they can build into this around food and locality plans and so on to try and strengthen that approach that gets built into the next local development plan it's all well and good putting it in policy we also have to deliver it to developers and others to actually bring forward sites and there's a bit of education there around our planners for development management but also the developers to bring forward those sites within the planning process just to touch on one point there as well that may be expanding where Sinclair was and it's just round about where developers are bringing forward growing sites that perennial question round about maintenance and revenue after it's been handed over still remains consideration of that and the planning policies would be crucial I would suggest Sandy, just taking back to the visits yesterday I'm understanding that Glasgow is going to add a further 250 growing spaces correct me if I've got that figure wrong but can you expand on that maybe can you tell me how many traditional allotments and how many growing spaces? The 259 plots that we're looking to add through the iteration of this food growing strategy we're looking at each electoral ward within the city so that's 23 electoral wards that we're looking to adapt growing spaces within some of them will be on pre-existing sites where the space isn't utilised to its maximum efficiency some of them will be on new sites and some of them will be in vacant and derelict land each of the growing formats that will be utilised will be determined essentially by the site that we develop so if it's an allotment generally be a traditional allotment site if it's on vacant and derelict land and other contamination issues it may very well be that it's a raised bed format accessible raised bed format and if it's new sites it will generally be traditional allotment format as well but we're also looking at the mixture of more of a blend of the community aspects within allotments as well allotments to a certain degree have been scyloed in as much as it's allotments for individuals but there isn't a community arises from that there's a community place there and there's a community experience as well and that leads to other developments of community and while it might not be everybody's cup of tea to go to a 250m2 plot they might be looking for something a bit less restrictive, less rigid there is space within allotments sites for that and within the grow chapel site which is the newest site we've opened there is certainly consideration of that there is plots for individuals there's plots for organisations small starter plots medium plots, larger plots which allows a life cycle to develop an introductory grower comes in it's starting a small plot, they prove their case there they may have obtained that in your plot that's also led to an increased demand in the city for urban farming as well and that's something we're currently exploring we're an economic development team and our partners in Glasgow community food network so we're also looking at vertical growing using technological solutions from an organisation called INGS who have a research and development strand up at the James Houghton Institute in Invergowrie there's a massive mosaic of different growing techniques that we're keen to explore in Glasgow in developed so it will be a blend of all of those, it won't just be traditional allotments, it will be community gardens and I'll be technologically led innovations as well as the regenerative agriculture approach in urban farming that we're looking at for the site that brings us to the end of our questions and a very useful discussion and I think that one of the things that's really come out both in this panel but also this morning's earlier panel for me is this piece of let's get people started younger and how can we really hardwire this into the education system so that we know at the moment we're having to kind of one of the things I'm hearing is people are faced with a large plot they don't really know what to do with but they have that impulse so if we can get that going in the schools I think that would be more consistently and I think one of the challenges I know from when I was working in the past one of the challenges is summer holidays when suddenly you're at peak growing period and there's nobody there to water the vegetable plot so anyway thanks so much for joining us this morning it has been very useful and as that was the last public item on our agenda today I now close the public meeting


What the previous administration did in terms of setting up and supporting negotiations between the stakeholders in Afghanistan toward a just, durable political settlement to that conflict, that basic frame is something that we very much support and we want to support negotiations between the Afghan government and the Taliban and others to get to that just and sustainable outcome.


Okay, hello everybody, please get seated. So our first talk, our first regular talk of the official DevConf schedule will be by Keith Packard from Flip Packard Enterprise and Bebe Project and he'll be talking about delivering software for memory-driven computing. Please give him applause. Good morning everybody, I've been here in Montreal for about six hours. To answer the most pressing question that everybody's been asking me, where is Bdale? Bdale just got off of a boat in Vancouver, Canada and is heading for the airport. So I'll let you know how his flight goes from the day and we shouldn't see him here tomorrow. Yeah, and I know that that was the movie. Please hand me your talk. Yes, I want to hear you. Thank you, I appreciate that, I do appreciate that but the only question I have been asked so far today is I figured I should probably answer that one first. Okay, so I'm going to talk about how we're using Debian to deliver software for memory-driven computing and I want to first give the 30,000 foot view of what we think memory-driven computing is. I'll show a little demo, a little video for a demo that we did. It's really boring, it's literally a video of a computer monitor. So I apologize for that.  My life in computer graphics, 30 years in computer graphics, it was really easy to do demos and now I'm doing memory-driven computing and it's like what do you do for a demo? You have a web page. So memory-driven computing, what is memory-driven computing? It's our vision of the future of computing. Why are we doing this stuff? The basic motivation is this gap that we've seen and I'm sorry for the corporate slides but this really does kind of give an introduction of why we're doing this, why we think it's important and then I'll show you what we're actually doing. So we're seeing a gap between the amount of data that's being generated and the processing speed of computing. This is basic, basic gap. Moore's law has ended, computers aren't getting any faster, we haven't seen significant improvements in processor speed for what, 10 years now? It's been amazing. And meanwhile, all of us seem to be uploading cat videos at an even ever faster rate. The amount of data that we're capturing continues to increase. Network bandwidths continue to go up. I mean we're shipping 100 megabits networking now. Who would have thought? I lived through the early years of Ethernet and we went from 3 megabits to 10 megabits in like 12 years. And now networking is getting faster, at a fairly good clip. But our computers, our ability to compute is not keeping up. And the basic problem is one of computer architecture. We've had essentially the same computer architecture for 60 years. The processor, and there's some memory, and the only way you can talk to that memory is from that processor. You have this offline data storage, you have these networks you can communicate in other fashions. But all your communication is very structured and very much software-driven. There's no hardware assist for the communication. If you want to communicate with another computer, you have to put together a little network packet and spit out a network packet on your network interface. We would like to think about large-scale systems as being much more tightly integrated than that. We're trying to get to the point where you can actually put a lot of data in one place and have a lot of computers, a lot of not applying arbitrary amounts of computing power to it. So what we're trying to move, we're trying to move from this notion where the memory is kind of a peripheral, a processor. Well, the memory is where all your data is. We're trying to take the memory out of this peripheral notion and put it into the center of the computer. So right now we have kind of two notions of computing. We have scale-up computing, where you get a bigger and bigger and bigger processor and more and more and more memory. Well, Moore's law kind of says that's kind of over. We're not able to do a lot of that. Or you have this distributed mode where you take the notions we built in the 90s, the old clusters and that kind of thing and you just make a bigger and bigger and bigger cluster. One problem with a bigger and bigger and bigger cluster is now all of a sudden your data is getting sharded in a tiny, tiny little fragments. So every processor can only see a tiny amount of the other problem. So you have to work very hard to get your problem to the point where you can actually do useful work on a thousandth of one percent of your data or less. As your data sets get larger, the only way you're able to satisfy that is by breaking into smaller, smaller pieces. So sharing everything, very difficult. From a hardware perspective, we're trying to do that. HPE has a couple of products. We have the Superdome X platform, which goes up to 24 terabytes, 24 terabytes. And then we have the units are hard in my world. I apologize if I misspeak them. And then we have a recently acquired SGI and their MC990X hardware is brought into the HPE product line. And that one goes even further than that up to, you know, 60 or 100 terabytes of memory. So we're starting to get pretty big scale-up systems, but those are really reaching the ends of viability. And so shared everything is not very viable either. So what we're trying to do is we're trying to get something in the middle, something where we have, something where we have the characteristics of scale-up computing in terms of every processor being able to touch all of the memory. And something of the characteristics of scale-up computing where you can add arbitrary amounts of computation and have it able to integrate into the systems smoothly. So we're trying to kind of find something in the middle, find new computer architecture, and basically between these two extremes. So that's what memory-driven computing is. We're putting memory in the center of the computer and then we're being able to attach computing to it. And so we're building, we obviously built something called the memory-fabric test bed in our machine-reach research program. And that's kind of an example of that. And I'm going to show you what that looks like. I have cool pictures. You can see this is actually in four columns in the office where my co-workers work in. You can see that the rack, it's a very deep rack. And the hardware that we built in, it doesn't fit in this very deep rack. It sticks out the front and it sticks out the back. So it's kind of maximally sized to that. The original plan was to actually fill an entire rack, but we had power and cooling issues. And we had actually another interesting issue. It's very difficult to find, to take this new computing architecture and immediately find problems to attack it with it. So we've spent 50 or 60 years sharding our programs into the scale-out computing. All the supercomputers in the planet are scale-out computers. All the problems in the world are designed to be scale-out systems. So it's actually difficult to find problems, big problems that need this kind of hardware today. And that's what we're actually spending this year doing is actually going out and finding some new interesting big problems. But we do have hardware working and it is pretty cool. Here's one of those little nodes. You can see it has... This is just one of the... There were 40 of these in that rack. This is just one of those. It has four terabytes of RAM and a many, many, many core ARM64 processor on it. And we put 40 of these in a rack. We designed it to scale up to 80 in a single rack. So the system that we built is 160 terabytes. You can obviously scale up beyond that. And the way that it connects all these racks connect is through this interesting new memory interconnect. We did this memory interconnect as a prototype of a new system interconnect called GenZ. This is not GenZ, but it's a lot of the same ideas are in the two systems. It's designed to be a load-store fabric, which means you plug all these 40 nodes together and you can execute instructions on the processor and fetch data over the fabric. So unlike a network, unlike even a rocky and enhanced Ethernet where you have this DMA capabilities, those still require software intervention for every transaction over the fabric. With this architecture, you're literally just executing CPU instructions and it's going and fetching data over the fabric. So your latency is very low. Your bandwidth is obviously very high and the complexity of the software is very, very low. You don't have to do any complicated mediation of offering data or figuring out where the data is going now. You can just execute instructions and take advantage of the enormous amount of memory. This is one of the network back planes you can see it has a lot of wires in it. There's a combination of copper interconnects and optical interconnects. Obviously, we're doing a lot of work with optics to make sure that we can reach beyond the rack scale and go to data center scale with the same kind of bandwidth and latencies. You can't get there with copper. It's too big, it's too slow and it takes a lot of power and so we're doing a bunch of work with optics. One of the chips that we made is this cool little X1 optical interconnect. It's literally a silicon that has lasers etched onto it. So we're using silicon fabrication to build these little ring lasers. I went to a tech talk last year at HPE Internal Talk about this and I was just blown away by what they're able to do in terms of improving the performance and getting the bandwidth out of the chip. So it's a multi-ring laser on a single piece of silicon that all feeds into this single optical fiber coming off of it. It's just like, you can do that. It's really cool. So we take advantage of that, of course. And then we try to find problems. One of the problems that HPE has these days is that we run a big network and we have a lot of data coming out of our system. And then we have a lot of people trying to attack that network. And trying to do analytics on the patterns of the attack that are coming in is very difficult because you have to be doing it in real time. This is not a bad process. You really have to be updating your database in real time. You really have to be doing analysis of the data sentence coming in and one of the techniques that we're using is this large-scale graph inference. People use this sort of social network in graphs. They use it to figure out advertising. They use it for all kinds of data where what you have is a bunch of independent agents that have relationships. You want to find out how those relationships affect what the operators are doing. And so what you want to do is you want to balance this enormous graph. It's got, well, this has got this way and this has got this way. The thing about large-scale graph inference is that you can do local computations, but every iteration of that local computation changes what your locality is, which is to say through every step of the graph you have to completely change the data that you're operating on.  The steps in this problem are very small, which means you're spending most of your time doing communication in a traditional scale-out architecture. Memory-driven computing gives us its ability to just do a step and then say, oh, I need different data. Well, let me just go fetch the data. It's out on the memory pool. I don't have to do any distribution and shuffling the data. Let's see, the problem that we're working on here, we have got some numbers for you. Three and a half billion web pages, hundreds of billions of computers. So that's, you know, the scale of the problem is starting to get interesting. The particular problem that we were working on here is actually a security analytics problem at HP analyzing a single hour of data. And a single hour of data that we have is like, I think it's, see if I can get that in. Not tell lies about the numbers here. Find my other piece of information here. And yes, I can't see my screen. So we have a single hour worth of data, which is 20 billion points in a data set. And what we're trying, the problem is, is if you're looking at traffic analysis of a single hour, you're missing any sort of long-term data. What we really need to be able to do, we think is to be able to get it out of the data, and so you take an hour, multiply it into a single week, and you're getting significant more data. So this particular video that I'm going to show you here is just a single hour's worth of data, and you can see how little of the memory fabric tests that we actually need to use. Let me actually go find the video here. Okay, then. If I drag it here, of course, external monitors, the delight of modern high-resolution graphics. See if I can actually make it not overfill the external monitor. I should be able to hit this key. Here we go. Okay, so this is just going to show you our little memory, the memory fabric has been executing this problem with an hour's worth of video trace data. So you can see I have 40 nodes, 160 terabytes of memory, and you're going to see just how much memory is required, what percentage of the system memory is required for this problem. So there's 20 million data points, there's 55 million connections, and I have an enormous amount of memory, and you can see here the little blue spots, those indicate the data that we actually are using for this problem, and they're scattered across the entire machine, and it's going to execute this fine analysis problem. So the algorithm is obviously computing. You can see it talking between the nodes to find data across the fabric, and it's going and collecting data from various points, and it's doing a single step here. It's not terribly, like I said, what do you do when you have a computer, the only thing it does is fetch from memory, and store back to memory, it doesn't do a lot of fun stuff. So you can see here the problem is actually starting to reach out, and using the fabric to take advantage of the fact that it can fetch data from all the way around the system, and not have to do any complicated communication. And so you can see this is actually showing each node, each computing node, going out and touching memory from the other nodes and fetching it for the analysis step of the problem. You can see how much of the memory bandwidth of the system we're using right now is 0.61%. So there's a lot of headroom available here, and this is hours worth of data, so we think we can competently do a week's worth of data in the system without too much trouble, which is pretty cool. And so that's kind of one of the problems we're looking at right now. So the bar charts here are showing you the convergence of the problem, showing you how the actual solution of it, and it's getting better and better and better. We're converging, you know, 32%. And that's to say, as you do each iteration, the graph gets weighted and rebalanced, and you do another iteration, the weights update, it solely converges on a solution. And you can see it's nearly converged by now, and it's done a lot of computation, and that's the excitement of demos on the system. This particular graphic right here, we've since updated the underlying system there this summer, we've got a couple of interns making that software more reliable. Let me show you what they've been doing. Oh, it's almost converged, it's so exciting! We got to show this demo over and over and over again in Las Vegas a couple of months ago. Yeah, that's the excitement of demos on the machine. So you can see we're getting a significant speed up, though. We ran the problem in a scale-up architecture, and the scale-up architecture is literally 128 times slower, so we really have some pretty impressive speed up. It really is all about getting rid of that communication overhead and getting the place where you can take your computation and apply it to the data without moving your data into the copy. Okay, now I actually want to talk about the software that we built. Oh, I have another one. Do I have a Monte Carlo simulation? Here's a Monte Carlo financial simulation. So the Monte Carlo simulation, obviously, one of the goals there is to be able to do a bunch of random analysis on your data. Well, it turns out that if you pre-compute a bunch of the data in this financial model and use interpolation within your pre-computed set, you can generate results a lot faster. And in fact, if you pre-compute a lot, like 100 terabytes of data, it's about 10,000 times faster than computing it from scratch every time. Who would have thought? So the availability of an enormous amount of memory, just an enormous amount of memory without a huge amount of computation can speed up some problems dramatically. You kind of look at the problem a different way. Instead of thinking that memory is just being a tiny memory like 10, 20 terabytes, I think FETA is the actual reasonable scale memory of a couple hundred terabytes. You can start really computing some stuff in advance and change your ability to change a problem from having to operate in batch mode to being able to operate in real time. So we did some financial risk modeling and we were able to take it from something which used to take several hours to take just a few seconds. It was literally 10,000 times faster, just by having a machine with an enormous amount of memory. If you had to do that in a cluster again, the problem with doing it in cluster is you'd have to distribute the problem to the entire cluster and somehow figure out which parts of that data were relevant for that particular request and that would take a bunch of time to transmit that data and in that cluster model it's faster to compute than cache but in a memory driven model it's faster to cache than to compute. I want to just get beyond this stuff and talk about what I actually came here to talk about which is Linux for memory driven computing. Debian is all we use for memory driven computing and because what else would one use? It's the universal operating system. Obviously it scales from my watch to the biggest computer on the planet. So all the software I'm talking about right now is off on GitHub and we're trying to do all the development in the open. It's really hard to take a corporate structure and move it from development in a little closed silo to saying actually your commits are available at GitHub all the time. So we're teaching people how to do that right now. The last step in the block we have is we have some Jenkins infrastructure which is tied to our GitHub Enterprise instance inside the firewall which automatically does our CI, our continuous integration and testing stuff. We haven't got that replicated externally so the developer is like I'm not going to push it, it's not getting tested. I'm like oh I really hate telling people that they have to move external and they don't get testing anymore. So we're fixing that. But that's literally the only stumbling block we have at this point is trying to get to the point where we have continuous integration and testing outside our firewall. But most of our stuff is being done externally. This is the system we built. We have hardware, we have Linux, we have a bunch of libraries and I'm going to talk about some of those. So we came to DevCon three years ago in Portland, four years ago, three years ago, I think it was three years ago to talk about HLinux and what we were doing with Linux in HPE. HLinux was something we were building for our Helion system. Helion was kind of an open stack deployment vehicle and we were taking Debian and customizing it for that. Debian took a bunch of work to make Debian suitable for that not because Debian wasn't ready, but because OpenStack has specific dependencies on a lot of different packages. So we actually take Debian and take random versions of various pipeline packages from anything from really stale to really brand new to try to construct a horizontal stack that can support OpenStack above it. And that's what HLinux is all about. How do we take a Debian system and make it be able to be very purpose-built for supporting a specific OpenStack deployment? And we did that. Helion OpenStack recently got sold from my employer off to MicroFocus and SUSE and it's all very complicated right now. But so HLinux no longer really has a role in our organization in terms of supporting the Helion system. And so what we're doing right now is we're transitioning that from this HLinux base which is how we started the Linux for the machine because we had a Debian system and we needed to support another architecture so we kind of built on top of HLinux and we're transitioning from that purpose-built horizontal Debian distribution to just running Debian and taking a small pile of packages and adding them in. So we have a Debian unstable system and then we just have probably 15 or 20 packages that we've built including some new kernel modules and new device drivers and that kind of stuff. So it really is just Debian running in most of our environments. We're not running this on the MFT yet because we need that kernel bits and a bunch of newer stuff that is not quite even Debian and stable yet. But we're getting closer and closer. So we're taking this transition from our very purpose-built horizontal HLinux distribution to just crunching it down and saying, okay, we're just running Debian and then we're just going to add a few packages. I want to talk about the packages that we're having. So we have two systems that we need to run Debian on. We have our external management system which runs management services and our file system metadata management. And this is where all the kind of interesting packaging stuff and a lot of the stuff that I'm going to be talking about in a few minutes is about. And then on each of the nodes, each of those 40 things, the green lights, they run another Debian system. It's running entirely in memory. So this is kind of like an NFS root environment but kind of not because we don't have... We are actually just... We're building in an in-randis and that's what you run. The nodes are entirely stateless that way from the perspective of the individual node. Of course, they have access to this fabric-attached memory, this enormous pool of memory which is persistent and persists beyond the life of this node. So it's kind of a weird little world that it lives in. So we have these very ephemeral, entire operating system instances. So in a lot of ways it kind of looks like a container sort of thing. We kind of spring this node into existence. It runs an operating system for a while and then it gets shut down. And the data that it's computed is stored in memory, this persistent memory. And so we needed to build a way of getting these things up and running. And I think... I'm hopeful that the piece of technology will kind of inspire somebody to think, wait a minute, that's a neat hack. I wonder if I could use that for what I need to do over here. So I wanted to talk about that. We have three hardware targets that we're targeting for this system.  We have an emulated environment that I can run on my laptop. I can actually build a machine on my laptop. Obviously it does have 160 terabytes of memory, but it does have all the same architectural characteristics. And I did a whole bunch of testing for a program that we'll be showcasing later this fall with a German research institute.  It's like, okay, that's kind of cool. I can do memory development. Memory-driven computing development in, you know, on the airplane. It's always nice. And another piece of hardware that we have is this MC990X, which is this scale-up-ish computer from the SGI division. And it is normally delivered as a pure scale-up system, up to 128 processors, up to a bazillion bytes of memory. I really don't know how much memory it could hold. But it turns out that you can actually take this thing and partition it and break it up into individual little virtual computers with a collection of processors and a networking interface and a bit of their own little local memory. And then they can communicate over the fabric that is in the hardware with other nodes in the same box. So we can build something that looks very much like memory-driven computing, with hardware that we're shipping today. And that means we can actually do a bunch of memory-driven computing research with hardware that we have available today. A newer hardware coming out is going to enable it will scale it up a little bit bigger. It has some other characteristics and interesting. But it lets us kind of do research in the software and systems development we need with hardware that we're shipping today, as well as this prototype hardware that we have. Obviously, with the prototype hardware, the big problem is availability. There's like, you know, four of them. Whereas with this MC990, you know, you can make as many as you want because it's a shipping product. That's three targets we have. So one of the other things we lost when all of our software resources went off to micro-focus was we lost our kind of our system integration and development team. We no longer have anybody working with us that maintains our build system. We don't have anybody working for us that maintains our build hardware in particular. So what we're trying to do is we're trying to build kind of a virtual build environment using containers. We actually had some interns over the summer build us a container that you drop it on a random machine running a random operating system that supports Suza, Fedora, Red Hat, Ubuntu. You just drop it into a random machine and it springs up a devian container and you say go, and it goes and defences all of the software that we need for our system, downloads it from get, compiles it and emits devs. So it's kind of a build in a box. It's just you dump it on a random machine and push it go. That was done by a couple of interns this summer. That's been really useful because it means that we could just go out and find a random piece of hardware somewhere that happens to be working today and get our software built. We don't have to depend upon having the magic build server that's off in the corner and it's, you know, gold-plated and never touched by anybody. And that's been very useful to us. And then we also have another little container that we run that can actually stand up and deliver a devian packages. That's a container that's got a bunch of aptly packs in it. And you just hand it a pile of devs and say go and it, you know, serves devian, constructs a repo and serves devian bits out of that, which is kind of cool. Did I question? No, sorry. Okay. And then we have this, we have external management services that actually kind of run on this external server. And we actually have another container that runs all of these as well, oddly. But we have the librarian, which is our metadata service that I'll talk about, our file system metadata service. We have this manifesting thing and that's the service we built that actually constructs these init-randis and takes packages and builds them and customizes them for each node. And that's kind of the software that I want to spend a bunch of time talking about in a few minutes. And then we have that pretty dashboard which shows all this stuff. And then we have the usual selection of random network services. So this is where all the bits live. They're out on GitHub. All the packages are there. Some of the, I think there's probably two or three packages that aren't currently being kind of developed externally. We're mirroring them externally, currently we're not actually actively developing them out there. And we're trying to, as I said, we're trying to get to the point where we can do that. Because obviously then other people can contribute and until we do that it makes it more difficult for other people to contribute. And that's our current plan. So this is all the stuff that we're shipping and I'm going to go through a bunch of these and tell you what they are. The ones with stars here are the ones that are actually that's where the project lives. There's no clone elsewhere. And we're working on making that happen for all of them. So this is the little container I talked about, the build container. So just a Docker container for building all of our packages. It's just got a script that runs through and get clones and then de-builds them. All the packages that get admitted are unsigned of course because there's no signing authority here. So it's useful for testing. We need to figure out how to make it useful for actual deployment and how we can actually get the packages signed and how to actually make it part of the continuous integration system as well. So this is what we're going to be replacing our currently creaky internal Jenkins instance which runs on a box that has no system in for it right now. God, I hope it doesn't crash. So this builds all the packages that are necessary to kind of stand up one of our little instances. It doesn't build a bunch of random stuff that we don't need for testing. It was done by an intern this summer. Austin worked on this. It is really cool to work at a corporation that has a very strong history and policy of bringing in high school and college interns. I think we had eight or ten of them. We have a group of eight engineers now and we had like eight or ten interns working with us. It was really cool. All of a sudden our group doubled in size in the summer with a bunch of high school and college students which was great to see. Here's the repo container that David did. It's either called DevServe which is the name he started with and then some marketing guy I guess got at him and said we have to put our corporate branding on that and called it the L4 fame repo container. This is not specific to L4 fame or our memory computing stuff at all.  It automatically generates indices, automatically starts up an Apache instance that will serve them out. So it's kind of convenient if you just want to build a bunch of Debs and you don't want to have to go through the problem. I use a mini deep, what is it, mini something. I don't even remember what it's called anymore. Just to do that on my local laptop but this looks like it's probably even easier than that because it's all automated and all you have to do is build the Debs and hand them to it. Very convenient. When you update the Debs, it tracks changes. Those are just, oh look at that, .dev change. Let me go rebuild the index. So kind of a mini decon I think it's called. That's the thing I was using. This uses aptly of course to generate all this data and it's all nicely automated in package. And the other container that a couple of interns did, my and Madison worked on was this, all of our management services are now in containers. You can just kind of, they can pull down the Debs automatically build up a little container and just say go and now you have all the management services. This means that we can kind of stand up test instances or infrastructure really quickly without having anything customized on your own box. So if you want to do memory-driven computing development, you can get this torms in a box, stick that container into machine, instantiate a couple of VMs that run the nodes and get L4 or fabric-attached remuneration running on your laptop in a matter of minutes without touching your base operating system at all. So for people who want to just come and toy with it a little bit and see what it's like, we're working on making it very convenient. I know the thing you worked on by Lillian and Annie this summer was our TM dashboard. That dashboard is that pretty UI you saw before. Trying to generate actionable intelligence about the state of a memory-driven fabric is really hard and trying to capture what's going on and figure out where the bandwidth problems are where the performance issues are where your application bottlenecks are. And so we're trying to build some infrastructure and so this is a little web... it's all web-based. It's a little web service that goes and touches all of the nodes and all of the infrastructure and says what's going on with you? We have a bunch of monitoring and logging hooks there and it captures all that data and presents it in a pretty little web UI that looks like this. You saw that before. So you can see, we're trying to generate data that shows the user what's going on in real time. And that was done by a couple of interns this summer. The new version of that. The old version from the demo was done by a team in Bristol who are now looking... now doing something else. So nice to have that kind of very customized system brought in house. The library monitoring protocol, that's what this management tool uses. It goes out and touches all the nodes and brings the data and shares that out to that dashboard. So the emulation shell script. This is kind of the first stuff we released. It's literally just a shell script that takes the Debian packages that we deliver and constructs a synthetic set of memory-driven computing nodes by using this QMU KVM hack called the InterVM shared memory system. So you generate, you take a pile of memory on your host machine on the host machine and you then make that visible as a device in all of the VMs. Now all the VMs can touch this memory. But that looks a lot like memory-driven computing to me. It looks like a lot like fabric attached memory. So that's the environment that we use to do all of our development for all of these tools. So the initial project we put together is some simple shell scripts that generate these nodes and we're working on improving that to automate at the point where you can actually stand up a memory-driven computing test infrastructure on a single machine. So the library is kind of the heart of our, the way that we take this fabric attached memory and present it to applications. Normally applications think about memory and you think about Maliq or maybe Mmap as a way of getting into memory. Well Mmap takes files and when we started this project a couple of years ago the researchers came and said well what we really want is memory that persists across the file system reboots and chunks of memory that are resizable and have names and have access rights. So we built this really complicated new system called the, what do they call it? The wholesale memory broker and it had this API that you call you pass it a name and you pass this little mask and then access rights and it would map that memory in your process. I looked at that and said you know that looks a lot like something that we have in the POSIX system called a file. Like no, no, no it's not a file. It's memory and I'm like yeah it has storage and it has an extents and it has a name and it has access rights and it looks like a file to me. So we created a file system and the file system is entirely in memory but it's in persistent memory and it's distributed across all these nodes and that's what the librarian does. The librarian manages, we grouped the chunks of memory into a group of pages what's a group of pages is called a book. Our books are a fixed size you know the smallest allocatable unit of memory in the MFT which is only 8 gigabytes and then you can collect a bunch of those so you have a substantial amount of memory into something called a shelf and a shelf is the same thing as a file. So when you talk about shelves and files sometimes you'll see it's talking about that a shelf is just a file but a shelf is a specific kind of file it's a file in fabric attached memory. So the file system is visible across all the nodes so now we have a distributed file system. Well what's the easiest way to write a distributed file system? Well the easiest way to do that is to have a single central server that serves out all the data about the file system. So that's how we built our first instance which is what the librarian is. The cool thing about the librarian is the librarian doesn't actually care about the data because the data itself that's all just memory mapped has to be able to do allocations. Well sure, you can have Book 27 I can't see Book 27 but you can play with it. So the librarian actually runs outside of the fabric and file map on a separate machine doesn't even have to run within the machine itself and all it does is serve the metadata that serves allocations and access rights and oh sure you can have access to that. There's hardware support in the MFT to actually prevent the nodes from accessing memory they're not supposed to go into that today. But that means the librarian running externally is actually literally able to control access to the fabric and cache memory. The way that we built this it's written in Python, Python 3 of course and the way that it hooks into the node operating system is by using fuse. Obviously the read and write paths don't go through the old fuse paths they go directly into memory and we also added end map support which we just typically bother to do. And so it's literally just a fork in the fuse code we forked the kernel bits, we forked the library bits, we forked the Python library bits we created our own parallel universe of fuse that does this librarian thing and that's what library files still looks like. It uses the VFS layer it uses the fuse bits to talk to the thing and the awesome part is the metadata is stored in a little SQL light database to make sure that we had it was persistent and transactional and that kind of stuff. And I am running out of time and I apologize I spent way too much time showing you how cool that we're doing computing not enough about what we're doing here. Metadata, we have a little Atomics library, we have a Hello World and the thing I wanted to spend a bit of time talking about which I'll spend the rest of the time talking about is about this manifesting thing. So what we do for manifesting is we need to generate a kernel for our, for the nodes and that's the only data the node gets the node does not have a real root file system anywhere it runs right out of the main disk. The manifesting service runs this little RESTful service somewhere in the network and it contains this little RESTful service that you talk to to generate these images and it also talks to TFTP server and a so bootp and TFTP and all those usual boot system and it also talks to a DNS service to get the names bound up and a DHCP service to get the IP address is allocated and then we have a little CLI application that talks to this service and it stores the kernels of ramdisk for pixie to use and so what we do is we create this golden image that contains kind of a usual devian install a red usual devian in ramdisk and then we first modify the image for each node we unpack the golden image go in and play with it to set the hostname to set all of its IP addresses and to give it all of its TLS keys that it's going to need to be able to operate in the environment and then we just hang out that ramdisk to the nodes and so the node the node is then able to operate in the environment and have whatever customization needs oh you can also add new packages to that it's like oh this node needs to have a patchy server and what the heck and so you can actually put into the manifest which is why we call it manifest you can put into that the package that you need and the customizations that you require for that system let's see obviously we don't have local storage so they run entirely on the ram so on the torbs we have the dhcd server the tfdp server and then it just serves out the stuff to the over pixie and that's a pretty simple technique obviously a usual system at that point is going to get that in ramdisk the only thing it's going to use it for is to go find its real root file system and it's going to go in fs-mounted or it's going to go find some device or it's going to go do whatever it wants to do but ours is really actually truly this list there's no it's just running on a ram the operating it's just crash as well its state is persisted in whatever fan operations it's done and so it runs entirely out of ram which is pretty cool so manifesting overview uses the empty bootstrap obviously we're just standing on the shoulders of giants as usual it's got this little restful service you can send commands, you can send oh I need to build a manifest for this so I want that manifest to be run on these 27 nodes so it really quickly really lets you quickly configure the entire machine to run the software that you need and then you can quickly transition the machine from one state to another and each of the nodes are running and get the system reconfigured very rapidly so the goal is to be able to transition from one project to the next by just rebooting all the nodes and having to come up with all the new software that we need and that's about what I have time for today thank you all very much for coming out this morning and I hope we have a great week this week I know I'm certainly looking forward to playing with everyone here thank you if you have questions please line up at the microphone hello I have several questions I may not have time for all of them um I'm probably digging into some of the stuff you were skipping over I was curious to go forward aspects of the hardware because of this innovative new design often in engineering we finally shift our choke points from one place to another so in this fabric attached memory we find that the memory controller or controllers are threatened to be your design choke point in the future it's very likely and the reason of course is if you have a lot of contention for data in the same location then you can have a lot of people accessing it's a true fabric with enough bandwidth to handle anybody to anybody at full speed but if everybody is focused on a single piece of memory then obviously you're going to be limited by bandwidth and that single piece of memory is to spread the data across the fabric so you don't have a single choke point like that and so you get fairly flat access but obviously now you have to design your data so that it's spread across the network spread across the fabric to avoid those kind of choke points and then you run into replication challenges yep I'll yield the floor I'm sure it can't be the only one to have thought about this for applications it must be incredibly quick for doing password cracking and stuff you would think how big a table can you get the other problem that I've looked at is chest inking very similar problem any problem we have an enormous amount of data and you don't know what of that data you're going to need today thank you obviously I should go into password cracking what can go wrong what can go wrong okay going forward following this question all computer science all data structures and so on were assuming that memory is in carriers like hash and so on so we have slow memory, fast memory all the databases for example all the indices are to be dealt with slow RAM is fast when everything fits in the RAM as far as I understood there is no penalty to access this persistent memory do you think we are on the verge of of revolutions databases will need to behave differently well we've gotten rid of a bunch of the storage hierarchy so we've gotten rid of your disks and your networks but we still have a lot of caches of course the MFT is not cash coherent between nodes and so you have explicit points where you're introducing latency delay in the system to synchronize data across the fabric and so there are new challenges and plenty of employment for our computer scientists going forward so fear not okay I think that's the last question we had time for I'm really sorry I'll be here all week if you have more questions whatever really great to see all of you and thanks for coming again