would be talking mostly about the NIST MAB. I'm not going to get too much into the details of the various measurements that we've used, but what I'd like to do is give kind of a larger overview of sort of the program within this MAB, sort of how it's become the commonly accepted reference material throughout the community and really talk a little bit more about the life cycle and quality plan of the material. I'm happy to of course get into the details of some of these analytical methods and questions that I thought would be better for this talk to really give sort of a high level overview of sort of how the quality is maintained on this reference material and a little bit of the background and resources that are available. But sort of aside from a lot of the scattering and neutrons and macroscale and measurements, but more a little bit on the line of the more commonly accepted well-characterized biopharmaceutical methods that are more routine in characterization and quality control of these materials. So we'll start off with a standard disclaimer that I'll probably show some values today on the slide. You'll see some numbers, but again note that there's always a reference material certificate. We call this the reference material information sheet. This is available online at any time. So if you do want to check numbers that are specifically associated with the material of interest and for a specific lot of the material, which in the next four or five years will actually have a new lot coming out. I'll talk about some of that in the future, but always just please refer to the current online data rather than these proposals or these presentations to look for values associated with any of our reference materials. So I thought at the links consortium we talk a lot about length scales. And so I thought it'd be good to start off with a little bit of analogy and a little bit of a geography lesson because of course, while many are good with geography, I personally am not the best with geography. So to starting out from where we are, of course, we start on planet Earth. Of course, we can get a little bit more granular at our smaller length scale and zoom in on the United States and even in Maryland, which is where we're located here on East Coast. If you zoom in a little bit more on the state of Maryland, you can see the national capital of the United States, Washington, D.C. And we are actually located just north of Washington, D.C. here in Gaithersburg and Rockville. The actual NIST site is in Gaithersburg and the lab that I operate is actually the Institute for Bioscience and Biotechnology Research, which is just a little bit south of there in Rockville. So zooming in a little bit closer on the geography now in the Gaithersburg-Rockville area, I pointed out a number of very important landmarks for the biotechnology industry. Of course, I actually have not gotten all of the small companies and even some of the bigger companies in the area located here. But the point of this really is to say that NIST is really uniquely positioned geographically amongst a little hotbed of biopharmaceutical development in a broad variety of materials. So monoclonal antibodies, CAR-T therapies, viral vectors, they're all being manufactured and developed in the area around Gaithersburg. So we're really uniquely positioned. Of course, I've also pointed out a couple of other very important landmarks. We're surrounded by a number of wonderful coffee shops. So we have a black fine coffee shop near IVBR. There's a couple of Starbucks around the area as well. And of course, if you think about geography, if you wait long enough, for example, let's say you're home for a couple of years during the global pandemic, you might see another Starbucks popping up here and there, and you'll get another and another and another to say that even our geography, as we go to work every day, changes. Actually driving in this new company where Gen X Bio literally built their building in the last couple of years. And so we really are a growing and budding ecosystem. In addition to biotechnology, wonderful coffee shops to keep us all fueled. So I give this long geography to lesson to say that monoclonal antibodies also require many scales of measurements. And of course, I put the scales and parentheses to highlight that it's not just the length scales of the scattering measurements, but really the types of information you're getting out of various measurements. So of course, you can think about macromolecular structural properties dynamics with things like cryo electron microscopy, various modeling techniques, etc. You can look at a little bit more detail of the monoclonal antibody and start looking at subunits. So you've got of course, fragment engine finding domains, you've got the fragment crystallizable region, heavy chains, light chains, and essentially you can start to really parse down this molecule in the smaller subunits. Of course, you can dig even deeper. We've got a number of post translational modifications to different amino acid residues here, we've identified with peptide mapping. There's of course, dyed, sulfide bonds in each of these individual subunits by sulfide bonds that hold them together, and as well as by constellation. There's also biophysical properties of these materials, sort of global charge heterogeneity, size heterogeneity. There's a significant amount of complexity in addition to a higher order structure of these molecules that really needs to be characterized. And of course, but last but not least stability. So stability is something I think we all need to really consider not only in our measurements, but in understanding our molecules. And of course, I'm putting here a figure of various modes of aggregation that can occur to these molecules. But keeping in mind that individual amino acid residues can be great. They can have chemical, chemically induced post translational modifications. They can degrade during storage, degrade during handling. So really understanding sort of the stability properties of your molecule are very important. So fortunately for us, the NISNAB was designed to be very stable and is a very robust material to be used for analytical characterization. The other point of this story about looking at these various measurements and sort of different ways to think about measurements for monoclonal antibodies is that the toolbox of measurements is always changing. And this is actually one of the impetuses for beginning the NISNAB reference material is that the landscape for measurements is always evolving. So daily you'll see a new report for new mass spectrometer, a new HPLC that can give you higher resolution, higher performance. And so we really need some sort of test metric to follow this evolution of analytical methods and really understand the capabilities as they evolve over time. So taking a little bit of a step back and talking a little bit more about NIS before we get into the reference material. So NIS is actually a non-regulatory agency within the U.S. Department of Commerce. So we don't actually regulate biopharmaceutical products, but we do contribute technology, measurement science standards, and really innovation with our biopharmaceutical partners to try to really enhance the economic security of the United States, improve the quality of life through advanced measurements and facilitating development of these biopharmaceutical products. So we do this program in biomanufacturing that started roughly eight or nine years ago. The program, of course, has developed that measurement science and standards associated with biopharmaceutical manufacturing and characterization. So in this mission, we work very closely with our other federal stakeholders to really understand the challenges of the biopharmaceutical community. We work with instrument vendors quite a bit. And of course, we work with the biopharmaceutical community to try to tackle those measurements and standards challenges that face them every day. So to do this, we draw on a very broad array of expertise. So of course, you have folks like myself that are analytical chemists, but we also have a wide variety of expertise in formulation, higher order structure measurements, things like cyber security. There's even a fire research department at NIS. So we have a very, very complex set of expertise that we can draw upon. And I think last but not least, I mentioned this before, we really are non regulatory. So one of the advantages that we have is we can work very openly with our stakeholders. And we're not worrying about an intellectual property concerns associated with the products that we're measuring, the technologies we're developing. So we can really try to benefit the community as a whole, as opposed to working on a specific product or a specific company. So NIS, as I mentioned, has a pretty broad array. And of course, we're talking about biopharmaceutical manufacturing today. Get a pointer up here. So of course, we're talking about biopharmaceutical manufacturing today, but there are a number of other programs in industrial biotechnology, regenerative medicine, and again, many, many, many other programs that could be discussed that are related to the biopharmaceutical manufacturing. So the reason we're able to get our hands in so many different areas of important to industrial commerce is we have a large infrastructure for this basic research, as well as applied R&D. So we have a significant amount of infrastructure for measurement for various biomolecules from proteins to DNA to metabolites. We have large infrastructure for cell based measurements, growing cells, bioreactors. And we're building out more and more bio analytics to help measure the processes of biopharmaceutical manufacturing. So the essence of this is really we do have a large measurement infrastructure, many, many tools that one would envision to be used to characterize biopharmaceuticals as well as many innovative tools are available to measure a wide variety of properties importance to the community. So we often use these fundamental research tools to better understand the standards needs of a particular industry, and we'll develop a physical test metric, perhaps a documentary standard to facilitate then characterization or some mode of industrial commerce. So a couple of examples that I put over here, there's a genome in a bottle consortium. This actually has developed a DNA material that's useful for calibrating and understanding the depth accuracy and precision of next generation sequencing. So we have a number of bio related materials that can help with things like PCR and NGS. But also we have the NIST monoclonal antibody, another one that I'll highlight today and we'll talk about mostly, which of course is intended for the biopharmaceutical community. So again, these are just representative. You can find a thousands of reference materials at our reference material shop online and sort of peruse them. There's actually recently YouTube video as well that I should have highlighted today, if you can find it popped up on NIST that talks about the reference materials program in a much more broad perspective. And you can really see the diversity of materials from homogenized meat products to sharpie pens all the way to these biopharmaceutical related manufacturing standards. So the program in biopharmaceutical measurement science specifically, we do this in a number of ways. Of course, again, as I mentioned before, we collaborate broadly with the industry. We have a broad set of expertise and we really apply this to a wide variety of modalities. So we're really talking about monoclonals today, but we've initiated programs and viral vectors, RNA vaccines and cell and gene therapies. And we're really attempting to apply these advanced measurement technologies to again, a wide variety of modalities that can be useful for different indications. So of course, this is just a very representative number of companies and agencies that we work with. This is by no means anywhere near exhaustive, but just to give a representative indication of the fact that we try to work as broadly as possible across the community. So as in the previous slide, again, we have basic research mission driven product delivery and products that apply our R&D. So diving a little bit deeper into the some of the measurement infrastructure that we have available that we've applied to monoclonal antibodies. Again, we look at things like protein stability. This includes biophysical type measurements. Looking at things like of course, neutron scattering, sacks, Raman, viscosity and other fluid based properties. And one thing I wanted to point out is a large amount of human infrastructure surrounding sub micron particles that's actually been done in our bioprocess group. And they've actually developed a reference material that's available to calibrate and understand as micro scale particle measurements. So protein structure is the area that our lab falls in and quite a bit of material research has been done on the NIST MAV material. So my group in particular does mass spectrometry and separation science. There's a group at NIST that has extensive mass spectrometry based libraries on this material as well as many, many other materials. It's one of NIST's most widely used products. The group at IVBR under John Barino has done a significant amount of higher order structure research and a number of you folks have touched base with this group in the past. They look at a number of things like HDX mass spectrometry, which is hydrogen, deuterium exchange, nuclear magnetic resonance measurements of higher order structure, and cryo electron microscopy. My group and others do things like binding activity. So surface plasma resonance and ITC to really understand the bioactivity of these molecules and sort of an in vitro setting. And of course, other protein folding and a significant amount of work of simulation that again, some of the NIST experts are contributing to the links consortium in this area. So last but not least one thing I won't talk too much about today is our production cell science group as well. So we're really getting into expanding some of the principles of the NIST map and reference materials to cells. And literally within the next few weeks, we'll actually releasing our first living reference material. This is being developed by Z. Kelman and Brad Odell at NIST. And this is actually a cho material that will express the NIST map. So it's a non originator version or the cho NIST map or CNIST map. So essentially, it should have the same salient amino acid sequence. But of course, it's expressed in a different cho host. So we're calling it a non originator material. But the cell line will really sort of extend the properties or the principles of the NIST map and sort of harmonizing and better understanding measurements from just the physical material all the way up into the bio process. So I think we'll see a lot of innovation around that material in the coming years. So I've been talking a lot about reference materials. So taking a little bit of a step back to talk about reference materials. There is a NIST special publication that talks a lot about sort of the tools that we use, how we characterize reference materials and what sort of categorizes as a NIST reference material. So the term standard reference material is a little bit confusing. The United States calls these standard reference materials. The rest of the world calls them certified reference materials or CRMs. But if you think about an SRM, it's the same as a CRM. So these are well characterized in the measurements in your report on our reference material. These are certificates of analysis. All sources of uncertainty are accounted for. And typically there's traceability to a fundamental unit to the SI. So think that we're tracing a measurement of concentration back to the definition of a kilogram. So we can really understand very accurately and precisely that the measurement of let's say concentration is very, very accurate. And it's done using typically a certified method. A reference material on the other hand is of course still homogeneous and stable with respect to the properties that we're measuring. But some points of uncertainty may not be fully discernible. And you'll see in some of the analytical methods that we used to characterize these material, they report things like relative abundance or populations. And they may not necessarily be traceable to an SI unit of mass. And hence these are typically referred to as reference materials. But it's important to note that all of these materials are produced under the same quality management consistent system in accordance with ISO regulations. And of course these materials are sold on a cost recovery basis. So we're not for profit. We are part of the US government. So essentially the amount of money that's put into a material needs to be recouped by the federal government to basically support continuous monitoring of the materials and development of new reference materials to again support commerce. So there is another type of material that actually grew a little bit out of the NIST map. And we'll see in some of the lifecycle management work that's called a research grade test material. And so these are materials that need to get out of NIST quickly. They need to be available to the community quickly to develop measurements that are of great need. So NIST does a limited quantity of homogeneity and stability and puts that material out into the wild into the public for additional use to evaluate its material properties. Sort of inform NIST on what additional measurements may be needed. And eventually those RGTMs will graduate if you will into a NIST reference material. So moving on a little bit to monoclonal antibodies as a standard, I think I talked a little bit about some of the complexity of these materials. Starting to think about the various aspects of post-translational modifications, linkages with disulfide bonds, things like post-translational modifications including like postulation, and of course, higher order structure. So you can really think about monoclonal antibodies as a very complex mixture of very highly similar species to one another. But of course, every monoclonal in this mixture is not the same. So think about all of these possible permutations in the single individual molecule. And you can tell that it's cousin molecules that are next door may have different levels or different oxidation at different points. They'll have a different glycan attached. It could be folded in a different manner. So it really is a heterogeneous population of very similar molecules. But one thing you can think about even from a specific monoclonal antibody of the same sort of constituent amino acid sequence is that although it's very heterogeneous, all monoclonals that are of course of the same class have very similar properties as sort of a general platform approach. So all of them have post-translational modifications. All of them have the same general structure. They all have the same canonical disulfide bonding pattern. So as a class, they actually share a lot of the same attributes. So in thinking about a measurement standard that can be used to sort of understand, harmonize, and evolve political measurement technologies, you actually can use one molecule to really understand the performance, at least initially, of an analytical method. Of course, you'll need more monoclonals to understand whether the analytical method is broadly applicable or platformable, but at least understanding the initial characteristics and the fundamental performance metrics of an assay can really be done on a commonly accepted, well-characterized molecule. So around 2012, we actually started to talk to the community and ask them about making a reference material. This idea actually grew out of a number of glycoanalytical methods that I was developing in this at the time. And the thought again from talking to the community was that everyone should be using a common standard. It would be great to have a large amount of this material that we could all share and discuss analytical performance methods. And of course, the more robust methods we have, the better off. And of course, we could develop those methods better and better and better if we had a shared understanding, shared test metric that we could all discuss in an open environment. So the vision of the NISMAV, of course, was that we would have a publicly available biopharmaceutical grade material. So of course, we didn't want this to be a regulated product. We didn't want it to be something that was actually a therapeutic marketed on the drug, on the market for use in patients. But we wanted it to be representative of one of those products. So the idea is it's an open innovation material. It's voluntary, not something that people are required to use in order to develop an actual drug product, but something that can facilitate developing their measurement infrastructure along the way. So we wanted this material to be very exhaustively characterized. And not just at NIS, we really wanted to get the industry and the community involved in the characterization. So the first thing we actually did with this material was we sent out the first lot, which is actually called primary sample, PB67, which many of you have used in the past. So this material was actually shipped around to about 100 participants who essentially characterized the material using a wide variety of assays. In most cases, we actually tried to recruit a number of laboratories running the same analytical method.  For example, they would come back together, discuss the analytical results, and then write a chapter about it. So the ACS books are really sort of culmination of both background on the current state of characterization, by current I mean in 2015, along with a series of data that characterizes this molecule with just about every analytical technique that was available at the time. So this is a wonderful place to start if you're looking for what a number of the state-of-the-art analytical methods are. And they have certainly changed over time in the last five years, but I think a lot of the principles and sort of this foundational characterization data for this molecule really still of course holds true today and gives a great overview of the types of analytical methods that are used to characterize these materials. So following the ACS book, of course, and you know this foundational characterization, NIST of course then had to assign materials to this value, make it then more broadly publicly available, and I'll talk about that in the next couple slides. So the uses of this material I think really are broad and I'll try to cover those today, but you know a sort of a vision for it really is to address shared challenges, advance analytical capabilities to better understand their performance metrics, and really evaluate sort of I call this the regulatory readiness, but you know how appropriate and where to implement a particular measurement tool. Because you don't only need measurements at lot release before it goes into the drug, you need measurements during the process, you need it during development, you need it during understanding the properties of the material far before the commercial stage of manufacturing. And not every analytical method is well suited to being a lot release assay, but its importance can still be very great in understanding material properties, looking at stability, monitoring the reference standard for a particular material, etc. And we hope that the NIST have sort of helped to define where and when to use certain analytical methods. So a little bit about the life cycle management. So I had mentioned in the previous slide that we did the community wide crowdsourced characterization as part of the ACS book project. So again the material was actually received at NIST at 100 megs per mill in these sort of set of large bags. So we actually took one of these individual bags and diluted it out and we call that primary sample 8670. So this material was used for the ACS books. This material was used for method qualification that I'll talk about in the next slide. And it's used for internal research. So when I say internal research I say initially, and especially when we need high concentration materials, primary sample 8670 is to go to. When you need low concentration materials we'll actually use the reference material because again that is the material that's broadly available to the rest of the community. But the reason that we have this primary sample was of course so that one we would have material quickly available for the ACS books. So this is analogous to what we would now call a research grade test material, but also this becomes our internal primary reference. So the 10 meg per mill version of 8670 is reserved for NIST internally use only and only as part of the measurement infrastructure to maintain the quality of this material. So this is actually modeled just as a biopharmaceutical company would do. We have a primary reference with which we always prefer back to some suitability to understand the material properties. Now to make commercial material we actually mixed a number of these individual bags and we did this on purpose so that we would have a large supply of the material that was homogenous and didn't have batch-to-batch variability. So we didn't want over the long term to have to make new batches of this material that would eventually potentially have different properties due to batch-to-batch variability. So we homogenized a large quantity of material that we had, allocated that out into one liter segments at the 100 meg per mill and each of these now one liter containers can be diluted tenfold into an individual lot of what becomes a commercial reference material. So this brightly available 10 meg per mill 800 microliter per vial is actually what one can purchase from the NIST website and this is the official material that's called a reference material 8671. So we initially prepared three locks of this material. We're keeping our full life cycle available on these materials doing continuous monitoring that I'll describe in the life cycle plan below but again we also have multiple of these one liter materials available that we can always dilute and vial more RM8671 and because again they were homogenized based with their stability maintains again they'll have the same properties. So looking a little bit more at the life cycle plan this sort of describes what I just talked about a little bit more detail that one container of bulk was actually used as our in-house primary sample as well as our high concentration sample of course some of this bulk is what the folks doing neutron scattering and high concentration work are actually using is the primary sample. The rest of the bulk was mixed into the standard lot which can then be diluted into individual commercial locks. So when I talk about the life cycle plan what I'm really talking about is having sort of a traceable set of physical materials that we can actually compare our analytical methods to. So again our large scale characterization effort initially was done on primary sample 8670. So in theory this is the material that we know most about this is the material that we've used the most and we use this material every time we run one of our analytical assays as a system suitability sample at least initially. What we then did is essentially use the primary sample in our measurement infrastructure as our in-house standard we use that to qualify one of the lots of materials are working lot. So now when we run day to day measurements we always run this working lot of material at the start and at the end of our analytical sequence. So this tells us that our analytical method needs to perform in accordance with our historical method performance data in order to say that that method is in fact in control. Once that's the case we can then measure whatever monoclonal antibody we want in the middle any other samples of NISTMAP or produced NISTMAP or external materials and we know that our method is performing. Now again we'll still use this primary sample 8670 when we do our stability verification and now we use this as our system suitability material to say that our method is still in control and we can again use that to qualify new materials of commercial lots of material as necessary or again do stability monitoring. So essentially the working reference lot is sort of our day to day system suitability control but when we need to evaluate reference assignments or assign new values to a new lot we'll actually make sure our methods are in control with our primary sample which again is our in-house only material. So fortunately if that didn't make any sense at all you can actually read about it in a number of the papers. So the life cycle plan is actually described in detail as part of this series of five publications that we did in analytical and bioanalytic chemistry. These are all open access journals but essentially what this describes is that long-winded life cycle plan that I tried to get out of one slide a moment ago. It also then discusses the qualification where we took the purity and stability indicating methods at least a select number of them from the HDS books. We further developed them and qualified them in the next two publications. So this includes things like CIEF which is used for charge characterization, capillaries of an electrophoresis that's used for charge heterogeneity, a number of size heterogeneity based methods so capillary electrophoresis SDS. So this is basically looking at fragments of material, size exclusion chromatography which looks at aggregates, dynamic light scattering for hydrodynamic radius and MFI or microfluidic imaging to look at particles of the monoclonal antibody. Of course we also developed a well characterized peptide map that has a robust digestion procedure that can be used to verify the identity of the molecule and then we used all of these methods in a value assignment to actually assign values to those three individual lots of material. So essentially we have a life cycle plan, qualified a variety of analytical methods that are stability and purity indicating and then we assigned those values to the NIST map again using that measurement infrastructure that I described on the previous page. The values then for this assignment are available online using this reference material information sheet. Again each lot of materials that comes out will have its own set of assigned values but it's worth noting that again most of these are going to be equivalent and in this last paper we actually verified the equivalence of the first three lots. So we do actually assign specific values based on the measurements but statistically they are the same molecule as they should be because we did that homogenization as part of the life cycle plan. So stability monitoring is of course extremely valuable to these materials so of course if you have the NIST map today and you have the NIST map in five years the hope is that you're getting the same material. So we do a consistent stability monitoring program and we actually update and verify our public reference material information sheet at least every five years although we're doing a number of these measurements pretty consistently in our lab. So we're also sort of continuously monitoring the stability of this material. So this is a very complex process when we do sort of our formal evaluation. It involves getting instrumentation, a homogeneous sampling of the remaining sample in the lots. We typically get about 10 different vials of material across the entire lot. We have to organize our analysts and really kind of then facilitate data collection. So the data collection analysis we actually do this for all of the property values that are on the reference material information sheet. So this is each of the methods that I showed in a previous slide. Those methods of course as I mentioned must conform to performance standards so this is using a method specific instrument qualification standard as well as that primary sample 8671 to ensure that the system is suitable. So we essentially go through data collection again of numerous samples that are selected throughout the lot. We document each of this with internal reports and there's actually going to be a external report coming out very soon that's much more detailed. This special publication that talks about all the details of the stability monitoring for at least our five-year time point. This then goes through a series of submissions and reviews, use internal paperwork, evaluation of the statistics by our statistical department and ultimately what you see is again an updated version of that reference material information sheet. Now of course what we hope in the stability monitoring is that all values are consistent with the original value assignment. No values have changed and the material is stable and homogeneous and this in fact was the conclusion after five years of course that all of our methods are still in check and we still have a very stable and homogeneous molecule. So just to go through a little bit of an example of what one of these analytical methods look like, this is one analytical method that I sort of preach that everyone should run especially if they're storing their material for long terms, if their measurement is a very long, long measurement. Doing something like secretary and post can assure that there's no degradation but essentially making sure that the molecule is not aggregated. So size exclusion chromatography is a separation-based technique that's done based on size so essentially smaller molecules can get into the pores of particles, larger molecules are excluded so larger molecules come out of the chromatography column a little bit earlier. We're calling these high molecular weight species so this is dimers and trimmers and tetramers. The monomer is of course that single individual antibody that we show in all the beautiful pictures and then we have a small peak of low molecular weight species that are fragments of the molecule so they split off light chains and other clips and degradation ceases. This is actually just a buffer peak in this and associated with the protein. So what we say here is that when we're running size exclusion chromatography we actually have a table here of our original assigned values. So the values have a size heterogeneity so the monomeric purity, the relative proportion of high molecular weight species and low molecular weight species and uncertainty with that measurement that was evaluated over multiple days, multiple users, multiple columns get a very robust understanding of the measurement position and then we have a control factor so essentially the expanded uncertainty is the combined uncertainty. Think of a one standard deviation multiplied by three and that's where you get your control limit or the expanded uncertainty. So talking about how we're going to evaluate the stability of this molecule essentially we're going to say that the property value should stay unchanged when they are the mean plus or minus three standard deviations which is essentially this expanded uncertainty. So when we run our value assignment again we'll use an instrument specific qualification sample so this is unrelated to the instrument it doesn't require any sample preparation and it's not the antibody but really just test the performance of the instrument. We'll then use our primary sample 8670 as the system suitability so this will have any sample preparation, true handling of the material can really be representative of the specific material the reference material because it is nominally the same identity and then we'll run the reference material itself that we're evaluating the stability so our injection sequence is essentially IQ system suitability a number of samples of reference material and we follow that again with an end of day check of the system suitability and the IQ. Ultimately what we hope to see and this is just a plot of again this is just our five-year stability we can see the individual data points from an early set again to run and triplicate but an evaluation of the zero year or the time of value assignment and then an evaluation of the five-year data you can see plotted we have our mean of the original value assignment as well as our plus or minus three standard deviations and just like any free to control chart again if your values are within the standard uncertainty your material conforms to expectation with that particular property value so again we show that our IQ standard our system suitability all within the expanded uncertainty so we therefore can say that this material is homogeneous and stable with respect to this property value so again we're continuously monitoring these samples we're running our working samples as part of our normal research work as well but this is just sort of our formal five-year stability verification that's required to assure that the material maintains its homogeneous homogeneity and stability to assure that again our stakeholders are getting a stable continuously maintained product so we also have a number of resources available so Katarina's mentioned this in her talk I believe at the the national meeting for links so then this map resource portal is somewhere where everyone can sort of go to get certain information about the material so you can always go back to a program summary that's on the NIST main page that we're updating actually at the moment but you can also look at the featured publications so this is really the ACS books the ABC publications these are sort of the things that I talked about that are related to the life cycle and quality of the material but we also have a list of publications of the NIST map that are continuously being updated so this is any publication that someone published the word NIST map one word or NIST space map two words so we sort of cover all the ground but this is where you can really go to quickly find a list of who's publishing what on the NIST map both internally and externally and sort of really see the full list of publications coming out of that in addition there's this frequently asked questions page that has a number of sort of commonly asked questions related to formulation and other things that might come up quite often in this map so one of the ways that NIST has actually used the material is through technology maturation so again I'd mentioned that NIST has a very large program and a wide variety of analytical measurement techniques everything from biological activity all the way to this very very detailed characterization of higher order structure with neutron scattering, prioem, nmr, and even very detailed multi-octopic mass spectrometry so again I've talked about this a number of times that technology really matures you know separation science started with only gels not to say that gels aren't still important and understanding the molecular properties but you know this has advanced a lot now to capillary based methods and these really high resolution analytical techniques so our measurement infrastructure is really focused on sort of evolving these analytical measurements and trying to better understand a number of these really high resolution characterization technologies so one of the ways that we've utilized the material internally is to essentially identify an analytical measurement that is evolving offers a lot of value to the community if it does advance but maybe it needs still needs a little bit of harmonization maybe the ability to make those measurements accurately and precisely across labs broadly isn't quite known yet so we'll essentially recruit a number of participants throughout industry academia and federal labs depending on the study will write a very rigorous analytical protocol often trying to minimize as many points of uncertainty in a measurement chain as possible so for example when we did a triptych digest in the multi-attribute method consortium we actually did the digested mist to take that variability out and only had the participants run the samples that were pre-made for them on their measurement system and we even defined the chromatography conditions of the separation so we try to harmonize a lot of aspects of that study so each of these individual labs are then blinded so they can submit their data back to NIST knowing that we're not sort of ranking who is making measurements better than the other but it's fully blinded fully anonymized and we can then look at the data from the study and better understand how the community is performing on an individual method so we can really cross compare you know understanding the performance across laboratories across instrumentation so we really say accuracy and precision it's not just in one lab but broadly how does the community perform this allows us to identify gaps in analytical methods for further development it allows us to actually provide feedback to those labs that might be just implementing a new technology because even though the data is blinded to us they can see how well they perform in comparison to their peers and so they can sort of better understand where they are in comparison to other other companies so this sort of allows folks to evolve their analytical measurements better understand their analytical measurements it's all done in anonymized blinded fashion but it really does start to inform on performance metrics of these new and emerging methods and helps to sort of evolve them as a community so I'll point you to a review that Katharina and our team recently wrote that came out in a special edition of Frontiers and Molecular Bioscience that really summarizes the outputs of each of these four studies and then of course if you're an expert in one of these methods really diving in to some of these results I think could be of interest to you but the real point of this slide is to say that our interlaboratory studies are extremely rigorous we try to define the measurement technologies and needs as much as possible and really try to understand the depth of these measurements and their capability to try to evolve them into the next stage so I think I'm going to skip a couple slides so that we can answer questions but those are really digging into the mass spectrometry study that Trina Mooshauer and our group led recently but just to say that these round robins really do have a lot of participation from biopharmaceutical companies, instrument vendors, regulatory agencies and even the software vendors that are helping to try to develop these methods. The new peak detection study this was actually a mass spectrometry based study and we have a second publication that came out this year on this study that has about 100 contributors to the individual publication so you know that the data acquisition and data analysis is of course a daunting task but getting a publication through a leader overview of 20 or 30 companies I think is equally challenging but of course we managed to do that because this is all done in the pre-competitive non-IP space it's all blinded and that's why we're able to really get the sort of big group data out into the world that are understanding these methods and I think that that really is invaluable. So kind of the conclusion of all of this is the NISTMAB we've characterized it both internally the NIST externally and a lot of people are using and publishing on this on this material so one of the questions I get a lot I asked a lot when I'm out is what makes the NISTMAB so valuable and useful to the community why should I use the NISTMAB what is valuable about it well I don't really say what's so valuable but you know it's the who right and what really makes the NISTMAB valuable is the community the folks that are making analytical measurements that are understanding the properties of the material and using that to advance biotechnology that's really why the NISTMAB is so valuable because if we look at the report on all of the publications all of the value that folks have actually used the material for there's of course been a number of interlaboratory studies both organized by NIST and organized by external consortia there's been the free books we talked about a wide variety of patent applications that actually use the material for example developing a new analytical method and it's been featured in a number of application notes by instrument vendors to essentially show off the performance of their analytical methods so it really is the use of this material I think the foundational characterization but every time someone makes a measurement on the NISTMAB material it benefits the entire community and it really does make the sort of ecosystem more important more valuable to evolving biopharmaceuticals so every time you make a measurement on the NISTMAB I really do think that it's evolving the landscape so there's still a lot to do of course so biomanufacturing of course you know I show this as a biological input cell going into bioreactor purifying and formulating and this sounds like a nice easy process but there's so much that we can still improve intensify to try to get value out of these materials so not only monoclonals of course they still need intensified processes to combat things like the coronavirus and other emerging viruses of pandemic potential to get the scale the speed to get antibodies available for use that actually can be very highly effective but all of these other materials that are being developed have a lot of measurement needs that are very similar adoptive cell therapies mRNA vaccines and viral vectors all share a lot of the same measurement needs to understand identity potency purity and stability in addition to that there's of course a large amount of bioprocess measurements so again we need to start thinking about moving analytical measurements more online to make things faster there's modeling and simulation of these products there's still a lot to understand about the formulation especially a very high concentration and a lot of modeling and digital twinning can be done in the bioprocess so safe to say that while the industry is very well developed there's still a lot that can be done and I'm very thankful that Lynx is using the NISTMAB and tackling you know one of I think one of the most challenging portions of this is really understanding that macro scale dynamics of the molecules and how that relates to formulation and stability and again having that data set available to the community is going to be a huge opportunity I think for developing new modeling and simulation methods I was mentioning to Mikhail that literally just this week we got a question about how sticky is the NISTMAB and you know how can we better understand those properties of the materials so again I think we're in a great space with this consortium and very happy that you all are working on that so the global conclusion is you know we at NIST really want to help to refine and help evolve analytical test methods and we do that of course in collaboration with everyone in the ecosystem so really those NIST measurement tools and reference materials of course being central to innovation and quality are really done in collaboration with a wide variety of folks and hopefully we can use that to implement and accelerate the variety of treatment options available so of course if there are acknowledgments there are so many people to acknowledge of course all the folks that have made measurements on the NISTMAB the NISTMAB team that's involved in the certification and recertification process our web developers at IVBR and of course this is not an exhaustive list there are so many people to thank including a couple Darryl Davis and Oleg Vorsov who really helped lead the book project with me and a number of other folks throughout the history of the NISTMAB that have been really invaluable and in making the material what it is today so with that feel free to reach out to Kat or I if you have questions on the material at any point in time we've also set up this kind of general email NISTMAB at nist.gov but of course you all know Kat and I if you have questions about the material feel free to reach out and with that I'd be happy to open up to questions I think I left a little bit of time and thank you all for having me


How do you feel like Dante Fowler came out of, whatever day yesterday was, came out of yesterday and into today? It seemed like he did a little bit less in team drills today than yesterday. What's that? It seemed like he did maybe a little bit less in team drills today than yesterday. Like how do you feel like he came out? It's good. Again, we're trying to be smart with him. Like he was ready to go yesterday and like I said, he's got the fresh legs. You know, these other guys are in week three through camp. But again, we got a plan to bring him along to make sure as we get through camp, everybody finds a role, come out of it and ready to go September 12th. And maybe this is just neat being due to you. Like when you have a guy in the yellow, like Ole, he's been around for a couple of days. Does that don't hit? Is that he's on? We'll keep him out. Yeah. But I mean, again, it's you can put him on there and if I put him in there if a bunch of runs and shame on me, it's for him, you know, he's out here returning to play. He's dealing with something and to let the guys know, like you get him working in 7-on-7, you get him working some of the passing reps and to know to just tag off him. And maybe I missed this. Jason Spriggs. Is there something going on? Yeah, he's just resting through something. When we signed Jason, that was the deal. You know, he was recovering from an offseason injury he had coming out of Chicago last year. So it was a long-term plan and he's a tough guy and we're just trying to make sure we're not, he's not having any setbacks, but he should be fine. It'll be a few days of rest. When we've seen Dante Fowler kind of like at his best, he's been a really dynamic presence. Are you excited about the potential of him going and getting him back to his fine performance? No, I'm excited about the potential of our defense and the finding the best guys are going to rush the passer and we may, you know, it's going to change week to week schematically what we're trying to do with all those guys. I mean, we'll see who the four or five guys are, who's up that week and what our plan of attack is on third down, but it's too early to say right now.  So, but we got a lot of guys that are in there fighting for roles and he's just one of them. There was something that you were talking about yesterday that I thought was very interesting. You were talking about having to get fair evaluations and how when you're looking at it, you have to be very objective and understand if there is perspective of what you're looking at because it can change because I'm about to be terrible at training. Right. At what point in your career did that really sink in for you? I don't remember exactly when it just kind of over the years. And then it really in going into 2019, it's just something I believed in and just even the way that you go through a preseason game because different situations come up. I mean, you hope to the games even so you can stay in rhythm and you get the best you can apples to apples comparisons with this with whatever you're calling, you know, you got to see the big picture and I've been around a lot of different staffs and thankfully in position now I can make some of those decisions where keep perspective of what the preseason is.  But you know, if you got a battle in there at guard, you need to call the same stuff to see them actually do the job.  It's not, it's not a fair evaluation. So, and then mixing the matches and making sure I can practice, okay, this guy's playing guard, make sure he gets to go against Grady. So you're trying to constantly evaluate that and it's never going to be perfect, but you got to put the big picture on perspective. That was a good thing. You're trying to win the press conferences, trying to win games during the season, and the same thing with the preseason. We want to go out and play well and we want to win, but you also got to understand there's a bigger objective as we go through the entire camp. Coach, how confident are you in that Drew Dolman's ability right now to step in and sort of play maybe two different positions right now, whether you're injury habits or just maybe how it makes the channel? Well, that's kind of a subjective turn. Look, if he's one of the guys up, whether he eventually earns a starting job or he's one of the eight guys up, we better be confident that we got the wrong guys dressed on game day. So that's what's fair to the team. So far, again, like all rookies, they're going to go through struggles, but pleased with what Drew's done with throwing at him. It hasn't been perfect, but happy with how he's progressing. Sure. Can't move it around like that. How is the sort of chemistry that you guys want on the line? Again, it's just way too early to say. That's what the situation is. You start camp. If you've got injuries, some years you may have five experienced guys, but you don't want to get complacent there. It's a fine line of making sure you've got the five guys they had enough reps together. But then they flip side of it, if you're dealing with injuries or you've got guys coming back that are maybe different spots coming off PUP, that's life in the NFL some years, where every week this guy's down. OK, who's playing left card? OK, we have a backup center. So those are all trying to problem solve us. We're going along too. So that's kind of where we're at. All right, Tom, you're right. Coach, are you at all, maybe not surprised the wrong word, because you've scattered in broad years as a free agent, but Mike Davis, his versatility to be able to bounce outside, one pass out, and he screwed one a couple of, I mean, you're going to get Deon with this couple of linebackers in the league. You beat him a couple of times. Are you surprised, or do you believe it? I was surprised. And it helps, you know, when you're putting together a staff and you've got guys that have been other spots, we've got guys that I've worked with them before. Charles and Dave are going to have been with them before. We scout them. You like what you see on film. You know people that have worked with them, coach them, maybe recruited them. And understand the role that right now, if he can do more, we're going to use him that way. So, but again, it definitely helps having guys on your staff that have worked with 3-8s in the floor. This could be a weird question, but your hat, your pullover, do you have like tradition, like superstitions or practice, anything like that? Your first year, so. Yeah, no, I don't think anybody is buying the ticket to come look at me on the sideline. So, I don't have that kind of ego. And a lot of it's just practicality, you know, that I do have more stuff in the locker. But it's just kind of getting a rhythm, you know, for that yesterday or the walkthrough, and it's grabbed the next thing. I didn't notice it. Now, I'll tell Ray, when I see him, he's wearing the same thing every day. It's starting to get yellow. So, Joey and the equipment guys down there, Hoss, need to get him a new pullover.  Some guys are like, you know, that's what I'm sure about. I mean, there are certain things, yeah, sure, that I like that I'm comfortable with. But it's not like, I don't think I'm sweet and awesome. You know, I don't spend an hour like Matt LaFour does looking in the mirror before he warms out there. So, you can tell Matt I said that. The match-up, the always-into-a-watching is Christmas and Vaughn against Blake and Jerry. And how those guys kind of compete like that. Like, what are some of the attributes that you pay there that allows for solutions to kind of compete on that level against the guy that had to? Both of them.  Grady's done a good job, but Grady's continued. I mean, every year, you've got to prove yourself in the NFL. And Chris, same with him. I mean, it's a good battle. You hope to have quality starters like those guys going. It's only going to make them better. It's going to make our team better. And they've got a nice job. It's a healthy respect. You can see that. That's what you want. Both those guys work extremely hard. Both of them are tough guys. But it is fun to watch them compete. Mary Zuck. Yes, Mary Zuck. I wasn't taking a shot of you when I said before. So, so. Chris, like with Cam, he said he spent the off-season kind of training with a young way, and everybody talks about distance and hang time, I mean, important aspects of the job. But also, you have to hold. Yes. Talk a little bit about that. Sure. That aspect and how important that is to determine the kicks for you guys. Yeah, it's another factor. I mean, and you're right, it's one that gets overlooked a lot, but it is important. Those guys, you know, I know a couple of years stretch there when I was in Tennessee, you could tell with Bo Brinkley and Brett Kern and Suckup, you know, they had a good. Rhythm going there. So, yeah, it, it all, it does matter and it's a big factor.  But chemistry is there, especially the specialists, you know, it's a unique cult. I forgot to mention yesterday, young ways, arms are definitely bigger than the Suckups too. So. As far as hang time, do you have preference to a man who has a combination of both? It just depends, you know, it takes an account, right? You know, obviously, where we play at Mercedes-Benz, you know, you should get pretty good weather, at least nine times, I guess you're saying 10 now, you get nine home games in 22 and you're only playing one in the, in the calling out of the Caesars Superdome. Yeah, so nine to 10 games, you know for sure you're going to be in door and what are you doing, you know, if you're going to get into the big directional punting game or you got a guy that can go up there and put it up there and with a lot of hang time. So take it all on the consideration. As a coffee corner kind of lost his duster, that's still part of it. You know, a great guy to talk to about that, Steve Hawkins, Steve's like, I mean, I'm not, to be honest, sincerity, Hawks been around the kicking game a long time in his league and he's a great historian of the kicking game. That'd be a great question. Just get him to talk to the Hawks, I'm serious. Steve, ask what that was, please, thank you so much. Anything else? Appreciate it, thank you. Go.