Okay, this is a problem where we are trying to calculate volumetric flow rate through Alaskan pipeline. The pipeline is four and a half feet in diameter and the length of the pipeline is 800 miles and the velocity with which the oil is flowing through that crude oil is is 0.0 feet per second and what we are up to is to calculate how many barrels of oil are flowing through and what is the volumetric flow rate and also the mass how much mass because the Oil is flowing through the through the pipeline. It acts like a storage bin for us So let's calculate first the volumetric flow rate volumetric flow rate is Calculated by knowing the area of the pipe and we know the length of the pipe is 800 miles so area of the pipe or cross sectional area would be Pi R square or pi over 4d square since diameter is given we could use that pi over 4 times d diameter is 4.5 feet now Square that will give us the area now area times length would give us the volume volume is now Pi over 4 times 4.5 feet square All right times The length that we have is 800 miles So this is in feet square or feet length and this is in miles. So we need to convert these miles into feet again, so one mile is equal to five thousand two hundred and eighty feet So now we have miles and miles gone. So we will have Everything in feet so the answer will be in cubic feet So that will be the volume and the velocity with which It is going through is six feet per second. So we can calculate the volumetric flow rate actually So volume of the pipe is this but the rate at which it is flowing will be six feet per Second, so let's calculate the volume of the pipe actually how much we can store in here So this is coming out to be six hundred and seventy-one 457 600 feet cube All right, this is the volume if the Velocity is six feet per second or let's calculate the barrels of oil that we can store here But immediately now with this barrels of oil that would be more logical here. So the conversion factor is one feet cube Equal to Point Two three 74 barrels Actually one barrel is 42 us gallons and we can convert to First gallons and then gallons to keep cubic feet also in a conversion factor But straight away one cubic foot is equal to point two three seven four barrels So now if we were to convert this To barrels now one feet cube equal to Point two three seven four This is barrels So now the answer would be in barrels So that would be so this will be one hundred and fifty nine four hundred and four Zero three four So we are talking about 159 Million barrels 159 point four million barrels of oil that we're talking about here That can be stored. So that is a large volume that we can store in the pipelines So similarly for natural gas also we have a lot of storage capacity in this country We have a million miles of pipeline at 64 bar or atmospheres pressure The gas is transmitted throughout the United States. So which means Basically underground storage or transport can provide a lot of storage area Now let's get back to this calculation of Mass flow rate if we know the volumetric flow rate, we can calculate the mass flow rate So what for volumetric flow rate? We need we have the area volumetric flow rate now This is the volume that we calculated volumetric flow rate Is we have the area and how much is passing through this cross-sectional area? so this is the area of of The pipeline and every second six feet is passing by okay, so the volumetric flow rate would be Pi divided by four 4.5 Square this is the area times Velocity would be six feet Per second So every second This much oil is passing through which is basically 95 0.3 Feet cube this is feet square Times feet here feet cube per second so every second 95.3 feet cube of Oil is flowing through so per hour if we calculate it would be times Three thousand six hundred there are three thousand six hundred seconds in an hour, so That would be so every hour we are talking about 343,359 Feet cube of oil per hour Okay, so we can we can go on to do the calculation in a year how much we can we can transport by multiplying this by if 24 hours of flow per day and Then there are 365 days There are 365 days so Days and days and hours and hour get cancelled so you know how much oil is going out per year So that would be Believe it or not that value comes out to be three Point zero Times ten raised to nine roughly this is feet cube three billion cubic feet of oil is flowing through So if we were to this is per year if we do the same calculation as how many barrels are flowing through We can multiply this by point zero point two three Seven four like we did before this is the barrels to feet cube conversion So feet cube and feet cube get cancelled how many barrels of oil are we transporting through this per year? We're talking about here 714 Million barrels per year. So that is the flow rate. All right So that gives us the volumetric flow rate mass flow rate is this times density Okay Volume times density. Let me erase this here. Okay now specific gravity is density of a medium divided by the density of water so specific gravity is given to us specific gravity is given as point eight six and Specific gravity is actually Row of oil specific gravity of oil here row of the medium divided by row of water Okay, so Since we know the row of water which happens to be sixty two point four pounds per cubic foot and Row of oil can be calculated because we know the fraction. This is point eight six So in this case a row of oil would be point eight six times sixty two point four This will give us pounds per cubic feet and we also know that There is the total Or mass flow can be calculated by mass flow is volumetric flow Times density volumetric flow is calculated as Three billion feet cube for a year Times density is Point eight six times sixty two point four pounds per feet cube So feet cube feet cube can be cancelled. So now we know the mass flow what we are getting This is coming out to be three point eight three times ten raised to ten This would be Pounds per year So this is the mass flow of the oil. This is a lot of mass. We're talking about Flowing through the Alaskan pipeline. That's it


Good afternoon. Welcome to our session. Distributed and enabling, sorry. Enabling coordinated checkpointing for distributed HPC applications. My name is Adrian Reber. I work at Red Hat for nine years now. I'm involved in checkpoint restore, which is the basis for all of what we're doing here today. For 14 years now, we're doing everything we do with checkpoint restore. It's based on CRIU. I'm involved there, I guess now 12 years. And since I'm at Red Hat, I'm mainly focusing on container migration. Today here with me is Radistin. Radistin, I'm a PhD student at the University of Oxford, supervised by Professor Weiss-Armor. And my research is focusing on enabling checkpoint restore for HPC posters. Okay, and today's agenda will look something like this. I will go through the background of checkpoint restore, a bit about the history, why the names are the way they are. Then I'll talk about integration into existing container run times, engines, orchestration, then I'll talk about use cases, why you would want to use checkpoint restore in combination with containers. Then I will do a migration demo. This is not coordinated yet, it's just a simple part migrating from one host to another host. The distance between will be about 500 kilometers. And then Radistin will talk about coordinated checkpointing in an HPC environment, why it's necessary, and what the challenges are there. So, background. So the tool we are using here is called CRIU. It's called checkpoint restore in user space. And the reason for the name, especially in user space, is because it's a tool and there were many different tools before. So checkpoint restore is a technology in operating systems and Linux for a long time. In the early years, 20 years ago, it was mainly done in high performance computing. And the main use case was fault tolerance. If you have 1,000 nodes running, something stops working, you don't want to restart from the beginning. So you're looking into checkpoint restore your compute jobs regularly. And at that time, there were two tools developed around 20 years ago. One was an external kernel module, which was never entry and which you had to compile separately to be able to use it. And then you had to preload libraries. So the solution was never what you would call a transparent checkpoint because you had to prepare your application, either recompile it or prepare it in some other way. To solve this problem, there was an approach for checkpoint restore, which was in the kernel. It was a huge patch set around, I think 2006 or 2005 or eight, I don't know, some time ago. And it was a working solution to do checkpoint restore in Linux. But the problem was the Linux kernel community was not convinced of the approach. So the approach kind of died. And so people were looking at another way to do checkpoint restore in Linux. And the next one is now Crewe, checkpoint restore in user space, mainly because the one before was in kernel. So now we're in user space. In user space means there's no specific code to do checkpoint restore in the Linux kernel. In user space also means that Crewe was designed in such a way that it was using existing interfaces as much as possible. Over the last 10 years, Crewe introduced additional interfaces. None of the interfaces introduced in the Linux kernel are specific to checkpoint restore. They're usually kind of, how can I get more information about my running process? And so often they are often used for debugging and similar things. So this is about checkpoint restore, Crewe. And now to the integrations. So there are multiple integrations of Crewe in different container. Runtimes, engines, orchestrations. And the first one I usually want to mention is OpenVz. I never used OpenVz myself, but they were basically doing containers before the word container existed. And what they were doing, you would probably call today something like a system container. So not like an OCI container where you have just one application running in there. You had a whole operating system running in there. And they wanted to provide their customers, their users a way to live migrate those system containers from one host to another. So they came up with Crewe. And the good thing here about Crewe to remember is that Crewe was designed for containers. All the other checkpoint restore implementations which existed were for high performance computing. And they are still used in high performance computing today, some of them. But Crewe was designed for containers and that's why it works as good as it does today. So another interesting integration of Crewe is Borg. That's Google's container, orchestrator, engine. Also something I never saw myself, but Google came to us, the upstream Crewe developers. Maybe, I don't know, seven years ago, they talked at a couple of conferences how they use it internally. So before using Crewe in their container engine, what they did is if there was a low priority task running on a node and resources were getting low, they just killed it and restarted it somewhere else, which meant they had to redo all the work. Today from what we're told, they are using Crewe to live migrate those low priority containers from one node to another node without losing any of the work. Examples they like to give is video re-encoding in the background, so if they need additional resolutions of video, then they do it in containers and those are the containers which they actually migrate today in production all the time from what we're told. Then there's Lexi, Lexi, Incos ecosystem. They also have support for Checkpoint Restore for a long time. Then Docker introduced it many years ago. I've been working since 2018 on the Potman integration of Checkpoint Restore. And one of the reasons I started to look at Potman is because Cryo is similar to Potman from some of the code. So I could experiment with Potman getting a Checkpoint Restore working there and then I brought it to Cryo today. And the reason to bring it to Cryo was to bring it to Kubernetes. And in Kubernetes, Checkpointing currently is available under the label Forensic Container Checkpointing. We made it, it was, so this took us a couple of years to get it into Kubernetes. It was moved to Alpha with one 25 couple of years ago, two years I think, three maybe. And just a couple of weeks ago, we were able to move it to Beta with one 30. And this means from now on, Checkpoint support is enabled by default. The, I say only Checkpointing is available because the Forensic Container Checkpointing story is basically you have a container running something unexpected happens. You don't want to stop it immediately, but you want to take a copy. So Forensic Container Checkpointing gives you the opportunity to take a stateful copy of your running container. The container will never know that it's checkpointed and later you can analyze the container in somewhere else or you can start it again in a sandbox environment. The Restore part currently in Kubernetes is kind of a, it's a bit of a trick. We trick Kubernetes into restoring a container. So we hook in cryo in the create in the start call. We detect that it's a checkpoint. And if a checkpoint is then we do a restore of the container. So from Kubernetes point of view, it's a starting of a container, but it actually is a restore of the container. And this is something I will show later in my demo. And now a couple of use cases just quickly going over why you would want to use checkpoint restoring combination with Kubernetes or containers in general. So the first one is quick start up kind of. It's you have a container, it takes a long time to run to start. You need to update your host because you need a new kernel. So you can take a checkpoint, reboot the host and the container will be restored much faster than it takes to initialize all the libraries. The multiple copies use case is similar to that. This is also people talk to us which they use in production. So you have an application. Yeah, you want to offer to customers. It takes a long time to start, maybe 10 minutes because it loads thousands of libraries. And so what they do, they take a checkpoint after the initialization and then they, whenever a customer requests the application, then they restore from the checkpoint and the customer has to wait just a couple of seconds instead of 10 to 15 minutes. Then the container migration is one of the most obvious use cases from my point of view. You want a container, how you have it on one host. You want to move it to another host without losing the state. Then yesterday we had a talk about checkpoint restore and spot instances. This can go away anytime. So what you want to do, if you get a signal that it goes away, you take a checkpoint and continue to run the container on some other host. All the examples at this point have been for stateful containers. If you have a stateless container, all this is not really helpful because you don't have a state you want to save. The forensic analysis use case is useful for all containers because even if it's stateless, maybe you want to analyze it without stopping the container and something which people talk to us a lot in the last year is AI training. So you have a container doing an AI model training with a GPU, you want to checkpoint it and run it somewhere else without losing all the work you've done so far. So let's go to my migration demo. So I have a, you can already see it here. I have a really simple pod specification. It's running two pods and I'm gonna, and this is about 500 kilometers away, this VM. So I'm gonna, it's not running. Let's start it. So this is of course a stateful application. It's a really simple application. It just has a state, nothing more. And I can talk to it just using curl. So now I have a counter. It says one, zero, one, two, no, two. And now three. Now I can create a checkpoint. The checkpoint is currently available only as a KubeNet only API and this command does everything I need. But it's basically just calling curl on one of the KubeNet APIs. And now the checkpoint is created and I have the name of the file in the variable. There it is. And now I can, we had a Google Summer of Code project using two great students who were helping us to write a tool to do the forensic analysis easily. And we have a tool called checkpoint control. And one example of what checkpoint control can do for example is the inspect command. And what you see here is now you see, basically it's the image it's based on when the runtime dates, when it was started, when it was checkpointed, the engine, the IP address, the size and so on. And to migrate this container to another host I have now to convert this image to an OCI image. Let me do that on the top. I'm using builder for that. So the first step should be to add the tar file to the new container image. Then we do, no. Then we add some annotation to it so that it can easily detect it as a checkpoint. Then we do a commit. And then I delete the intermediate image. And now I have to push the image to container registry. Let's push what there is. And I'm gonna call it 51. And now on another VM, so this is now local on my machine here. So previously I had a pod with two containers and now I'm migrating one of the containers into a pod with a single container. And again, really simple. I'm just replacing here the 50 with the 51. And now I will say apply. And now what will happen is that cryo in the background will download the container image. See it's a checkpoint image. And then do a restore. Kubernetes will believe it's a new container but it's an old container like we will see soon, hopefully. And now I will access the container and now it should say, what was the last? No, I don't see it anymore. So I think it should say four or five. I don't know, let's see. Five, four. So the container has been migrated from one host to another host over a large distance, maybe. And so with this I'm at the end of the introduction part about cryo checkpoint restore. And please rise in. So HPC applications are driven by scale and performance. And in order to process large amounts of data they need to be distributed or to be running on multiple different servers. The HPC community has been increasingly adopting Kubernetes mainly because of the features and benefits that it provides. However, the current checkpointing implementation is not able to checkpoint multiple containers at the same time. And the fundamental reason is because our cryo was not designed to checkpoint distributed applications. It was designed to checkpoint a single process tree. So the question is how can we extend cryo to support distributed applications and how can we enable dysfunctionality in Kubernetes? And there is a large amount of research done on distributed checkpointing and the checkpointing and rollback recovery protocols. This was then the fundamental concepts were created in the 1980s and 1990s when the resilience of computer systems was fairly low. And the most recent work, for example, that enables distributed checkpointing is DMTCP. DMTCP is a tool similar to cryo. It provides system level checkpoint restore. The way it works, it works in user space but the way it works is using the LD preload mechanism which means if you, it's not fully transparent. So to be able to checkpoint and restore application you need to be able to, need to start this application with DMTCP launcher. DMTCP is also very similar to cryo in a sense that has been integrated, for example, with container platforms such as AppTainer. However, the way, the main point is that it provides a coordination mechanism that allows to essentially synchronize the checkpointing between different instances. And it's also then used during restore. Another example is Apache Flink which provides very advanced full tolerance mechanism to provide uninterrupted execution for data analytics and data stream processing, batch of jobs. However, Apache Flink is a framework. It cannot be used with existing HPC applications. So to be able to create a consistent global checkpoint we first need to understand what does it mean to be a consistent checkpoint. And essentially this diagram illustrates on the access to time of how different containers are running and the red circles are checkpoints. And the checkpoint wave is essentially when we create checkpoints for all containers and when a message has been sent, a checkpoint is consistent when all arrows are messages being sent between different messages. And for checkpoint to be consistent it means that all arrows should start on the left side and finish on the right side of the checkpoint wave. For example, messages that are starting on the right side and finishing on the left are called orphan messages or we have recorded in the checkpoint the received event of the message but not the sent event. So this is the definition of inconsistent checkpoint. So most distributed checkpointing implementations use a mechanism called barriers. And these are essentially points in the checkpointing process when we have to synchronize the checkpoint across different instances. Crew doesn't provide this type of barriers but it has functionality called action scripts or essentially it has hooks or places during the checkpointing process and during the restore process where it can execute an external utility and waits until this external utility completes. And in this case to essentially enable synchronization we developed a tool called Crew Coordinator which essentially is used with this action script functionality to enable synchronization between different crew instances. So the Crew Coordinator tool has a client site and server site. It's using the client site to essentially register different instances with the server and essentially specify dependencies for which crew instance depends on which others and then use this to essentially pause the checkpoint and continue when all the dependencies have been satisfied. The restore process works in a similar way. Essentially during restore we execute Crew, Crew starts the coordinator and then synchronizes and waits for all other dependencies to get to this point. And when the dependencies have been satisfied it continues to the next step and this essentially allows us to provide distributed checkpointing without essentially needing to modify Crew or modify Kubernetes or essentially we can reuse the existing projects with this functionality. So this is a simple demo showing how the coordination mechanism can be used. So we first start to counter applications that are simply just outputting a number every second. Then there is a configuration file for the coordinator tool defined in the images director where the checkpoint will be created. And then we start the server on the right side and on the left side. And when the checkpoint is created the application will, Crew will start checkpointing but it will stop until the dependencies for the checkpoint are satisfied. And when the second accrues and starts then it connects to the server and this essentially triggers the synchronization mechanism to continue the checkpoint. So essentially both checkpoints complete at the same time. And similar during Crew Store the Restore process starts with the first application then with the second and essentially it makes sure that both the checkpointing and the Restore process both happen at the same time. So now that essentially we can synchronize the checkpoint when we have application running on different nodes we also want to transfer these checkpoints to a single location or a single server where we can essentially form a global checkpoint that can be used during Crew Store. And for this we essentially have, we're using this streaming mechanism in Crew that allows us to transfer the images from all nodes during checkpoint and then collect them on the destination site or where the server is running. And some future work is to essentially integrate this tool with the cryo and container D engines to essentially automate the dependency detection in Kubernetes or these are essentially identifying the which container depends on which other containers. This is also very similar to the way you would, for example, start a pod. Kubernetes is using a list of dependencies to, for example, start the initialization container before application containers. And the future work is also to enable integration with Kubernetes objects such as stable sets and deployments. I think that's it, right? Yeah, that's it, we're at the end. Any questions? I think you need to go to the mic or on mic which is coming to you, I don't know. There is one coming, yeah. So I will answer the basic checkpoint restore questions about pre-U and Kubernetes and he will answer the... First of all, thanks very much for the talk. So question one is in your demo when you did the checkpoint in the restoring, if I understood right, Kubernetes took, knew that it's a restore by the notational image. Again, please? How did Kubernetes knew that this is a restore? Okay, Kubernetes doesn't know, it's just the container engine which doesn't know it. So that was a notation? That's the annotation because yeah, so what we do, we parse the image to, so basically Kubernetes parses the image during create to cryo, cryo downloads the image and then we check if it has an annotation, if it has an annotation, then we go an extra code path to do the restore. So there is a simple if condition in the container in cryo. Yeah, it's really just if string exists basically. And the second one is like, I assume it is, do you know anything what's related being done in this space for the networking to resolve, let's say give the same pod AP address, service put on the same like networking stack? Yeah, good question. So for the podman migration work I did before, I actually, I'm restoring the IP address of the container to be the same so that TCP connections can stay alive. For Kubernetes, I never really looked into it because from my understanding, the network stack can be really complicated, but there's actually a Google Summer of Code project again this year in combination with P4. Yes, so P4 is a programming language for programming protocol independent packet processors. It's essentially language used for programming switches such as the phenol into the phenol and the EP4 organization participates in Google Summer of Code this year and we have actually a project that aims to solve this problem. Essentially how do we migrate containers and enable changing the IP address? But the solution is essentially using a load balancer. So you have all clients sending packets to the load balancer and then the load balancer keeps track of essentially the updated IP address. But I basically ignored it so far because the whole networking thing is too complicated for me to understand. I'm focusing on a very small part right now. So yeah, but that was, yeah, please. Hi, this is Abhishek from IBM Research. Oh, hi. Hi. So I have a question related to AI workloads. Oh, you have to speak up. I cannot hear you. Sorry. I have a question related to AI workloads. Does it also checkpoint the status of the GPU memory and other things? Okay, so the question is if we can checkpoint, basically can we checkpoint containers using the GPU, right? Something like this. Okay, yeah, so it depends. So we can checkpoint processes which are using AMD GPUs because AMD came to the CRIU project and they provided a plugin to support the GPU. So basically whenever you have external hardware which is not directly connected by the kernel which you go around like InfiniBand GPU, then there is an additional state in the GPU, in the InfiniBand card, which we cannot extract from Linux easily. So we need the plugin I mentioned. And from my point of view, this means that basically the vendor has to step up and write a plugin. AMD did it. There are videos around that. There's also NVIDIA doing it. I think Microsoft had a video with JetGPT where they claimed they're using CRIU and NVIDIA GPUs. There was a video from Membridge in combination with NVIDIA GPUs. So people are working on it but people are not talking to the upstream CRIU developers. So we are not aware of it just that we see videos on YouTube on it. So what is necessary to enable GPU checkpointing is essentially the ability to save the state of GPU and restore it back. AMD, the driver for AMD GPUs is open source. So they already implemented this and you can checkpoint restore GPU applications with AMD GPUs. NVIDIA also have a team working on this and they were supposed to have a conference a few days ago where they presented something that hasn't been released yet but it's a proof of concept demonstrating that you can checkpoint restore GPU applications with CRIU. There is also a startup called SEDANA that are also focusing on essentially solving this problem and yeah, there are multiple people working on essentially this. Hello, thank you for the presentation. That was great and interesting. Indeed, you mentioned like a couple of use cases like running some jobs, basically long running jobs that I could imagine be using that kind of feature whenever there is a node that is reclaimed because actually Kubernetes nodes are reclaimed regularly or are updated with versions. So it's like a very ephemeral environment. So I believe that's very interesting a solution for that kind of use cases but the demonstration that you made was really manual. Is there any plan to automate all of this? Actually make just asking, requesting the plan. Yeah, yeah. So my goal kind of is to have migration working and in the best case it's done by the scheduler. If there's a low priority container part it will be automatically moved anywhere else. But for the first part which you saw now this took at least four years. And it feels like this is 10% of the way. So yes, but it's a long way. Thanks for the presentation. Question about the HPC part. How do you determine at what point or in terms of time how do you determine whether all the parts or the containers is replaced where you can take a checkpoint? I think this is for you. Can you repeat the question yourself? Sorry. You talked about flink. For example in flink they send a message to say hey that is the point all the processes needs to take a checkpoint. Similar to that do you have something in the case of HPC when you do a checkpoint? Yeah. So essentially the question is how do we synchronize the checkpoint or how? For example flink is sending messages between different essentially to initiate a checkpoint. And with Cree we can't really do that because essentially Cree is not something that is continuously running on the server. It's something that we need to start. So essentially when we want to create a checkpoint we start all Cree instances and then just go back. No, I can't. But essentially we have an action script hook called freedom which essentially allows us to pause, create until all other instances are running. And this allows us to start the checkpoint at the same time. Did that answer your question? Hi, first congratulations on your work. One question, under distributed setting when you're doing to achieve a consistent cut how are you capturing the messages in transit? So are you using sort of a chandel and port algorithm or something like that? Yeah, I mean we don't really capture the messages. In our case we just freeze the whole process stream. So essentially when all Cree instances are synchronized at the freedom hook then in the next stage Cree uses the freezer C-group to essentially pause all processes. So in this case all applications will be paused. And then when the checkpoint is complete again it will synchronize the checkpoints and then it will resume them when they're all synchronized. Yeah, I think currently we don't have, and in this part there's no messages could get lost definitely. If we talk about TCP we'll do the retransmission hopefully if we talk about InfiniBand then we're out of luck. If you're looking at what MPI does basically there used to be the MPI, I know about OpenMPI there used to be the framework to do checkpoint restore there and what it did before was basically it acquires all the processes in the MPI calculation to don't do any transmissions anymore. But if you're talking about InfiniBand then your message will probably currently be lost. If we're talking about TCP then we hope that retransmission saves us. And this is your research work so it's not something that can be used in production yet. Maybe we'll introduce the two-phase commit where we can capture this type of issues. Yeah, thank you for the presentation. I was thinking of the use case where we run basically a job like for an AI workload or something like that and maybe during the execution of the job which has a high value, we don't want to lose it. Something silly happens to the node, I don't know this is full or memory corruption and maybe to kind of devise a way to restore the job on another node, like in which case do you think a node corruption would still be able to, we would still be able to do the checkpoint and restore and in which case it would not be possible. So, hard question. So, I guess the thing is as long as you can read memory and registers, it might work but if your storage subsystem doesn't work anymore then I don't know then you maybe have to do it over network. So it really depends on the things and again going back to HPC there are lots of papers about fall tolerance using checkpoint restore and there's a lot of research, what is the optimal checkpoint interval to regularly take checkpoints but you don't want to create too much IO for the IO subsystem so that it's only writing all the time and not doing any calculations. So this heavily depends on your error, heavily depends on the application so basically we cannot answer it but yeah, it depends. Thank you. There's one more question in the second row. In the Kubler API path, so you have for coordination some pre-dump action scripts when we use the Kubler API path is there a way to tell CRIU to take some actions pre-dump state because some of the applications require some preparation like deleting some files or just something to prepare the application for a checkpoint. Yeah, so basically I guess your question is can you tell CRIU to do something differently because CRIU has many options and you want to select one of the options during your Kubernetes checkpointing creation. Yeah, hooking up an action. Yeah, so when I initially brought this up and we had to introduce the new CRI call I had a couple of parameters there like TCP established and so on but they were all removed during review because the story currently is the forensic container checkpointing and so it's not needed. I think it would be nice to have it in the future. The question is do you want to expose it one to one to? So there is also a pull request currently for me open to do it on KubeCTL checkpoint. This also exists, it's not merged. So the question is do we really want to expose all parameters from CRIU in KubeCTL and then pass them all the way through? There are also the configuration files but the configuration files are not really useful if you're talking about a Kubernetes cluster because the configuration file has to be on the node CRIU is running. So I'm undecided but I totally understand that we have to have a way to have additional parameters passed from basically from KubeCTL down to the Kube to CRIU the lowest level. And actually there is a run C supports custom configuration file. So technically you can define a configuration file for every container that your checkpoint may understand. This allows you to essentially specify additional pre-options that are being used for checkpoint. Oh, okay, it looks like we're done. Thanks a lot for your questions, for your time. You just turn it down, okay.


If you choose to pursue the path of self-development or to gain your potential through this path, then that can take a lifetime. I want to find out what is the balance between trying to pursue my development versus doing something and finding that balance. I mean, how do I go about figuring that out for me? Was that clear? Yes, yes. Okay. I don't know. Now, when you talk about self-development, we are not talking about self-development here. Self cannot be developed. What you're referring to as self cannot be developed. It is not something that you can develop or you can put it down or anything. It's just that it's been… it's been lost. You have lost contact with it. So, getting in touch with it, in case it takes a lifetime, in case… not necessarily. In case it takes a lifetime, how much time do I give for that and how much time to do other things? What are the other things you do? Okay. So, the other things that you do in your life, will I have enough time for it? You shouldn't do any other thing.  Everything, everything that you do should be towards enhancing your own nature. Now, you're doing family, you're doing work, all these things you're doing, everything can be used as a spiritual process. It is not that in twenty-four hours time, this many hours spirituality, this many hours materialism, this many hours family, not like that. Every step, every breath, every moment of your existence can be used for your development of who you are. So, this is full time. And in that, what activity you wish to do, if you do whatever you want, what is the problem? Because what we are talking about is of inner nature. What… the other things you're talking about, external, isn't it? So, whatever you're doing, you're using everything as a process of growth, everything.  It's twenty-four hours yoga, only yoga, nothing else but yoga. Whatever you do, you make it into your yoga. Every single act in your life, you make it into a process of growth. Only then it'll work. Part-time spirituality doesn't work. Now, will this take me away from this or that? Not necessarily, not at all, because millions of people who have done the program are all people who are involved in variety of activity. Just a handful of people are full-time into it because some people need to do it full-time. Yes? Suppose I also pursued some other profession and you know, only in my vacation I thought inner engineering, you wouldn't be here today, isn't it? Yes? So a few people take it up full-time because they see that's their choice, they want to do that. And when you can go to the office and do software programming or management or something, why can't you also go out and teach something? It's not that you have to take it up. I'm saying for an individual person, what is the problem? Somebody chooses that. So, only thing is, only criteria that may be disturbing people is, they're not making money. No, no, I'll come. So that's not going to disturb you. There's no problem. You're doing what you want to do. You either choose to go to this job or that job or that job, isn't it? So, with Isha Yoga, if you come, instead of working five days a week, you'll end up working seven days a week. That's the great change that you'll have because you experience your whole life as a vacation. You learn to experience your work as vacation, then you're always on vacation. So, will this take me away from something? This doesn't take you away from anything. It enhances your life. But if you choose to get your… After some time, you'll retire from your job, isn't it? When you retire, it's a question of your choice, isn't it so? Somebody goes out fishing, somebody goes out meditating, somebody goes out dancing, somebody goes out drinking. These are individual choices they are making, isn't it? Right now, unfortunately, people like you… Yes, we're… See, because most of you, I'm telling you, you're only first or second generation of educated people in your… this thing. So, you're just thrilled about a few things that you have access to. If you are tenth, fifteenth generation of educated people in your family, you would have total disdain for education, doing well and all this nonsense. Yes? Because you're only first, second generation of having access, you think that's everything. But your children won't think so. You will see this.  Yes? So, what you choose is just individual choice, isn't it? Does doing this necessarily mean taking away? Not at all. People always ask me, Sadhguru, what is the mission? So I jokingly tell them, my mission is to plant undercover yogis. So today I can… I can very proudly say I have planted many people who are in peak of yogic experience but they're managing banks, they're managing corporations, they're… you know, they're housewives, they're all undercover yogis. Understand? They're not dressed like yogis, they don't look like yogis but they're all… experientially they are very much yogis but undercover. So, you want to be undercover yogi or overt? Undercover, okay. That's fine. That's my mission too.


Climate models produce data with inherent biases which can affect hydrological simulations if not properly accounted for. Statistical bias correction methods attempt to correct these biases but are limited in their effectiveness due to the timescale of the fluctuations they consider. A cascade bias correction method has been proposed which addresses this issue by considering multiple timescales simultaneously. This methodology has been shown to reduce the negative impacts of bias correction on longer timescales making it more suitable for use in hydrologic applications. This article was authored by J. O. Herter, S. Hageman, C. Mosley, and others.


Next question is from Salar seven. Does training fasted make you stronger in your workouts? I always feel much sharper and mentally clear which really translates to my lifts. Yeah You know what this highlights? This highlights the individual variance that you find with almost everything because studies will show that Eating a couple hours before your workout will result in better performance pretty consistently like they show pretty consistently Performance is better when you eat, you know a couple hours before and you have a protein carbohydrate rich Meal is what they'll say you'll have better performance However, there are cases like this where this person is like I feel better and sharper when I train fasted You know who else feels that way? I do I feel better working out faster than when I'm necessarily fed or whatever So this is a big individual variance thing. You got to kind of listen to your body Most people will do better being fed but that being said there are people that do better And I think you know sometimes there's a window to that So if this is a new concept like if you've just started kind of trying to be in a fastest state You're realizing you're actualizing the fact that you have more mental clarity and sharpness and You're sort of you know fueling that into your workout But at a certain point if you've been doing it long enough, which is what I've experienced if I'm not fed I don't really at a certain point like my energy dips and it drops if I if I don't have that to pull from so There's there's sort of a benefit in a detriment depending on You know obviously the individual variances to that But I do see like a mental sharpness and in terms of like being able to access the central nervous system You know that's been helpful But you know these and that the type of the workout matters to if I'm going for more endurance Or I'm like it's a really rigorous Type of a workout. I need to be fed. Yeah, so my personal experience with it is I'm the opposite of Sal. I've actually hate being fasted. I do not feel as strong. I don't feel as the energy I do notice mental clarity That's why I love to be fasted for like podcasting or if I'm riding or I'm doing something that I I need to be clear and sharp I do see lots of I see the cognitive benefits of doing that. I do not see the performance benefits Now that being said this last year or two there's been quite more often than not I'm actually training fasted and I have felt my body adapt to it and get better at it So what felt miserable before like when I first started playing with that I remember when we first started talking about intermittent fasting and you were actually talking about oh man I feel better actually feel more energetic and you're talking about all the benefits I was like, okay, let me play with it and I was like, no, dude. This is not for me. I feel way tired I don't feel as strong. I feel super weak. I halfway through the workout I want to quit early and I hated it and so I kind of went away from it But and not trying to train fasted it just started happening because we're training in the morning So a lot of times I come here on nothing, but maybe a cup of cup of coffee. So I'm not completely fasted But pretty much right. I'm going into these workouts with not a lot of calories to support And my buddies have gotten used to it that now I don't feel miserable why I do it I feel actually pretty good and okay, but I still notice if I have a day where I've had two or three meals And I have a later training session much stronger I I tend to have a longer better workout when I do that Yeah, I think you know too Justin you brought up a good point the type of workout It's gonna make a big difference. I think if you're gonna do an endurance workout That's gonna be long and rolling. You're probably gonna want to eat Yeah At some point you might hit a wall and that's not gonna feel very good but it is really interesting at the variance between People and how they feel and you really got to look if you feel better Working out first thing in the morning without having any food then do it if you feel better eating an hour or two before Then go ahead and do it. I really don't think there's a wrong answer here


This is stock number 9420. We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used SUV headquarters. Today we are checking out this super clean 2015 Toyota Sequoia Platinum Edition. This vehicle has the 5.7 liter V8 motor. From this HD video you will be able to tell that this Sequoia is in excellent condition. All the way around inside and out. It does have the front sonar factory fog lights. This one has the 20-inch Goodyear Eagle LS tires. These are 275-55R20s and they are on the 20-inch painted and polished aluminum alloy rims that are in really nice shape. No scuffs or scrapes. And these tires have a really good amount of tread left. I'd say probably close to 70% of the tread left on those tires. Blizzard white is the color. It does have some metal flake in it as you can see. Headlights are nice and clear. Front grills in pretty nice condition. No cracks or anything like that. I didn't see any dents on the hood. We shoot all of our videos in 1080p so if you have HD capabilities on your computer tablet or smartphone device turn them on right now because it is like you're right here looking at the vehicle with us. As we go down this side of the vehicle you can see just how clean the body is. How reflective and mirror like that paint is. We take these HD videos so if you are far away or even if you're close by and just can not make the trip down you can still see the vehicle, hear the vehicle and have confidence in what you're looking at before you even get here. Really clean down this side of the vehicle. It does have factory running boards. The back rims are in really nice condition as well. No major scuffs or scrapes or corrosion and the back tires have a lot of tread left on them as well. See the frame and underbody on this vehicle is in really nice condition and it does have the wheel liners which keep your underbody nice and clean as well. As we come around to the back you can see that the rear bumper is in pretty nice condition. No major cracks or dents on it. It does have backup sensors. Full towing package which includes a receiver hitch four pin and seven pin wiring. The tailgate's in pretty nice shape. I didn't see any dents on that. This is a power raised tailgate. Back storage area is nice and clean. It does have the power fold down seats so you can see that those go down like so and they also power come back up. This one does have the side window shades side antennas and the back storage area is nice and clean. To close this door you just press and hold that button for a second it'll beep at you once and then it'll come on down. You can really see that metal flake in that paint. A really good looking color on this Sequoia and as we go down this side of the vehicle just as clean as that passenger side. This rim is okay condition. Got a couple little scuffs but nothing too bad but just want to point that out for full disclosure. As we go inside the vehicle the platinum package gives you the brown leather interior. Both of these seats are heated and cooled. You have a memory driver's seat. No rips or tears on that seat. Carpeting's nice and clean underneath. Floor mat there. JBL synthesis sound system power windows power locks and those are your memory driver's seat buttons. You got all your controls right there. This one does have blind spot monitoring. That's for your rear gate, windshield washer fluid on the headlights and then you cannot reverse tilt your rear view mirrors. As we hop inside here you can see that this vehicle has 61,445 miles. Instrument cluster is nice and clean. It has a wood grain trimmed and leather wrapped steering wheel. That's in really nice condition. Bluetooth audio controls on there. Cruise controls on that side as well and then information center controls right there. This one has the factory navigation system with the JBL sound system. This is also where your backup camera shows up right there. You have your power slide down where your windows stability control. You have your different modes right there as far as sport and comfort and then you can also adjust the height of the vehicle as well with those buttons. Turn dial, four-wheel drive, get your information center controls right there as well. It has an outside temperature display. You got home link on that mirror. I don't know if you can see it there but yeah you get your home link buttons right there and your outside compass or your compass display. It does have the power sunroof. Passenger seat is very clean as well. No rips or tears on that. This vehicle smells very clean inside. Dual glove boxes and we'll take a quick look at the back seats and then we'll check out under the hood also power telescoping and tilt on the steering wheel. In the second row we have the quad seats. These seats do adjust like so. They slide nice and smooth. These seats back here are both heated as well. You have rear air controls back here as well as some headphone jacks and you do have a 120 volt 100 watt plug-in. This one does have a blu-ray DVD player. You can see the side window shade right there. Passenger seat is in really nice condition as well. Carpeting is nice and clean. These seats do fold down and up like so to get to the back and the back seats are nice and clean as well. No rips or tears back here. Very very clean and you can also control those seats with the buttons right there. We'll take a quick look at the under the hood. Under the hood we have the 5.7 liter V8 motor. Engine bay is very clean. Runs very smooth. This Toyota has been fully safety and inspected by our service shop has a fresh oil and filter change. All the fluids have been checked and topped off. Vehicle has been gone through mechanically 100% and is 100% ready to go. To see more pictures of this Toyota Sequoia or one of our other 400 new and used cars, trucks, SUVs, minivans, wranglers, halftones, recorder tons, one tons, you name it, we got it.  Videos of every single used vehicle that we have. All at summitauto.com. Thank you so much for checking out the video. If you want to make this one yours give us a call right now<PHONE_NUMBER>. Ask for one of our sales associates to make this Toyota Sequoia yours today. Once again that number is<PHONE_NUMBER>. Thanks again for checking out the video. If you want to check out more go to youtube.com slash summitauto. Remember to like, subscribe, and share on this video and all the videos you see there. In fact in a second you'll see a link to subscribe to our YouTube channel on your left. A link to more used inventory vehicles like this one on your right and if you haven't been to our website on the bottom link to this vehicle on our website. Click those check us out and we really look forward to help you with this super clean 2015 Toyota Sequoia Platinum Edition. Thank you so much.


Hey, how's it going? I'm Marc. There are many English books, however, none of them may be helpful. Jennifer wants a big wedding. Meanwhile, her fiancé wants a small private one. Learning how to use an iPhone is difficult for my mother. Buying it for her birthday was a mistake indeed. It was only a small gift, but it was gratefully received nonetheless. In this lesson, we're going to take a look at conjunctive adverbs. So stay tuned! Right, conjunctive adverbs. Let me tell you that I've already talked about conjunctions. If you want to watch all those lessons, you may click here. So let's start by describing adverbs. Adverbs are words that modify verbs, classes, sentences, adjectives, or even other adverbs. If you want to watch a lesson on adverbs, you may click here. Conjunctive adverbs are adverbs that join two classes together. These classes are independent ones. We can mention accordingly, additionally, besides, instead, likewise, however, rather, still, etc. I'm going to post a graph listing all the conjunctive adverbs on my community section so you can download it. When we talk about conjunctive adverbs, it's important to say that punctuation is extremely important. A semicolon or period, what is a semicolon, is this sign, this one. Or period, period, is used at the end of the first independent clause, at the end of the first independent clause. Then follows a conjunctive adverb, followed by a comma. So the formula would look like first clause, semicolon, or period, plus conjunctive adverb, plus comma, plus second clause. So let's list some examples. First one, following a semicolon. There are many English books, semicolon, however, which means by contrast, none of them may be helpful. So you can see the first clause, right? There are many English books. Then we need the conjunctive adverb, however, which means by contrast, and then the second clause. None of them may be helpful. Let's look at another example. You need to stop watching TV that much. Semicolon, otherwise, which means differently, you'll become blind. This is also an example of, yes, the first conditional. So you see, you need to stop watching TV that much. This is the first clause, and so the condition. Otherwise, you'll become blind, and this is the result. If you want to watch lessons on the first conditional, you may click here. So let's look at yet another example. I'd like to invite you up for dinner, semicolon, unless, which means except, you had other plans already. This is also an example of the second conditional. If you want to watch a lesson on the second conditional, you may click here. However, this is, yes, the condition. I'd like to invite you up for dinner, and then the result, unless you had other plans already. So I'd like, I would like, and then had, past simple. For our next group, let's list the ones that follow a period. Jennifer wants a big wedding, period. Meanwhile, which means at the same time, her fiancé wants a small private one. So you can see Jennifer wants a big wedding is the first clause. Then we have the conjunctive adverb, meanwhile. Then it follows a comma. Her fiancé wants a small private one, the second clause. Let's look at another example. I've decided to quit my job. I've decided to quit my job, period. Furthermore, which means in addition, comma, I have exhausted all the options. So you can see I've decided to quit my job, which is the first clause, and then we have a period. Then we have furthermore, which is a conjunctive adverb, comma, and then we have the second clause. And for the last example, my son was supposed to go out with John last night, period. Instead, the conjunctive adverb, which means as an alternative, comma, he chose to stay in the second clause. Now I'm going to erase this and we're going to look at yet other two uses of conjunctive adverbs. So let's look at other uses. If the break of the main clause is weak, then do not use commas. What do I mean by this? So let's look at my introductory example. Learning how to use an iPhone is difficult for my mother. Buying it for her birthday was a mistake indeed. So you can see that here indeed, which means certainly, doesn't have a comma before it. Why? Because the main clause is weak. So we need this adverb or this conjunctive adverb. So buying it for her birthday was a mistake indeed, certainly. So it reinforces buying it for her birthday, so the main clause. Let's look at yet another example. Driving to work during a blizzard is hard. I will therefore, without a comma, I will therefore, which means consequently, call a taxi tomorrow. So therefore, which is the adverb here, is needed in the sentence. Without it, it would be weak the sentence, so we need to reinforce. I will therefore, consequently, call a taxi tomorrow. So the last point that I want to talk about is that conjunctive adverbs are also used at the end of a clause. Don't forget to use a comma before it. So my introductory example was, it was only a small gift, but it was gratefully received nonetheless. So here nonetheless, which means however, is preceded by a comma. So we have a comma right there. So let's look at the last example of this lesson. After playing chess with his friends, Sean was too proud to admit defeat. He did admit he wasn't concentrated, however. So however means yet. It's the same thing. And it's preceded by a comma. So we can even use nonetheless, like here, or there for, or indeed, is a conjunctive adverb. But remember to use a comma before it, okay, if you want to use it at the end of the sentence. I will cite a quotation by Henry Rowlands just to give you another example. So that's it for today. If you have any questions, comments, or if you want to post your own example, you may do so under this video. If you haven't subscribed to my channel, I suggest you do. If you want to share the lesson, please do. I will really appreciate it. Having said that, I will see you next week with a new English lesson. Take care. Bye bye.


So we are really pleased to have Dr. Pamela Moreno here today. And Dr. Moreno is the former chief of biochemistry and bio related chemistry at the National Institute of General Medical Sciences. And Pamela joined NIGMS in 1994 and played some very critical roles in raising the profile of the field of glycoscience and bringing it into the mainstream. Dr. Moreno was a key leader in the construction for functional genomics and later was instrumental in helping to establish the NIH glycoscience common fund. Today she is going to tell us a bit about the history of the NASA glycosciences report and the subsequent activities at NIH around glycosciences and how they were impacted by the NASA report. And I from everything I've heard there, we're going to be able to learn a lot from Dr. Moreno's experience so please go ahead and thank you for being here. Thank you. Can you see my slides? Yes. Great. I can't see you when I'm in presentation mode. That's why I have to ask. So I want to thank Steve for inviting me and I wasn't really sure what I could bring to the RNA community when I first heard about this. I remember Tosi recently discovered that RNA is decorated with glycans and there's considerable effort going on now to determine what they do. So that was my first thought is, oh, another area for glycobiology but then I realized after talking to Steve, you're really interested more in process. I asked him for some specific questions and these are the ones that he gave me. So I'm going to try to address these very brief with some very brief slides. I'm going to really rip through these so hang on to your hats. And then I'll take questions from you folks. Hopefully, when I give this presentation, most of these questions will be answered. But I think it's important to give you some time to ask me questions. So the role of a federal program staff, program officer. Oh, okay. Yeah, I was just making sure your slide for advancing. Can you see that? Okay, so the role of a federal program officer and the tools that are available them to move their portfolios are pretty unique. And I'll go over those a little bit. But paramount to this is the use of outside opinions to identify research opportunities and then it's incumbent on the program officer to find mechanisms to address these opportunities. In my case, it was glycomics and glycoscience. So program directors have they handle portfolios of grants they advise on the priorities for committing federal funds they act as advocates to scientific areas. They advise on the development of emerging areas of research initiatives operating procedures and policies. And I think it's important they serve as a fiduciary for federal funds, which always has to be taken into consideration. In 1995, my portfolio is. Can you hear us Pam. Yes, I do. I don't see your slides advancing. Are you. You don't see me going forward. Yeah, can you try moving them forward. Okay. If you'd like to share your slides on your own behalf, can you try sharing the screen again? Yeah, I'm sharing my entire screen. In Zoom or elsewhere. You're muted. I did it from from share screen. Did it not work. We're still on the first one. Can you see if you can go forward? Sure. The left corner to go to go forward. Is that changing them? Can you share your screen again? You want me to share a screen again. It's not working. Do you want to share the slides for me? I emailed them to you. Sorry about that. I don't have a Zoom, so I'm working off of. Your web connection. There you go. Okay. Can I advance those or do you need to advance them? I can give you a patrol to. I didn't have some on your behalf. You'd like to start. Yeah. Okay. Sorry to break your flow again, but you can. Yeah. Okay. So this is the information that was requested. How do I forward this? There we go. There are tools available. Next slide. Next slide. So glycans are the third information rich molecule. Polymer in the, in the human body. After nucleic acids and proteins, but we don't have, or we didn't have good tools to study these. To synthesize them to understand their structures. And so it was incumbent on our portfolio to actually try to make an attempt. To provide those tools. Next slide. In 95, when I, I embarked on this, both NIH and NSF had interest in the field. NSF had a database called car bank that was. Ending. And both their bio and chemistry directorates were involved. GM had my portfolio. NCRR had two centers and several other institutes were supporting some glycoscience awards, but didn't have portfolios in this area. Next slide. So the charge that was given to me was to expand the glyco conjugate portfolio. And going about this as a program officer, I could raise awareness through meetings and workshops. I could seek partnerships and leverage resources. That meant partnering with staff. We shared interest from other institutes and agencies. And I could utilize opportunities to initiate or augment existing initiatives or try to develop new initiatives. Those would be through funding opportunity announcements. And over the course of the last couple of years, things like the Eureka program, the Aero program, through supplements, through small business, and through intramural NIH itself. Next slide. So in 98, Dr. Zuhuni took over NIH and the NIH budget was doubling by NIGMSC, and the NIH budget was doubling. By NIGMSC standards, large team science was not something we embraced, but because of the doubling, it was something that we had to embrace. And so we developed a program called the glue grant initiative. These were $5 million awards, 10 years, and to solve a major initiative in research. Also in 99, the first high throughput tools for studying glycans came about. These were glycan arrays. And at that point in 99, if someone scanned all of the glycan libraries that were commercially available, you could come up with less than 100 purified glycans to put on an array. 100 glycans to study an infinite field of structures is not very good. By February of 99, at a Gordon research conference, I made a presentation to them on large collaborative grant mechanisms, the glue grant program, and suggested that this might be a great opportunity for the glyco community to coalesce and come in with a large collaborative grant to study glycans and address the issues to bring the field into the glycomics era. The community caucus, they formed a steering committee and they decided to go after this grant. Next slide. About the same time NSF was also very interested in glycans held a meeting and Laura Kiesling and Jerry Hart chaired that meeting to explore the future of lycoscience and the role of biology and chemistry in this area. And the meeting report that they developed called for the establishment of infrastructure, including chemical and biological tools, databases, a workforce and integration with the broader scientific community. Eva Barak brought this report to me at NIH and given the glue grant program and the opportunities to address these issues. This report was part of the basis for convincing our council that the glue grant program and glycoscience should go forward. Next slide. So where did we start in genomics and proteomics? You have structures, sequencing, synthesis, tools, knockout models, databases and a workforce. In glycomics, we had a few hundred structures. Mass spec was just being utilized to start sequencing. Chemical synthesis was in its infancy. There was chemical, chemo, biological and bio fermentation, but they were hampered by methodologies and lack of enzymes. There were a few tools, maybe a couple of dozen enzymes that had been synthesized by different groups, a few probes, one high throughput assay, a few knockout models, database was going under and no workforce because there was no training grants. So we had a big mountain decline. Next slide. The CFG was an experiment. In large team science, it started out with about 30 people. It ended up with over 600 labs internationally and became a model for a large team science. It's been written up and it became one of the successful programs that underpinned the NIH roadmap that Dr. Zuhuni developed and which evolved into the Common Fund program. So they were going after structures. They were developing methods for sequencing, for synthesis. They focused on chemo enzymatic. They developed new tools, knockout models. They developed a huge database that imported data through an electronic notebook that gathered information from all their cores. And the expectation was that people would be trained when they were working in this area. Because glycoscience is unique and the cores, they developed cores. Could have the next slide, please. Several iterations of the program got written up and eventually they settled on having a steering committee and then utilizing cores that were going to do all the work. And people would just give them their samples or they would provide resources and compounds and libraries to people. But everything would be free. All of the data would go into an electronic notebook and be searchable through the consortium databases. So this set up a model for doing large team science that was open to the broader scientific community and obviously worked since it attracted over 600 labs over the course of 10 years. And they found ways to keep this going. So it's still an active consortium out of Harvard. Next slide, please. Next slide. So the databases and partnerships that came out of this, obviously over 10 years, you know, you plan something the first year by five years, you renew it, but, you know, things change, science progresses, and this science progressed very quickly. So we knew we had underfunded the informatics efforts. And so we held a workshop. We brought in people and experts. We talked about this and it resulted in administrative supplements. These were a $1 million administrative supplements to the informatics efforts in each of the glue grants to try to get their informatics up and running very quickly. Japan, the CFG, NCI, NCRR, Dental, DK, the proteomics interest group at NIH, and Hart Lung all came together and held the frontiers and glycomics, bioinformatics and biomarkers disease workshop, which led to a white paper. And that ultimately led to the alliance of glycobiologists for detection of cancer and cancer risk, a U01 program that's still ongoing at NCI, and to the programs for excellence in glycoscience, which is a P01 program at Hart Lung. So you can see this theme is developing here. You set something up. You bring in outside experts. You look at the field.  How does this advance the field? You bring this back. You synthesize it into program efforts. Distill it down and then work with your institutes to develop programs that address the needs to move the science forward. And of course, these are always competing against other interests from other portfolios. So you need strong information from your outside experts to convince your institute that they should invest in a given program. Next slide. The Naocot mouse project was ongoing. GM was given a million dollars to invest in that. And I convinced our director that the 22 glyco-relevant strains on the list of available mice to purchase would be a great investment for the glyco field. And we ended up getting all 22 strains. Subsequently, we partnered with the COMP program and completed all the mouse efforts within the CFG through COMP, which saved us about $300,000 a year. And we were able to move that money into other efforts. Next slide. Glycan synthesis libraries and standards was a sticking point. If you don't have the compounds and the compound libraries to do the screens or make the tools or understand the competitions, you can't move the science forward. So GM put together a meeting expanding the chemical space for carbohydrates, recommings, who is at Emory at that point and is now at Harvard, shared that meeting. And that resulted in GM's council giving us money to run an RFA for chemical synthesis. Child health also had interest and they ran an FOA for the synthesis of human milk, oligosaccharides, which is now a very active area for probiotics and commercialization. The RFA was really successful. We developed a number of methodologies that had the potential to move forward into automation and to benefit the larger community from those efforts. Carolyn Bertosi ran a meeting called expanding the chemical space for carbohydrates, roadmap to automated synthesis for NIGMS and GM's council approved a contract called for development of glycan libraries that funded about 15 companies to develop libraries and put them out commercially. So now we went from 99 compounds back in 2000 to thousands of compounds by 2011. Next slide. High throughput tools were gaining real traction and the CDC was using these to screen for the bird flu. Other arrays were starting to be developed. Jillian Air ran a workshop for us on, for us in AI on the information that was coming off of all of these different glycan arrays that were being developed and how we would interpret these. And this resulted in the CFG, putting together bridge projects for cross-platform comparison of these diverse arrays and era supplements for development of new novel arrays and their characterization in YGEM presentation. Next slide. The American Recovery and Reinvestment Act was a unique opportunity for us to put in challenge grants for the functions of glycan binding proteins since the structure and function of glycan proteins since the structure and function of glycans. And we were able to make eight awards through that act and beef up the infrastructure. Next slide. Glycoenzymes, especially enzymes for synthesis of carbohydrate compounds since chemo enzymatic synthetic methodologies were becoming very popular. Josh LeBear had the libraries and Kelly Moorman and Don Jarvis had expression systems for them and these three got a large era supplement to be able to put this all together into a repository so people could order the expression vectors. Next slide. By 2011, the profile for carbohydrates and glycomics was becoming more high profile. Certainly many more institutes and agencies were interested and a number of workshops actually a flurry of workshops between 2010 and 2012 took place on the NIH campus and these in turn led to both RFAs and to interest by multiple agencies into doing a larger study of the field to see what needed to happen if we were going to take advantage of glycomics and glycomic efforts. Was this a place where we really needed to make a sincere investment? Next slide. So to sort of build interest across agencies and to coalesce what we wanted out of a report was that NIH issued U54s for resources for NIH and pathogen arrays and NIH's glycobiology SIG which is an intramural effort of laboratories put together a seminar series a course in glycobiology in an annual meeting to raise interest on campus and the interagency glycoscience steering community that I chaired brought together NIH, FDA, NIST, NSF and DOE to meet on our regular basis to discuss our shared interests and where we could partner and one of those partnerships was between NIST and GM for developing glycan standards and mass spec libraries. This all raised interest between multiple federal agencies and institutes and at this point GM went to OD and suggested that we commission a National Academy report in this area to look to see what needed to be done to move this field forward and several agencies stepped up DOE, FDA, NIH, multiple agencies within NIH and put together the funding for assessing the importance and impact of glycomics and glycoscience. The report came out in 2012 and it was transforming glycoscience a roadmap for the future. So this report has been used both by program officers at NIH and the multiple institutes to support funding for this field and to address the roadmap that was put together by the group, by Dr. Walt and his group. But it's important to note that I've shown you that these reports of various types get used by program officers and staff across the federal agencies to understand their portfolios and to help develop funding opportunities to move the field forward. But we expect that these reports are going to be nonbiased that they're going to be honest reports that show us where the opportunities are where the deficiencies are and to lay out what needs to be done if we're going to take advantage of the recommendations that people that are writing these reports are giving us. So whatever you can do to tailor your report to where are the opportunities and who's going to benefit from those things and what do we need to do in terms of a timeline to actually bring these things to fruition in a timely manner. Those are the type of things as a program officer I would look for in a report to justify making an FOA pitch to my institute for money. Next slide. The committee was held put together a great report. It was chaired by David Walt and we brought David back actually later in the program to do an evaluation of the chemistry portion of the common fund. So I'll talk about that a little later but remember his name. Next slide. This report provided a roadmap and I would refer you to chapter six of that report. It laid out a plan and it gave us a timeline to be accomplished and what could be accomplished if we made the proper investments within a timely manner for synthesis, analysis, modeling, enzymes, informatics, and education. So it was incumbent on program officers across the institutes and agencies to take these requests and make these recommendations and goals timelines. Next slide. For glycoscience, lots of people back into this field. Actually, everybody in the field has backed into it because their own avenues of research basically led them into protein, carbohydrate, and interactions of some sort. And then they were stuck. They were tied up in a traffic jam because they didn't have tools or expertise to move the science forward and really important science would just get dropped because they couldn't find a collaborator to move that area forward and answer the questions that came up. And then they could help them get around these traffic jams. Next slide. So as a program officer, I had already done what I could do out of a single institute or with my other colleagues across institutes and agencies. We needed something else that we were going to be able to use to push this field into the broader scientific community. We built up infrastructure over the years, but we really needed to get glycoscience embraced by the broader scientific community. And to do that, we needed a large concerted investment that could go across institutes and could address the deficiencies in the infrastructure. The NIH Common Fund, which came out of the NIH roadmap, actually the CFG was one of the models that was used for this. And we were able to get across the finish line and that effort was focused essentially on integrating the field with the broader scientific community. And that meant making simple tools that people could use, kits and reagents. Things of that nature that somebody that was not a glyco expert could actually use to answer their problems and get out of the traffic jam. So the Common Fund was established in 2006 out of the 2004 roadmap. It's been used to support a series of short-term, exceptionally high-impact trans-NIH programs, and certainly glycoics was obviously trans-NIH. And it was intended to change the way science is conducted by delivering data tools and technologies for broad use, which is exactly what we wanted to do. To be competitive for Common Fund for funding, you need to be transformative, unique, synergistic, cross-cutting, and catalytic. And we made a pitch that we fit all of those categories. And next slide. Luckily, we made it into the Common Fund. There are a number of hurdles that you have to go through to get into the Common Fund, but we made them through. And Dr. Lorsch was the chair, was led out of GM leadership team that were involved in this program. Officers that represented them are here. Next slide. We developed this Common Fund effort in three stages. There were three arms. We wanted to automate synthesis and provide facile methods for large library synthesis. We wanted to have tools that were familiar and easy to use and commercially available for the study of glycans, and we needed integration and analysis tools and a database. So these three became the three arms of the program. I headed the synthesis arm. Carl Kruger headed the tools arm. Amanda Malillo and Doug Shealy headed the database arm. Next slide. The Common Fund had a deficiency in that people made these huge programs, but then they fell off the books because you couldn't renew them and people struggled to keep them going. We didn't want to fall into that. We wanted a short program that made these tools and technologies and we wanted them translated into commercialization so that they continue to be developed and upgraded, replenished and advertised to the broader scientific community. So we put a SBIR STTR effort together through multiple institutes since the Common Fund doesn't use this mechanism and mirrored the Common Fund efforts with SBIR STTR funding opportunities. These were extremely successful. We got large numbers of applications every year. We put in about $3.6 million a year over the seven years to get these applications funded and to get all of the tools and technologies that were coming out of the Common Fund translated into companies. Next slide. The synthesis arms solicited applications. We were interested in targets that were justified on the relevance to human health and we were really interested in facilitating automation. Next slide. We funded 18 applications. We were able to put large libraries of natural compounds on the market when we developed new chemical and chemo enzymatic methods for facile synthesis so that you could go into a university chemistry department and hand them a paper and they didn't have to be a carbohydrate expert to make you the compound that you wanted and we were very interested in transitioning these to automation and we came up with I think four different platforms that are now available and commercially available to be able to synthesize carbohydrates so if you put a DNA synthesizer you could also put in a carbohydrate synthesizer into your cores at your universities. These are still being developed but for the most part the field is a quantum leap from where it was in 1999. Next slide. Chemo enzymatic synthesis is really useful. Anybody can do it if they have the right enzymes Aero allowed us to make these expression libraries and the Common Fund allowed us to push this forward, get all of these vectors expressed into enzymes and commercially available through glyco-expression technologies and several other companies that have stepped forward because there is real interest in these enzymes commercially. Next slide. The tools we put out initially are R21s for high risk high impact awards and a few U1s that were obvious things that could be done immediately and the R21s were then followed on with that opportunities to apply for U1s and we got a huge number of really unique tools out of this that are easy to use. Next slide. We've got reagents and tools that fall within these categories and most of these are being commercialized now or already commercialized. Next slide. The informatics effort was the last effort to go on the books. It's evolving at this point. It's an international effort and it fits in with several other efforts from Japan and Europe and Australia and Canada and these tools are all actually freely available on the web through this website so you can go from your protein to your glycoprotein and look at the structures and functions as they exist in the data. Next slide. These data analysis tools include standards for metadata, standards for representation, common ontologies, assignment of structures and simultaneous mining. The PDB has been cleaned up through an effort through the common fund and so if you go in there, if you see carbohydrates attached to your protein they're actually in the right linkage and appropriate structure. Next slide. The common fund has completed its work. I'm actually sitting in on the evaluation of the program tomorrow out of OD but as you can see it was built upon the National Academy report. I think the only thing that the common fund couldn't address since it doesn't have educational efforts within the fund was the training and that's been picked up by GM and Hart Lund who have put training grants on the books for glycoscience. Next slide. We've developed catalytic and chemo enzymatic methods. We have automation platforms. We have analysis labeling and modeling tools with demonstrated proof of concepts most have been commercialized. We have uniform informatics efforts and we're moving forward to integrate glycomics with other molecular databases and we have the commercially available glycoenzymes. So this program to my mind has fulfilled the National Academy of Science report and one of the things we did was to hold large workshops and meetings symposia for the chemistry division and we brought David Walt in as one of the panelists to evaluate the work that was done. This was a two day symposia. It's available online through the NIH video website and David was impressed that we had moved and kept through the timeline that the academy had set out for us. Within the seven years we had definitely checked off all the boxes and we're well on the way to the 15 year goals that they set out for the chemical synthesis of carbohydrates. For the other efforts we likewise had numerous presentations in final symposia which were well attended and highly viewed. All of those are also available on the NIH video cast website under archives. Next slide please. So we started out in 2000 with NSF laying out a plan and we established the first three criteria through about 20 years of RFAs, FOAs, common consortium for functions like comics and other programs that were held through RO1 portfolios and the common fund made the effort that was necessary for integration of this with the broader scientific community. The preliminary data that I've seen from the evaluation suggests that we've achieved that as well. Next slide. The 2012 report provided the roadmap, provided the timeline, pointed us in the right direction, placed the emphasis on synthesis, analysis, modeling, enzymes, informatics and education and disinformed the efforts of myself and numerous other program staff at NIH and other federal agencies. In terms of how to guide our portfolios and how to present to our bosses to develop the necessary funding to bring these across the finish line. Next slide. That's the consortium. That's the common fund team. A lot of those folks were in the consortium for functional like comics. We met every year and we reported out from all the different groups in person and each of these three teams met regularly, monthly actually by video to work together to develop the program and to move it forward. I'd like to congratulate them. At this point I'll take any questions. Thank you. We have a question already from Susan. Hi Pamela, thanks very much. Can you hear me okay? Yes, I can. I had a question about the SBIR STTR decision for this. Right. My understanding is that you need an actual company to get an SBIR and an STTR. No you don't. Actually the company just has to be on paper and so as long as you've contracted with your university as long as you have a piece of paper from the university that says they'll wrench you bench space in your laboratory for your company at a given rate per square foot or whatever then you don't actually have to initiate the company until you have the grant in hand. So just to understand that number one the synthesis seems to have gone more to academic labs or that's not true? That's not true. So academic labs basically push the synthesis forward and several of them have developed their own platforms now for automation which was supported by this common fund effort but they've also spun off companies from their laboratories because training is one thing making compounds and maintaining machines and pushing that forward is another. So it's much easier to get a large grant out of the SBIR program to set up your laboratory as a company and use technicians to continue to make the enzymes continue to make the compounds that people are requesting from you since you published on them. So that's basically how that's moving forward but there are several other commercial companies that have come in and partnered with them so they've essentially taken their IP and licensed it to move that into commercial platforms. A lot of our chemists developed tracking reagents tracking reagents, things of that nature and that IP was purchased by companies that are making these products and selling them commercially. The tools themselves some of the people that developed the tools have taken out SBIRs or the sister program, STTR where you find a company and you get a technology transfer grant part of the money goes to you to develop it and hand it off to the company and then the company further develops it and commercializes it. So there's two flavors to the business aspect of those. All of federal agencies use SBIR STTR and the people in our program the Common Fund program seemed very adept at being able to find partners or to work SBIRs into their own laboratories. So can I ask this is Brenda Bass, can I ask a nuts and bolts question? In your head, there's always a they and it goes back to another they and I'm trying to understand how this all gets put into motion. So I think there was a consortium I don't know how that started and then the consortium I think was involved in getting the Common Fund initiative is or the program are you talking about the consortium for functional glycomics that that was a glue grant and that arose out of my initiation of conversations with the glycoscience community at the meeting continual dialogue with the glycoscience community to develop an application to NIH to take part in the glue grant program and then several iterations of their grant went back and forth and eventually moved into a model where they were going to put together cores and an administrative group that would handle requests for the reagents and services of the cores. All the data would be public would be in a searchable database so that turned out to be a really exceptional program and it continues today at Harvard but that's not the responsibility of it's not the responsibility of a consortium of investigators to move a field so that the program directors like myself would be the people that look to all of these information that's coming in and look as portfolio managers we have tools to look at our portfolios to look at deficiencies to look at new areas where we would want to invest just like any portfolio manager and in that regard we use the scientific community as a sounding board so I showed you a number of instances where I myself or other members from other program officers from other institutes were involved in holding a series of workshops and symposia and meetings where we could bring in information from people that we invited to tell us what the needs of the community were and what the opportunities were from that community if we could meet those needs so they all develop white papers those white papers we as program officers would sit and digest and then we would decide if those things look to be necessary to move our portfolios you know could we put together an FOA or could we partner with other institutes to put together an FOA to move things forward that went on for almost 20 years to develop the infrastructure for this field and move it forward when it was essentially the point where glycoscience was really becoming established and people were in need of moving this into the broader scientific community that's where we as program officers myself and my colleagues from other institutes that were interested in glycoscience work together and made a pitch based on the national academy report that we we commissioned to NIH's office the director because common funds seemed like the logical place to utilize their funding to move this into the broader scientific community that was common funds mission that's what we needed to accomplish so we did the hand glove thing and melded with the common funds so we made our pitch to the common funds to become part of the common fund they rejected us we made our pitch again they rejected us we made our pitch again and finally got over the hurdles and we were able to move into the common fund and obtain the funding that we needed to be able to go out and solicit applications to make this field simpler to get things into kits to commercialize a lot of materials so that we didn't have that traffic jam you know all of us as program directors were getting constant calls from the broader scientific community saying you know I backed into this and oh I found out that this carbohydrate is involved in turning on this receptor and how do I study it and so you try to find them a collaborator but that's a slow process and collaborations don't always meld so we really wanted them just to be able to have the same access that people in molecular biology have you know that you could just go to a company and buy a reagent or buy a kit and that was the impetus to get us through the common fund and into the funding funding area so but all of those steps are driven by program directors who utilize the information they get from reports like the one you're writing. So if you're writing a national academy report you're not doing this de novo I mean somebody commissions you to write this and they obviously have interest in this field and hopes for utilizing your report to move something forward so whoever commissions you to put this report together you know take advantage of their knowledge of the field and what their interests are but I can tell you from glycomics perspective chapter six where they laid out this is the advantages that this field is going to provide us if we move it forward and these are the things that are necessary and here's a reasonable timeline for accomplishing X, Y, and Z to be able to get this accomplished so they gave us a 7, 10 and 15 year timeline under the categories that I laid out and you know that makes it easy for program officers to check off the boxes and write FOAs and push this forward we certainly utilize the academy report in terms of the synthesis and tools and databases to say these are things that are necessary but now we need to make them broadly available and accessible to the broader scientific community we need to keep them simple and easy to use and so that you don't need to be a glyco expert to be able to utilize them and that's basically what the common fund was about was making this accessible to the broader scientific community Thank you that was really helpful are there other questions at the table right now what was that to you can I ask one question one more question so is there anything that really laid out what we need to put in the consensus report that is most helpful to program directors, program officers is there anything you would say was unhelpful is there anything we should not put in the report don't recommend funding mechanisms and remember that this isn't just helpful to program officers it's helpful to applicants for funding so I can't tell you how many grants I've read where they cite this report and as justification for whatever it is they're pitching to NIH be it synthetic methods or commercialization of given products they use the report they use what the report suggests is important to the field and necessary for moving the field forward and why is that important because the report lays out really clearly what the benefits to human health and DOE and FDA are if these things are accomplished if we can do X, Y, or Z if we can have these tools available then this is going to move A, B, and C farther along and save us a lot of money I mean the folks in the Parkinson's field are really, really lovely news piece about having all of these glycol glycol products, glycolipids available where they weren't before and how this was going to open up all new avenues for research in that field this is the type of thing that program officers jump on and that's what we want to see okay thank you so much we have to move on but that was so helpful we really, really appreciate your time today sure and I would suggest the Academy report there's a PDF it's freely available you can download it take a look at the intro and the final chapter chapter 6 where they lay out this roadmap and they put together a beautiful pamphlet it's like a two pager people can't walk around it's like a heavy report but a two page pamphlet we took this around we showed it to people we pedaled this and quite honestly Europe and Japan both jumped on this and developed their own reports on glycoscience for the Europeans and for the Japanese and I ended up having to present with both of them on what their nations had developed so you can have really international echoes of what comes out of these reports yes thank you so much and I did just put the glycoscience report on our google drive for everybody thank you again good luck take care steve thanks camp okay