Hey there, Site 42 staff, Dr. Theron Sherman here, and today we'll be reacting to episode 1 of There Is No Antimemetics Division, a miniseries created by and directed by Andrea Joshua Aznicar, and hosted on his YouTube channel of the same name. Now since I'm going to be reacting and commenting and pausing over this whole thing, make sure to head to his YouTube channel and watch the episode proper there first. And don't forget to subscribe and leave a comment saying that Dr. Sherman sent you. Don't worry, I'll wait. Preaching Sector 7G, eh, that's nothing to worry about. Good! You're back! Excellent work, Agent. Now, let's get on to the video. The SCP Foundation is a top-secret organization tasked with the containment of supernatural objects. Within the Foundation Apartments, the Antimemetics Division deals with anti-memes, which are essentially information black holes preventing any knowledge about them from being retained or communicated, although someone at the Foundation would argue that. There is no Antimemetics Division. Dun dun. Can I smoke? Oh, you...  I've asked you that before, haven't I? Twice. In the last 15 minutes. Ah, we're forgetting things early, all right. It can come through now. Marion Wheeler. All right, that's where we're starting this story, excellent. Just like we like our site directors, incredibly handsome. So what's this about? Oh, I got some meeting invitation. No agenda or anything. I mean, I know if I've said it's jump, you jump, but... Most of our meetings start like this. It's okay. Here we go again. I was just about to say the same thing. Yes, nutrient brick. God, that's the worst day at the commissary, I tell you what.  Alice de Grey. With an E. I don't trust this rhyming man. Paul, are you the site director? You seem a little on edge, young man. The figures? Show up on my first day and they drop an atom bomb on me. Turns out my whole life, I have been protected from dangerous knowledge, from the truth about what is really out there. Like the entire outside world is a childcare center. What they showed me in there, how could any of that be real? Supernatural, it is meant to be fiction and... You didn't answer my question. We all go through this in orientation. You didn't answer mine. Yeah? Yeah, I did. I'm Paul. I'm from... You used to speak four languages. One now and soon. Zero. I'm trusting the rhyming man less. But there's too much world for one human race to ever understand. There's too much human race we should bear down. Are we getting SCP-5000 up in here? Excuse me, just for a moment. Hey, there's an intruder. I think it might be a... What do you call it? An SCP. Darkhead. Hello? Intruder. SCP, secure, contain and protect the bad thing. There's one right over there. Dark, come on. Hey, hey, hey, hey. People, people, look at me. Look at me. Hey, hey guys, come on. First day and already getting anti-meamed. Man, oh man. No. No, no, no, no, no, no, no, no. But yes. Harry and Wheeler. 49. Loving husband. Two boys. Full foundation credentials. Which we never issued. Including access to installations and rooms. Which don't exist. Or do they? It's not yet. That's before we get to your access control lists. Which I can only term as a gracious. Jesus spy. Let's get this over with. How you dullard. I'm your chief of anti-memetics. You don't have an anti-memetics division? They always say that. A telecontainment division? Ops A. Ops B. Personnel. D-Personnel. Two dozen others. But we don't have an anti-memetic division. Do we have an irony division? No? Alright. I'll have to check. We'll try this. Why do you think the anti-memetics division would show up in the listing? It's a division of the forgotten. There are SCPs that get into your mind and infect other minds. Contagious concepts that require containment like any physical threat. Secure. Contain. Protect. Right. Then, there are the other ones. Entities and phenomena that harvest and consume information. Particularly about themselves. Ideas that consume other ideas. That eat segments of reality. That was such a good acting moment. I'll talk about it later. Do you get it yet? There aren't any dangerous SCPs on this site. So either you're not dangerous or nobody knows you exist. You're either brand new or... You seem on edge. By the way, both these settings are shot so beautifully and they look so good. The colors. Ah, the framing. Also, scary James Marston is freaking me out. Spreading. Search anti-memetic tags. Here, SCP-4739. If you're reading this, you've already been isolated from the Foundation in large. Attempts to signal for help or futile. You are now inside 4739's gullet. You need to get to lab S41. Continue the research until you find a way to stop or kill Gray before he kills you. 4739, let's look that up when this is done. So, disgusting, decrepit, disorganized, a standard research lab. SCP-4739 is a power from the outside world by making it impossible to be perceived. The SCP log that you are currently listening to is created and maintained by victims of SCP-4739. Because it is only audible to victims of SCP-4739. It's in research and continue until you find a way to... Based on the research, nothing works.  It doesn't work. They've tried feeding him memories to overload him. It doesn't work. They've tried sacrificing other people to save themselves. It doesn't work. So that's it. I'm fucked. We've all had to come to terms with that once in a while. Sometimes we make it out okay. The population is constantly eating itself. These are class double enestics. The weakest type, suitable for continual use. One pill per day. Tastes like strawberry. You do. You forgot to take a pill. And then you forgot all the information that the pills were helping you to retain. You forgot about me and my entire department. How many other anti-memes are there? Ten that I know of. Probably at least five more. Two in this room at this very moment. Don't bother looking. It's pointless. There's one right here. It's like a demanding pet that eats my memories instead of food. But I've learned to control it. Where's the other one? I remember that from the tale. You can't skip a dose of nestic. You can't forget to take a pill unless someone prevents you actively from taking it. Get back on track. As ever, the search for a stalemate continues. I'll let you know if we get any closer. Have we got a plan? A solid block of information has got to be something in here.  Last chance, Paul. Last chance. An info weapon.  You seem on edge, young man.  Oh, talk about a hard drive. It was chewing its way up the Antimimidics Division hierarchy. Congratulations on demonstrating a basic level of competency where dozens of others couldn't.  I want to say I threw my phone and realized information hurt him. It was my first day and I got lucky as hell. I want to say those things, but none of them would be true, would they? This isn't my first day I've probably been working here for, what, ten years? Instinct and muscle memory are just deep forms of training. The ability to piece your life back together and all your past knowledge. That's what we drill into you. This isn't the first time you've seen me put myself back together, is it? And it ain't gotten old yet.  So what now? I need you to update everything you know about Gray and his predatory patterns. I need counter-measures for when it comes back. So it's not dead? Wait, I know this one. Ideas don't die. That was so sick. Ah, so cool. Let the credits run, let the caretists run. Give the people their props, give the people their props. Oh man. Alright, well that's running. Let me look up 4739. Ah, we love that creative commons at the end. We love it when all our creators keep things above board. Alright, so 4739 is not Alistair Gray. It's just spoiled milk, I promise, by Rock Teeth Moth Eyes, good author. January 18th, 2019. Specialized locker, 3.8 liter container of 2% milk. Expiration dated December 31st, 2018. Okay, so this is not the cannon. 4739, is it? Testing chamber, gunshot. What? What does it do? If you try to open it, it will react violently, inflicting harm upon the handler. Um, so... Okay, so the wiki's 4739 is not what we're dealing with here. But, if I go to Introductory Antimemetics, which is a tale on the wiki, by Quantum, then we get Junior Researcher Kim. We get all the hell of a first day, Alistair Gray with an E. Okay, so we're taking the, uh, lovely quotes here. Uh, very good, going down. Spoke four languages. One now, and soon zero. Woo, that's nasty. Good stuff. Excellent adaptation. Do-do-do-do, unlocking the terminal. I'm fucked. Excellent. Scrolls again, hard drives. Okay, so this is a very lovely and close to almost word-to-word version of this article. And of course, we gotta go to, yep, we need to talk about 55. Is the second tale that we're quoting here with Marion and 058. And, when I said wonderful acting choice earlier, it's because when Marion was standing with her back to the mirror, talking about Antimemes, she said, yes, we have multiple anomalies that handle X, Y, and Z, and the other ones. And when she said the other ones, she looked at the other guy knowingly. And so that was really funny. I knew that that's what they were going for, and that was a great little acting choice. Uh, whether that was her or Andrea directing that, most cool, most cool. This was gorgeous. I mean, it looked beautiful throughout the different color palettes for the A and the B plot. Both of them had a great tension, although, you know, it's a short and it's two stories in one, so it's two shorts in a short. And bouncing back and forth, the tension was able to keep going. The music and sound did a really good job of keeping that going, too. But the chase for Paul, the conversation where Marion's forgotten and she needs to convince 058 of what's going on. That was so good. I love to see it. I love to see good art in the SCP Foundation universe. I want to see more of it. So after this, we're going to go right into the next episode. Make sure that you're supporting Andrea over on his YouTube channel. Like, comment, share all that. Make sure that he gets the support needed to keep making more of this stuff. Thank you guys, Site 42 staff. This has been an amazing episode one. I'm very much looking forward to the rest and I'll see you all in the next video.


Hi and welcome to program analysis. In this lecture of the course we will look into path profiling which in short is an efficient approach to tell which path through a program are Executed. We'll see why this is an interesting approach. What algorithm you can use to solve this question And then we'll also look into some applications of this idea of path profiling. This lecture consists of three videos And this one is the first of these three So here's an overview of the three videos that we have in this lecture So in the first one I'll give an introduction into the topic and will motivate why this idea of path profiling is Interesting at all and we'll also look into some of the challenges And we will see why a naive approach that you might come up with if you maybe think about this problem for a few minutes Does actually not work that well Then in the second video, which will probably take most time We will look into a very beautiful algorithm to address the path profiling problem And that's the so-called ball-larus algorithm and we will go through that algorithm step-by-step using an example and then finally in the third video We look at the generalization of the algorithm that we talk about in the second part of the lecture And we'll also look into some applications of this overall idea of path profiling to see where it can actually be Used as usual everything I'm saying here is based on some other literature In particular, there's this one paper on efficient path profiling that describes the algorithm We are covering here and that is at the core of this whole lecture But there are also some other papers that you may want to look at that basically generalize this initial idea or show one of the many applications of path profiling All right, so let's get started by defining what this problem of path profiling actually is So in short, what path profiling wants to do is to give in a program and some input that makes this program run Count how often every path through a function or more generally to a program is executed Why is this interesting? There are many many different applications I'm listing only three of them here and we look at another one in the third video of the lecture So one application of this idea of path profiling is so-called profile directed Compiler optimization where the compiler does not optimize the program ahead of time without having seen any Rounds of the program but actually looks at an execution profile that tells you something about how this program executes and then uses this to Optimize the program and specifically a path profile can for example be used to optimize those part of a program that are Most most relevant and that are that are executed most often Be your automated automated compiler optimizations You can of course do the same through manual performance tuning So if a developer wants to know which parts of a program are most worth to be optimized then one way to Answer that question is to run path profiling which tells you which path through a function or maybe an entire program I executed the most and then this is where you should probably spend your time on if you want to tune performance and Then beyond performance, of course, you can also use path profiling in testing for example To find out which paths are not yet tested So if you know which path I executed and how often they executed you also know which path through a function Or maybe an entire program are not yet tested at all or maybe not tested well enough and should Get a little bit more testing time Now if you hear this for the first time this idea of counting how often every path through a function or a program is executed This may sound like a simple problem But doing this is actually not so easy and the reason are three challenges that I'm listing here So one is that we would like to have an approach that imposes low runtime overhead Of course, if you have all resources of the world, it's relatively easy to come up with an approach But in practice you do not want to slow down the program too much when you do this path profiling And doing this whole analysis with low runtime overhead is actually a pretty interesting challenge the second challenge that That needs to be addressed here is a curiousy So what we want to have is a precise path profile that really tells us how often every path has been executed And we do not want to use any heuristics or approximations here Of course, you could take a different design decision and say it's okay to use heuristics It's okay to do approximations That's also a valid choice. But what we look at here aims at precise profiles that do not approximate the correct profile And then finally, of course, there may be infinitely many paths through program in particular because we have loops and also recursive calls which essentially mean that there are cycles in the control flow path graph and how to deal with these Potentially infinitely many paths is part of the challenge of addressing this path through finding problem To make all of this more concrete, we'll use a running example throughout this lecture, which looks as follows So we look at this example not by looking at source code, but by directly looking at the control flow graph And the the reason is simply that you've by now seen enough Examples where we went from source code to a control flow graph. So I'm assuming that You know how to do this. So the graph that we look at looks like this So we have these basic blocks or they could also be statements that I'll call abc def And then we have edges that tell us which statement or basic block may be executed After another so b may be executed after a But also c may be executed after a so there's some kind of conditional Then after b c may be executed Then there is d which may be executed both after b and c And then down here we have again this kind of situation where after d e and f may be executed and after e We may also get to f So assuming that a is our entry node and that f is our exit node You can now think about which different path through this Control flow graph exists and let's just draw a little table To do this. So I'll give each of them a number and then I'll list the different nodes that the path goes through and what we want to know at the end and that's the problem of Path profiling is what is the frequency of each of these paths? So given an input that may go through this Piece of code that is shown here in the control flow graph multiple times How often is each of these paths actually executed? So this is the big question that at the end we hope to answer through path profiling So let's now go through all the possible paths one by one And I'm starting with the path that goes from a to c to d to f. So basically Like this So that's one of them Then there's another one that goes from a to c to d but then also goes to e before going to f So this one will look like this And then of course we can also Take the turn From a to b at the beginning. So for example, there's one path that goes from a to b And then to c Then to d and then to f So this will look like This Then there's yet another one Number three that Also goes from a to b Then also to c then also to d but then also goes through e before ending in f So this will look like that and then yet another one that Takes a short path Because it goes from a to b directly to d and then to f So like this And finally there's one left that we haven't covered yet, which is a b d e and f so like This and now given these six different paths that exist um If we Execute this function or this piece of code that is represented here multiple times The question is how often is each of those executed and this is the question that we want to answer here So looking at this problem one idea that you might have is what Is called edge profiling So this approach basically starts by looking at each of the edges in the control flow graph and by counting How often each of these edges is executed? So in practice if you would implement this you would basically instrument every branching point In the code so that you then know whenever there's more than one One than more edge that you could take which of these is actually taken so that you at the end know how often Each of the control flow graph edges is indeed executed And then giving these individual edge counts one could compute or maybe estimate the most frequent path by basically following The most frequent edge all the time So we would start at our start node and then whenever there's more than one edge We would follow that edge that has um the higher frequency until we reach the exit node and then assume that this is the path That um is also the most frequent one So let's illustrate this idea of edge profiling with our running example again So this is the same control flow graph that we've seen before and now let's assume that we have computed How often each of the edges in this graph is taken and that we basically have Associated a frequency value with each of these edges So for the example, let's say that This edge has been executed 120 times this edge has been executed 150 times which basically means The code has reached a 270 times and then 120 out of those it has Continued with b and 150 out of those it has continued with c Then let's say that um 100 times we have seen this edge to be executed 20 times that edge 250 times this edge here 160 for this edge 110 here and 160 the e2 f edge Now using this edge profiling um idea We could now try to answer the question which of all these paths through the program is the most frequent one and as I said, um the idea here would be to basically look at All nodes where we have more than one outgoing edge and then always take the edge that has the higher frequency So we would start at a because that's the entry node We can choose whether you go to b or c. Um the edge from a to c has the higher frequency So we would go to c and see there's only one option which is to go to d In d we have two options again. We could go to e or to f Based on the edge profiling idea We assume that the edge with the higher frequency is taken Which would be the one that goes to e and then from e there's only one choice which Gives us this Supposedly most frequent path a c d e and f So from the way i'm explaining this approach you might already see that This may not be the correct way to actually do it So um if you're a bit skeptical you may ask well really Is this really the most frequent path and is this the only one possible correct solution here? and to Answer that question let's have a look at two different possible path profiles that may actually um have happened So in this table you can now see again the six path that we have already seen on the on the previous handwritten slide where we Yeah, just list all the six ways to go from a to f and now let's assume that we have executed our program where This control for graph is part of um two times and that we have obtained with some approach that we do not yet What that i haven't explained yet And let's assume that we have obtained the following path profiles on profile number one We know that this very first path that goes from a to c to d to f has been executed 90 times The second path has been executed 60 times The third one the a b c d f one has never been executed The fourth one has been executed 100 times and the last two 20 and zero times Now this is one profile that we may have gotten by exercising this program with some inputs And now let's say we have some other inputs and exercise the program again and then get some other profile called profile two and again I'm just giving Some numbers here that we may have obtained In this profiling that again just tell us how often each of these Paths have been executed So now if you look carefully at these numbers and basically do the math You will see that for each of these two profiles both profile one and profile two We would have or we could have gotten um the blue frequencies of Edge executions So both of these profiles are compatible with the edge Frequencies that that are annotated here on this graph But now if you look at um the most frequent path that we see in these profiles Then you see that here for profile one. It's actually this one a b c d e f Which is different from the one that we have seen up here Whereas for profile two the most frequent one is this one A c d f which again is also different from this supposedly most frequent path So the bottom line of this is basically that um this idea of edge profiling even though it sounds simple and maybe very efficient um Doesn't really tell us The most frequent path simply because the information that we are gathering is too local in a sense and it doesn't tell us enough about The entire execution of the program and how it goes through this control flow graph So i'm getting back to this general algorithm of edge profiling What we've basically seen in this example is that this idea of edge profiling Fails to uniquely identify the most frequent path and may actually give us a wrong answer So this is not the solution that we have been looking for All right, and this is already the end of the first uh video in this lecture on path profiling You now know what the problem of path profiling actually is and you've at least seen now how to not solve it So I hope you are now interested in seeing the second video where we will look at the ball arrows algorithm That will actually solve this problem of path profiling. Thank you very much for listening and see you next time


Okay, before NHGRI can issue a contract solicitation for a research and development contract, it's required to have the underlying concept be approved by an advisory group in an open setting, public meeting. And we've always used the advisory council for all FOAs that we issue so that you'll know the full range and scope of things that are being planned by NHGRI. Now we've heard earlier this morning presentation about the NIH data commons and that was sort of a segue or introduction to the concept that Valentina DiFrancisco is going to present to the council. There will be ample time for a discussion with the council, but at some point when that discussion is concluded I will be taking a vote asking the council to approve or disapprove the concept. So Valentina. Good afternoon, I am Valentina DiFrancisco, I'm a Program Director in the Computational Genomics and Data Science Program at NHGRI. And I'm speaking on behalf of the NHGRI Sandbox Group. This is a group of program directors and program analysts that we have been working together for several months now to develop this concept. They're all listed here, but a special mention goes to Kevin Lee, Chris Wellington, and Ken Wiley. Ken Wiley is going to help me out with answering some of the questions today. So the outline of the presentation is the following. I'm going to touch upon some of the current challenges that are caused by the genomic data avalanche and they are associated to genomic data sharing and analysis. I will describe the features of the proposed sandbox. I will touch upon some of the available tools for interoperability that can be used across different emerging data commons. I will describe briefly the type of users that we expect for this type of resource and finally describe the funding mechanism. So the challenges of the genomic data sharing and analysis. In an era with increasing sequencing capacity and decreasing cost, the bottlenecks are becoming more and more linked to data management and data analysis. So for that reason, researchers need scalable high-performance storage and computing infrastructure and technical expertise. The problem is that the technical expertise and the scalable computing infrastructure is not readily available everywhere at any research institution. Some research institutions have all of that. Some others really actually need a lot of help from that point of view. The other challenge has to deal with distributing big data over the internet. People have been complaining about the inability to transfer terabytes and petabyte of the data over the internet. They use hard disks, tapes, and so on and so forth now. And also, the distribution is very inefficient because in some cases, the same data sets are distributed in multiple local systems or even copied multiple times on the same cloud resources. The other issue is that, for example, EnergyRI is funding a number of sequencing studies, and data resides in different local systems. There is little to no attempt in trying to consolidate and harmonize the phenotypic information that is being made available by the studies to the rest of the scientific community. And so phenotypic data integration necessitates a little bit of coordination. And finally, and this is something that is quite some recent news, NCBI has recently communicated to EnergyRI as well as to other institutes that it is reducing its data archival role for the NIH. Okay, so this clearly has a big impact on what we're going to do in terms of providing data generated by our studies to the rest of the scientific community. So the solution that is proposed today is this concept of an EnergyRI data sandbox. Really for us, this is a resource that is meant to democratize genomic data access, sharing, and computing. It leverages a scalable cloud-based infrastructure to collocate data storage and computing capacity. We commonly use tools for analyzing and sharing data. It provides both unrestricted and controlled access data and metadata from EnergyRI fund the programs. It is a trusted partner of the big app. With respect to genomic data, what I want to emphasize here that I don't mean just genome sequence data, but also data coming from genomic studies. And so genomics should be really used in a very broad sense. The other thing that we would like the sandbox to do is also to provide some data harmonization services across the EnergyRI fund the programs. And also, as was pointed out earlier today, not all the tools are readily available and run on the cloud, on a cloud platform with how efficiencies. So we expect personnel in the sandbox to actually keep on developing and optimizing the tools for the cloud platform that they will use. And finally, facilitate interoperability with other data resources and data comments. So you've seen these slides earlier today by Vivian, so I am not going to go through it again. But you can just generally imagine that this resource will have some an access portal, user interfaces. It will implement security measures to make sure that controlled access data is kept private. It will implement fear principles, the fair guiding principle. And if they do not exist, it will help contributing to the development of those. And so on and so forth. So I'm not going to go into the details of this slide. One thing I want to however emphasize is that in 2015, the NIH has issued a notice that basically allow for the use of cloud computing services for storage and analysis of controlled access data that is subject to the NIH genomic data sharing policy. So that basically is allowing us to propose this concept that relies completely on a cloud platform for sharing data with the rest of the world. And the other thing is that in order to do that, we are going to make a requirement for the cloud providers and for the group that is going to support the platform to utilize a cloud platform that is FISMA-moderate and HIPAA-compliant. So you heard from Vivian earlier today about this emerging concept of a federated data model. The one on your right is the EnergyRI data commons or sandbox. We have been talking about what type of data sets we may want to use to initially populate the sandbox. We came up with a few. These are just proposals. They are now written in stones. And of course, the data sets will increase. They will change over time. But for the moment, we thought that encode data that comes from the eMERGE network, the data from the 1000-genome project, and the data from the genomic sequencing program will be good data sets to initially populate the sandbox. At the same time, as you heard, there are multiple NIH data commons that are emerging. One of the NCI genomic data commons, one from H&P, the one to support a human microbelom project data. And other data commons to support top-made data from NHLBI programs, and as well as to support the PMI efforts. So the issue always comes, and we heard it earlier today, is how are we going to make sure that this data is not siloed and it's actually going to be a support for users to basically go across different data resources and deploy methods across different cloud systems or across different data commons in general. So these are the tools that we think will alleviate some of the concerns with respect to interoperability. For example, the establishment of a common user authentication system. The adoption of common APIs for data access and computing could be the GH4GH or others that may emerge. The adoption of fear principles. The broad use of Docker containers to deploy analysis tools on different resources. Catalogs of digital objects so that people can easily find the data where it resides. And also the adoption of common data standards and ontologies. I'm listing all of these because I want to emphasize that there are tools that are either already built, and tools and technologies are either already built or that we're already working on. Some of the BD2K programs are actually really pushing a lot of the development of some of these tools. So I think the real issue here is really mostly to do with social engineering and making sure that the right group of people are around the table to make decisions about common adoption of some of these solutions. So in terms of users of the sandbox, we expect two different type of users, primarily. One is a computational genomic scientist who is familiar with cloud or high performance computing, developed a tool and wants to test it out or utilize it on a high performance computing platform. The other one is a researcher that actually does not have this high level of expertise and may want to use graphical user interfaces or interfaces to analysis workflows to basically simplify their analysis needs. We don't expect really genome sequencing centers to utilize these resources. One, because they have their local system and they will probably will not need to come to the sandbox to do their own analysis. And with respect to what type of users also will come to us, there are two types that we think will come to the sandbox. One is an individual user, a user that is not part of an energy at I funded consortium. And they may want to upload and compute on the sandbox and on the sandbox data as well as on their own data. They will be able to upload their data, combine it with data that resides already in the sandbox and do the analysis that they need to do. So the other feature that is also very important is that they will not need to download the data from the sandbox. They can basically do all the analysis where the data is hide. And by doing that, basically we're minimizing potential data incidents. The other thing is that they may want to have access to commonly used tools and workflows. As I said, one of the activities of the sandbox is going to be optimize the tools and workflows in a way that will run smoothly on a cloud computing platform. So they will be able to use those tools and also have a workspace for their own new tool development and sharing. So how will, so if a user, if it's our user will want to have access to the energy at sandbox, how would that happen? Well, we're not changing anything really. The way that this works is the following, is that that's the way we propose it will work. That a user will be a registered user of the ERA Commons. It will have to submit a data access request to DbGaP. It will use all the data authorization protocols that have been put in place in DbGaP. So if they're asking for NGRI data, the NGRI data access committee will review the application and will approve it. And then with an approved DAAR, then the user will be authorized to use the data that resides on the NGRI data sandbox. No changes to the current protocol, basically. We're utilizing a system that is already in place with all the security checks on approvals and consents and so on. And there will be no transfer of data from DbGaP to another system. OK, so with respect to potential costs to individual users, so the sandbox, if you think of it, given what you heard, that means a user could upload potentially terabytes and gigabytes of data, which is going to cost. And we will not be able to basically cover the cost for all their needs. It's not a free resource. So there will be cost associated to data storage and also cost associated with computing and as well as cost associated with data egress.  I just want to emphasize the fact that costs are changing all the time. They are decreasing quite rapidly lately. And also the cost will change depending on the actual tools that are going to be used, depending on the optimization of the tools that people are going to use, and so on and so forth. So please take these numbers just as a general indication. But anyway, the current data cost is about $350 to $450 per terabyte per year, depending on the speed of access to this data. If you need to have fast access to the data reciting on spinning disk, then it will be the most expensive version of that cost. So with respect to computing, you can imagine a user that wants to use a standard Vader calling pipeline. And in that case, the cost that I was quoted is about $270 per sample on AWS on Amazon Web Services on Demand. Or if you use the AWS SpotEast, those costs actually reduce significantly to $30 per sample. Again, with an RNA-seq analysis pipeline, with the tuxedo tool suite, basically the cost will be around $16 on AWS on Demand, or $2 per sample for SpotEast instance. These are just an indication of what the cost could be. So the other type of users of the Sandbox are potentially members of an NGRI Consortium. So Consortium members could use the Sandbox to have access to cloud resources where they can deploy new data sets and workflows. They have a common infrastructure to share and compute on consortium data. And they can take advantage of interoperability tools and data submission services to other data commons if necessary. And that includes also data submission services to DBGAP in case they will accept those data. So members of a consortium could interact with the Sandbox in two ways. They could either have their own separate data coordinating center, and then the data coordination center could submit data to the Sandbox. Or they could basically leverage the data management services of the Sandbox. So in all intents and purposes, the data coordinating center, some aspects and functions of a data coordinating center could easily taken over by the data Sandbox. And in that case, each project could basically directly submit their own data to the data Sandbox for storage and sharing with other members of the consortium. So a consortium member, in that case, who wants to access data from the consortium, all that he needs to have is an account on the data Sandbox. In that case, program officers will work with the energy rise Sandbox, will basically recognize the user as a consortium member, and they will grant authority to the relevant and appropriate data sets. So user authentication and authorization is going to be done by the Sandbox in collaboration with energy rise staff as well. So we think that this system should greatly simplify access to data generated by a consortium compared to current methods. So finally, with respect to the funding mechanism, as Rudy mentioned earlier, we're proposing this resource to be supported through a contract funding mechanism that is required to establish a trusted partnership with DbGAP. And the reason why a trusted partnership is necessary is because we want the Sandbox to make accessible controlled access data to the broad scientific community. The contract also allows us to basically have a clear set of deliverables and milestones that the staff of the resource will have to meet for achieving additional, for basically receiving additional funding. They will have to provide special reporting requirements and mostly they will have to work really closely with energy rise staff. As I said, there will be a lot of interaction with the selection of the data sets that the Sandbox should support. There will be a lot of interaction with the type of services that the Sandbox should make available to consortium members and to other users and so on and so forth. So the funding period is for seven years. And really, we select the funding seven years because we wanted to make sure that this resource will have time to basically develop and also be tested in terms of its utility to both outside users and consortium members. And typically, programs in a GRI funded programs are for five years. Having said that, as I said, there will be milestones that are going to be set up for this resource. And so if for whatever reason the project is not successful, we will not support it any longer. So the timelines, because this is a contract, is going to take a while before we're actually able to issue an RFP, we think that we should be able to have an award in the summer of 2018. And we think that in about nine months after that we should be able to have a resource that is open to the public. And with that, I'll take any questions. And Ken too. Cattle, go ahead. Thanks, Valentina. So I have a couple questions actually. And the first is, how prescriptive do you envision the RFP being with regard to the data sets that are expected to be in this instance and the tools that are to be available to the researchers? Is that going to be defined up front or is that going to be the responsibility of the applicant to actually come up with those ideas? So that's one. The second one is, can you say a little bit more about the governance model over those seven years? Is it going to be entirely NHGRI staff or is there going to be an external advisory board as well sort of monitoring progress? Yeah. So with respect to data sets, I think we will select some data sets to initially populate the resource. So we will be prescriptive from that point of view. I can tell you we had a lot of discussions, internal discussions about what would be a good set to start with. But we really want to make sure that we start with a set and people know how to plan when they're making their budget proposal and so on and so forth. So with the tools, we have not had any discussion yet about the specific tools. So I can't answer that question yet. The second question was? Governance body. Governance body. With all our programs, we always had a scientific advisory board, so a group of outside members that will serve as advisors to this. So it's not going to be just NHGRI staff. So for that advisory board, is it going to be a combination of the contract awardee naming people to the advisory board and NHGRI or is NHGRI going to? Typically what we do with contracts, there are some people that are being proposed and typically NHGRI staff who will vet it and approve it or not. But most times, we will make sure that there is no obvious conflict of interest in the management and oversight of this group. Before we get too much into the details of this, can you just give me an indication of how this is different and how it relates to the NIH data commons? I'm still confused about that. Yeah. So this huge cloud is supposed to basically give you an idea of a number of different data sets from different projects across the NIH that are supposed to contribute to the NIH cloud. So what happens really is that NCIS set up their own data commons, which is part of the NIH cloud. NHGRI would like to set up our own data commons, which is going to be part of the NIH cloud. And the question then remains, how will these groups of these data sets interact with each other? And what I'm saying is that we are either building tools that will facilitate interoperability on the NIH data commons. So all of these data sets, the PMI, TopMed, HMP, the NCI data, NCODE, and GSP, eventually they will all contribute data to the NIH commons. But then why do you need to have the sandbox populated with your own data? It seems to me what the sandbox sounds like to me is a workbench with all kinds of tools that you then use to access data that's in the commons. Because we need to basically facilitate coordination across all of these activities. Right now, the situation is that people are just contributing to the big up and data is not coordinated. There is no help in trying to have tools that run across multiple data sets and so on and so forth. So if you have one place where you have the ENCO data sets, the GSP data sets, with tools that run on it, I think that is a good service for a user, for users. And right now, that is not doable. And I'd like to add that we also, if you look across the programs in HRI, oh, sorry, we look across programs just in our institute. I mean, we've set up some programs that have set their own cloud providers. And the idea is that this will help us provide an avenue not just for us to work interoperable with our own data sets, but also with other sets. And let's face it, we can't have one comment, one cloud provider is going to hold all our data. That's not feasible. So you're forcing, we really don't have a choice but to try to parse out these elements that we can and build it in a way that we can make it interoperable with each other. And I understand that's a challenge, but it's something that we're, at least the groups that are representing this model here, are working to try to build that in that capacity. So I have some concerns about the computing costs that you suggested. So you were estimating it would be about $270 for aligning a 30X coverage genome. And so I guess I have some concern that those kinds of costs might be prohibitive for a lot of users. So for contrast, that to my own experience doing things like that at my home institution, computing is heavily subsidized, and so that would be sort of effectively free for us. And so users are going to be trading off the clear utility and advantages of doing cloud computing against the costs. And I'm wondering about what your thoughts on that are. I'm just worried that the cost might greatly hit the usability of this. I mean, it is a concern, right? But the point is that not, as I said, our intention is really to make sure that users, especially researchers that don't have a local system that can support this type of analysis, can actually have a place where they can actually do it. This can actually solve it. I mean, the local system isn't really not for free. I mean, in a sense, NIH is paying for the overhead and all the other activities of different institutions. But that's true to some extent, but for an individual investigator you're doing the calculation, is it cheaper for me to do this computation on a server in my lab or on a system that I have locally as opposed to in the cloud where I'm paying for the compute cost? So I can tell you that there are now people that are developing tools that will give an estimate of the cost of your compute before you actually submit it to a cloud resource. That may also help the investigators make a decision about what type of cost they're going to incur and decide whether or not they can afford it. But I hear you. First of all, thanks for a very nice talk, Valentina. And a comment and then a question. So I think some of what Jonathan is worried about exists because groups like his and mine and people around this table and your funded community in general have not had this resource. So one only wonders in five years or in x years forward if we would continue to make those decisions to build up our own computational infrastructure, I think the idea of this if it works is that you wouldn't have to have any of that and you wouldn't be making those decisions. But your point is let's make sure then that we get it right. Back to Mark's question about how this relates to the commons as a whole, I think it is a good question to really understand what the relation is. And my question relates to your perceived allocation or focus. So there's sort of hardware, which can be broken down into computing versus storage. But then it seemed like there was quite a bit of a software construction and service component to your vision. And I was hoping you could clarify some of that. So just to kind of give some concrete examples, you mentioned for seeing a bunch of data harmonization. And that could be light harmonization or that could be a major activity. So for instance, would you recall every genome sequence in the sandbox using a common pipeline? That would be very useful but also a major commitment. Or did you mean other things by harmonization? And as another example, you mentioned that simpler tools and web interfaces might be provided by the sandbox for users who are not computationally savvy. And I'm wondering how many or how much development effort you foresee going into those tools by this initiative versus simply providing a place where some of us can put our tools we've already developed or in the process of developing. A lot of questions I realized, but they all relate to the relationship at the ground level with the larger project. So with respect to what type of services the sandbox is expected to provide, a realignment to a new version of the human genome, I can easily imagine that it's going to be one of the services. But once again, we're going to have a governance body, including energy or eye people and outside researchers that are going to tell us whether some computational exercises need to be done with the associated cost associated to it. So the answer is yes to all of the above with a careful evaluation of priorities and cost. And that's the kind of thing that a contract will allow us to do. So what was your next question? Well, I think if it's yes to all of the above, then it gets all of the questions. So the harmonization part, the harmonization part also what I was really referring to was mostly about harmonizing phenotypic information. Standardizing information associated to data sets and things like that. But again, we are going to decide over time, depending on, as I said, we're going to set different milestones. And the harmonization component is probably not going to be the first milestone as opposed to maybe the fourth or the fifth. We will have to set priorities about what we need to do first and then later. And then back with relation to the commons, presumably all of those NHGRI specific activities would somehow live in the commons framework that the commons is supporting. Is that right? Yeah. Yeah. So a slightly different question. You talked in earlier about milestones and sort of judging. Can you give a little bit more idea of what you're going to consider metrics for success of this program? Yeah. I mean, we have not really decided yet how to come out with the metrics for evaluating this program. And that's the answer. So let me just add one more. So some of those are going to be built. And this is, as we're thinking this through, some of those we built off of what we've learned from existing models that are currently going on in both within our institute and outside. We have certain programs where we are actually looking at things such as the user interactions with the data sets that are in different clouds as in one of the matrices. We're also looking at how are these different data sets in cloud platforms that we have? We have data sets that are from different platforms. How are they being utilized to actually work together and build a collaboration that wasn't already there? Those are things that we can also look at. So it's not completely in the dark where we're coming from as far as matrices that we would consider as far as milestones. We're looking at this, building up of what we're seeing around the landscape. As a starting point, I understand that that could potentially change as we move forward. The sandbox is not something that's designed to be scalable, both upscale and to downscale, also based off of these matrices that we're going to put in place. But that's to help answer your question, that it hasn't been finalized, but we are looking around the landscape of existing infrastructures in cloud platforms to help us guide us in that direction of what we should look at for our matrices, milestones. And I appreciate it's a very fluid situation and the field is moving extremely quickly, but it makes me a little nervous when your basic answer is, I don't know what those metrics are. So I can tell you some of the metrics that have been used by similar resources in the past. What we didn't do is to make a decision about whether or not these are the ones to use, but I can tell you the initial population of the sandbox with data sets and making these assets available is definitely a milestone. Making this resource a trusted partnership on the big app is another milestone. I mean, these are key aspects of this resource that need to be achieved. Implementing some of the tools like, for example, a genome browser that will basically be able to serve the data to some of the users. And X number of tools which we still need to define will be another sandbox. So there will be a different staging steps for this resource that they have to meet otherwise. And finally, also another thing that has been done, which I think we're definitely planning to do is to provide incentives to outside users to come and use the data and test it out. This has been done by the NCI Cloud pilots and they learn a lot through the system. And I imagine that a version of that, giving incentives to people to come and test out the resource so that we can improve it, is part of the plans. What we have not finalized yet, what those metrics will be, but... Yeah, and I appreciate it. I mean, I would not expect final metrics at this point. You've got time to think this through and there are a lot of issues that you need to think about. But I would think that one of those metrics would be, if only 10 people are using this three years after it's there, we're probably not where we want to be. And that's where you're seeing in some of our programs where we're testing those auditing, like how do we define what is the critical level of interaction with data sets to say you should be kept. And we're testing those out. And like Valentina, we haven't finalized those things, but we are looking across and implementing some tests of these ideas of these auditing tools in different programs and we're trying to collect that data, but we are collecting that data to help better assess how we want to move forward. So can I just clarify when people are talking about milestones? We're talking about milestones for the offeror meeting goals, or we talk about milestones for it's useful to the community and it's getting broad use or both. I was actually thinking more about the broad community, but I think the two are interrelated. So it would be both. As you can tell from this conversation, this is a very rapidly moving field. The technology's changing, the nature of the data's changing, and the needs of the community are changing. What are the plans to stay very flexible and meet the changing needs, particularly of the community, because they are our constituents, using a contract mechanism? How are you going to interact with any contractor? So this has nothing to do with the solicitation, but how do you plan on interacting in a contract environment to maintain maximal flexibility to meet the needs of the community? So there are contract mechanisms, and unfortunately I cannot go into the details, but we're looking at a contract mechanism that will allow us to have exactly that flexibility with respect to definition of tasks and concrete projects that need to be achieved in a reasonable amount of time. The mechanism exists. Yeah, I guess I had a couple of questions. One, somewhat building on this discussion. We started out and was mentioned earlier this morning that one goal of this was to make it available to universities and sites that may not have this infrastructure, but I really didn't hear anything about training. I mean, those people are unlikely to have the resources on hand to be able to use it, and I would certainly think it would be very important to build that into the concept, and I tried to look through it, and I also didn't really see that addressed. I have to admit to being concerned about the seven years, particularly if you're talking about not even awarding it for a year and a half, if I understood correctly. Well, it takes a lot of time to develop a call for proposals. But it's really hard for me to imagine what our compute space is going to look like in eight or nine years from now, and so I assume there's some type of annual review of this or update, because, yeah. There it goes. No, I understand you can't talk about dollar amounts. No, no, I wanna give you an example of a contract that we also oversee that's very large that just got renewed for seven years that is in the past iterations ended up doing things that were not written into the contract from the beginning. Right. And so you can write these. I just wanna assure everyone you can write government contracts of the nature that we're talking about to be incredibly flexible and they can be terminated at any time. Okay, well, so that was my question.  You started out by saying DBGAP is limiting the amount they're going to store in the future, but it doesn't sound like you're talking about this space being long-term storage. So I guess I'm a little confused about the storage versus accessing it for a year while you're computing on it, but that's different than... So the news from DBGAP and CBI is actually fairly recent. And so as we were designing the concept, we were not thinking about this serving as an archival system. We were assuming that that was going to be done by DBGAP. So it is, we have not, we still have to change the concept and decide what to do with archival. I think I recall that many years ago, NCBI was criticized for developing tools for literature searching and things like that, that was pushing out private companies from doing that, that they sort of dominated that space and therefore suppressed the development of those tools in the private sector. You're talking about developing tools for sequence analysis, and I wonder if you might be, if you're worried about being subject to that kind of criticism in the future. I'm actually, I mean, this is not supposed to be, the resource personnel on working at the sandbox are not going to develop tools as opposed to adapting tools to make sure that they work on the infrastructure and the interfaces that they have. Okay, users, individual users or consortium members can utilize the computing resources and the storage area and the workspaces to basically test out their tools or develop new tools. But I don't expect the sandbox people to really be the people that need to develop new tools as opposed to optimize the tools that have been done by other people and serve them to the rest of the community. Okay, this is a very interesting proposal because I think having all of these different data sets together in the same place where you can compute on them is gonna be really valuable. But on the other hand, I'm a little concerned with the cost, I think, like Jonathan was saying. And other than things like the fly stock center and stuff like that, I'm not really aware of anything where you have to pay to get something. Like, if the Santa Cruz genome browser, you had to pay to look at that, people probably would not use it. So I think the cost has to be perfect. Like, it has to be such that people, it'll be cheap enough that people will use it. And that's, like Jonathan said, a lot of people basically have free compute power at their institutions where, even though the NIH is still paying for it, it's through everyone's indirect costs, so PIs don't see it. As soon as you start charging it on direct costs on their grants, it's gonna be totally different. So this has to be so good that people want to pay a lot of money to use it. Or it has to be basically free. So I think that's gonna be a really hard balance to strike. And so I don't know if there's a way of giving people like NHGRI grantees a certain amount of free use per year. I don't, I think it's gonna be very tricky. Vivian, were you gonna talk about the credits model? That's exactly what I was gonna say. Why don't you go ahead and say a few words about that? Right, I didn't talk about it this morning because of time constraints, but we're working also in the commons for what we call a credits model, which allows you to get dollar-denominated credits from NAH for a grantee to expend on a collection of services in the commons. And that test is going on 81 requests that have come in now, we're gonna be testing it this year. The purpose of that is to see how are people using the data and to give them a method so they can actually pay for that and see what the cost is. And we expect the cost to be changing, in part because it's a changing landscape and in part because we see the cloud providers wanting to be more part of this. So there's a down cost, there's a reduction cost there. But we're also seeing this as a way of saying, okay, NAH would be paying for a certain amount in this credits model so that you can then use it for those services. Right now that's outside of the grant system. The intent is to move it inside the grant system. So a grantee would get, say, an R01, wants to consume NHGRI data and they would be able to get some dollar-denominated credits to then expend over that to use that. It gives us, somebody was asking about metrics here, how much you're using and what cost is it? And what is NIH actually expending on the use of storage of that data and the use of it? That's another part of the commons model. For those who wanna know more about it, I can take that offline. Yeah, I think something like that would help a lot because if the cost model is wrong, it won't be used. So actually my question was again about the payer model and it's only mentioned in the draft as that the contract will be used to build a resource but not for the users to use the resource, right? So there's gonna be expectation that people have to pay to use it and then there's not really any details. So is the credits model kind of what the thought is for? I mean, it's really, I'm having a hard time figuring out if this thing was built and if I wanted to use it, do I use my credit card, do I use grant funds? How would an individual user actually pay for use of this resource? And it's very high level currently in here. I'm wondering if that's gonna be spelled out in the RFP or if that's gonna be something completely different. We will have to spell it out in the RFP, yes. I wanted to go back to the question about the single versus multiple solutions. I can see all the value of having just one. It's already complicated enough and expensive enough, no matter what the cost is, but I also see the challenge and this is a little bit about, I think Mark said that about, it stifles certain types of innovation and the standard of tools and analytical approaches are developed by anyone on top of it but there is inherent innovation to the sandbox itself, to how it's engineered, to the design choices that are made there. How would that be handled and what would be the opportunities for opening up more than a single one and learning some lessons in the process? So I don't know, Vivian, if you want to say something about what NIH is planning to do through CIT and so on with respect to using multiple platforms. Again, Valentina, did you point out? What CIT is doing with respect to using multiple cloud platforms as a solution for NIH. Okay, but I'm trying to tie in with what Vivi, you were just saying, if I understood this right, you're asking the sandbox itself is in for its own level of development, right? Yeah, I'm asking on the sandbox itself, not the general, in the NIH there will be multiple solutions, they will all play with each other. I think we've asked that question before. Here there is a choice to make a single one for the NIHRI and I'm only asking about that choice. So in terms of making a single one as opposed to using multiple cloud platforms, as you said, it's a matter of cost. Now, we are hoping that we can leverage some alternative contracts that NIH is negotiating with other cloud platforms so that we could leverage, reduce reduction in the cost. And that would be great from that point of view because then it will allow for competition across different vendors. With respect to the science that goes into data management and to serving a research such as this one, absolutely. I mean, this will have to be an efficient resource, right? And some of the tools that need to be developed are brand new. So there will be a component that is related mostly to computer science and data management that the contractor will have to be able to satisfy. And that will be reflected in again, in milestones, in metrics, in efficiency, and so on and so forth. So we will make sure that that research component is included in this, but is going to be focused on the things I just mentioned, data management and support of the resource and funding efficient solutions for support in the resource. Let's disambiguate because I think, Aveve, you're asking an important question which is the workspace itself requires work in itself, right, and how you do that across the different places. So it's actually a multiplicity of issues. The first one was answered just by Valentina, right? Second one, which is about across the platforms, is currently underway in part of the discussions we're having across the commonses. And Valentina's comment about Andrea Norris and CIT is she's helping us with some of the negotiations for the cloud providers in the right place we need to do that. And then there was an additional kind of questions related to how does this fit into the overall commons? This is an example of a commons, right? The architecture is the same. We're all working on a very similar platform. What we have to figure out is how do we work, we're all doing similar things. Where are they the same and where are they different? That's what we have to do. We haven't done that yet. I don't have a sense yet of whether, are we talking about a single contractor that's likely to provide the services, not multiple contractors who will provide different components? We're looking for one. One award, one award I may have. They may have subcontractors. And that's likely to be a commercial entity? No, actually, especially for the R&D component that we were discussing, we would love to see also academics applying for this, to be honest. And what do you anticipate will be the impact on NIH personnel itself? Is this something that's going to have to expand the workforce here in order to maintain this long term, or is it going to reduce staff with economies of scale, or what's your sense of the staff impact for this? You mean from the extramural point of view or? Oh, really from the intramural point of view of interacting with the research community to help maintain this in a way that the community wants to see this develop? Yeah, that's a good question. I don't think we attached up on that yet, but it's something to keep in mind. Just one more point of clarification, kind of following up on Sharon's comment about outreach. So is there going to be an outreach requirement in this? Or is there going to be a separate group funded to do? No, there's definitely going to be an outreach requirement and training requirement. Right within that. Right within this resource, yes. I come back to the point that I think I made earlier, and that's the business of how you're going to harmonize this workspace, I declined to use the word sandbox, with other planned or underway workspaces at PMI, at NHLBI, and I don't know what the role of NCBI is in all this except I just heard that they decided they don't want to have anything to do with it, and I'm not sure how that decision got made, but that's a separate question. So the business of harmonization with other, how are you going to arrange that in a contract that you're going to issue to a sole vendor to make sure that they have harmonization with other vendors who may not be the same? You make it as a task, as a requirement, right? It's that require interoperability with other entities as they emerge. Vivian gave a number of emerging data comments, they're all emerging, and the NCI genomic data comments is adopting the GA4GH APIs. We will probably keep the requirement genetic without specifying the specific APIs that will need to be implemented, because again, the world is changing over time, but what we typically do in the past is just to set up the interoperability requirement as a task of the contract. So I think this is another example of the importance of making sure that we are all working together between the commons and HRI and HLBI and those other entities that we can, because as we are starting to develop our language in our contracts, we have a better idea of how to make that language interoperable with each other so that we can have, so if we have a solicitation, they understand what the expectations are, that it's not just for us, it's also how we work in a federated model. I think that's how we're able to start by having this conversation with our colleagues around an age. And by the way, with a number of us program officers, we have been involved in a lot of these discussions in the development of these interoperability tools. A number of program directors and at GRI have been working, been part of working groups and so on to make sure that these tools are developed properly and that we can adopt them whenever they are ready. So what's the vision for what happens after seven years in the best case scenario? If it's successful and everyone depends on this, is the idea to renew the contract at that point for another X years or would the idea be to internalize this among intramural staff or some other idea? So at the end of the seven years, we would like to do an assessment of what has worked, what has not worked, and overall assessment and which will occur annually during the process. And work, again, we look at this as working with the federated data commons model system and actually come to the conclusion of what should be the final step, next step in these for this iteration. What parts of this effort works? What parts doesn't work? Where do we actually need to focus our attention on as far as making this better? Because we know what Vivian has talked about, what Valentin has talked about is what you've heard across the energy in the community. We're still building some of these concepts out into, and we're gonna continue doing that. So after the end of the seven years, I can see us actually being able to reassess and look at what the community wants, make further adjustments, and do maybe another iteration with the federated commons are, let the community help us decide what's gonna be the best way to go forward. So I have one metric that might be too crazy broad to bring up, but why not? You're looking like you're ready for something. And that is, will there be an expectation that there'll be increasing numbers of, let's just say, extramural funded right now projects that will be cross-institute? That if you've got the capacity to have these interoperable data, big data, what I wanna say, focused on different kinds of diseases, but at the same time, in a lot of things that are interrelated, do you guys envision that the silos that the institutes now are in? Now, I recognize there's cross-funding sometimes. I understand that, but do you think that, do you envision a model of sort of investigation of human disease in a way that will, that this can really facilitate cross-institute funding of projects? It would that be pie in the sky metric, but it might be, okay? My reaction, I'm not sure it could be a metric that would be a term of the contract. I can imagine it'd be a metric for a term of whether we think this whole kind of line of development is desirable on that, could be something that we'd be very pleased to see, if that's what you meant by metric. I would agree with that. Valentina, I think you wore them out. So we've heard. I think Jay, who's been quite quiet, always worries me a little. I think this is, I think it's a great thing. I'm very positive about this.  And I'm just thinking about, one is trying to think in advance about how, defining, and this came up a little already, but defining success in advance, I mean, do you anticipate that 10%, 20%, 50% of extramural investigators will use this, you know, 100,000s, kind of trying to, before this gets going, try to have a sense in advance of how we're gonna say that this was successful relative to the investment that was put into it, right? And the second thing I was just wondering about is I could imagine this starting off and there being a temptation to issue RFA's where we require people to use it, right? And I'm just curious if there has been thought about, oh, we would never do that, or maybe we'll do that. And, because this, you know, if you do start requiring people to use it, then how well it goes will have enormous ramifications for everything else we're doing, right? So something to think about. Do you wanna take the, so the first one, the first question, so I have to take it from Vivian, make sure I understood what you were asking, is how do you determine the validity of the sandbox itself as to whether or not it should be continued or shut down? Is that... How do you measure success, basically, and what are the expectations for success in advance in terms of usership? Okay, so that is still something that's posh this in development, but I could give you an idea of a potential way to do this. We can look at, first within the ERP programs, we can look at surveys from the members who are using it and those who are not using it, and maybe they have their own, still have their own infrastructure, client infrastructure, and do a comparison analysis to see if what the sandbox is doing is actually fulfilling those goals. If you're on the outside of the ERP program, let's say an outside investigator, okay, we can track those individuals, you know, and I should be careful using the word track, but the individuals that are using that and send out surveys and on to see what they feel those efforts are, this fulfilling their needs and their goals. Another thing that we could also do is actually have, you know, annual meetings to determine, you know, feedback from the scientific community. There are multiple ways to determine the validity of this sandbox, both within the infrastructure itself and outside the infrastructure with the community. And so that's one way that we can determine whether we should scale up, scale down, or shut down in those terms. Now I forgot your second question. So now I may turn that to the Valentina for the second one. Yeah. Of course you have a few ambitious ones for it to require you to use it. Well, I- Well, of course Nick, it's accessible, but hurry up. I frown upon the use require, okay? We would like to encourage. So the DNCI Cloud pilots have used this to give incentives to people to use the resource, which is, you know, a little bit of funding to experiment with the environment, right? That worked. That got a lot of attention to one or two of the three pilots that they had in place. So we actually, we discussed it, proposing something similar to it, is actually part of this whole project. So the answer is yes. Planning to do that. I just want to make sure I understand the intent of this sort of at the grassroots level. At my institution, we have a bioinformatics core that will serve as people who do, for example, RNA-Seq and do their MATVAX form and that kind of thing. Is it your intent to serve as that bioinformatics core for those people to displace my institution's bioinformatics core? And which wouldn't be a bad thing, frankly. The intent is not to displace it. The intent is to complement it in case some of the bioinformatics core at your institution cannot deliver on some type of analysis for whatever reason. Too much data to process. The tools are not the latest version of the tools that people could use for whatever reason. It's really to complement it. I don't expect this to replace it at all because as was pointed out, there are some costs and in some cases people are not. I think your costs would be cheaper than going to a bioinformatics core. Right, but in some other situations to scale up and try to deliver some analysis results on large amount of samples and data, the bioinformatics core, and I'll be able to do it. I think what would benefit us, just listening to this discussion, is a few concrete use cases. So for instance, use case one, I am a researcher who has a few thousand genomes I have collected from my patients, say, or from my study at my university. Is this a use case? So I would upload all those data to the sandbox and then I would pay a staff member who works in this extramural entity to assemble a line against the reference and call variance and then give them back and then deposit those in a file in that same commons. Is that a use case you have in mind? That is one use case. The other use case could be you have a consortium that wants to work with the sandbox and they all generate terabytes of data. They need a place to consolidate the data in one place and to run one pipeline on all of these data sets. That's another use case. What's hard to envisage or to foresee here is the amount of staff time. I think computing time is one thing, but then for instance, our bioinformatics core, most of the expense, it's also very expensive, although effective, I would argue. But most of the resource is actually the bioinformatics analyst you're paying to do that and that's not gonna scale in the same way. It's a different scaling problem. Well, the bioinformatics analyst, it doesn't have to be a sandbox staff member as opposed to maybe the data managers of the data sources. We'll run their own pipelines and get the results and make sure that the results are shared across the centers. So it's not necessarily somebody that works at the sandbox as opposed to just making the environment for somebody else to run it. Park, you have a follow-up, right? Yeah, I think there's two levels of users here. There's the naive user who doesn't know how to do any of these analysis and they're probably not gonna wanna learn to do it. And that's the kind of guy who goes to my bioinformatics core. There's the more skilled user, like Jonathan I think is alluding to, who knows how to do it and is probably not gonna go to the sandbox if the costs are too high. So I think there's those two groups of users that are wildly different that are gonna have to be dealt with. And I'm not sure one operation can deal with both of those. But even the, even the, oh, I'm sorry. It's actually an answer now to both, I think from my perspective. I actually think that first for the, regardless which user it is, a lot of the reason that there's so much work going on into this, all these analysts and so on, is that things are done so in such a non-engineered way. Everyone, the data is not sharded right. The, everyone has to send their own pipeline from scratch to basically do the exact same thing again with their own code in their own place and to learn the best practices pipeline. They're not doing innovative work in this way. They're just doing the same thing again, again, again. They're moving the data around a hundred times and the data is getting bigger and bigger and bigger. At some point it won't be that movable for these types, for certain types of analysis, not for everything, not for everything any of us necessarily does, but for certain types of genomics data that's no longer going to remain a viable model. But on top of that, there's all this organizing and massaging and moving arounding that is not done in a best way. And even if it mimics the right analysis steps, it's not done in a very effective software engineering way. And so it's slow and it breaks a lot and it doesn't scale. And so I would say, I think it would reduce the amount of personnel required, not increase it. They would still be distributed people in different institutes and so on, but they would have something to work with that they don't have to build from scratch every time. And I think also for, but that's for Jonathan Moore to say for his own people, but I think for certain types of analysis done at scale by computational experts, at least I can talk about my own lab, at some point you don't want to do it for yourself every time because that's not where your core expertise is. You want to develop the next method, not to build the next infrastructure on which the method would run. I think it merits trying it out. It doesn't preclude everyone to still doing their own homegrown solutions, but that would presumably be the incentive for people to go in. So to me, I think that the rationale for this is great. I mean, I think that the concept makes a lot of sense and I think that the only thing that I'm concerned about really is whether the cost is going to be high enough that everybody's going to be doing the cost benefit analysis for themselves. You know, there's the time for the people in their labs, there's the cost of doing it locally versus in the cloud, which may be better for reasons you're saying. And so you're going to go through these calculations and I think for this to be a success, the costs have to be low enough that people will go through that and a lot of the time they'll say, yes, I'm going to take advantage of all the great things that are up in the cloud and the system instead of reinventing the wheel locally. And there's going to be some kind of cost point that will make an individual investigator flip one way or the other. Aviv. And the main issue with the cost is what Brenton said before. The actual cost could end up being cheaper, but most people live in weirdly subsidized worlds. And so they actually don't know the actual cost. I don't know the reason. And if we know it, we don't care. And if we don't know it, we quote unquote don't care. In fact, we sometimes end up actually paying for it, not just through indirect costs, but we don't keep like a mental accounting of it. So we don't really know what we paid for anything because it's distributed in all of these little pockets and bits and pieces and so on. So it feels cheap even though it's expensive. Here it would feel expensive. Even if possibly, I don't know, it will be cheap. But maybe they would give it everything for free, the cloud vendors. That would be nice. My Brent. Yeah, I just have a logistical question and part of my naivete. Since this would be funded through a contract mechanism, does it go through peer review and would it come back to council at some point down the road? It will go through a process called a technical review, which is what you're thinking of when you think of peer review. It will not come back to council. No contract comes back to council. Yeah, we cannot discuss funding plans with council. All right, I'm imposing the five second rule. I was foolish to suggest that she had worn you out, but I think we do need to come to a close. We've heard a lot of good comments here and the staff will bear this in mind. But first I need to call for a vote. Can I ask for a motion to vote? I'm gonna ask for a motion to vote to approve the concept. Can I get a second?  Thank you, opposed? Abstaining? Thank you. Wonderful, thank you all. Okay, moving right along. One of the working groups of council is the, excuse me, genomics and society working group. I believe there are nine or 10 members and Shanita, Gail, and Jeff. Look, they're all lined up there. These are our council members. Yeah, we clustered you together, right? Yeah. With his microphone off. Okay, and a requirement of any working group of council is an annual report. And Lisa Parker is here. She's the chair of the working group. Lisa is professor of human genetics and director of the masters of arts and bioethics program at the University of Pittsburgh. And she's gonna give you the report, the annual report.


Where do you have the facts about the killing of Eastern Ukrainian people? You sound pro-Russian? Eastern, it's... Man, look into... Look up Scott Ritter. Scott Ritter talks about it. Look up the Doran. Here, the Doran. Check, follow their work. Okay, seriously. The Doran do a search. Follow their work on rumble. They're on bitch-shoot. They're on you sensor tube as well. Check out their work and you'll get a good feel. It might take you a while, but like, pro-gamer, that's a fact. No one denies that. Right? It's just it's just not being covered on CBC, NBC, Fox and all this stuff, but Russian Ukrainians since the 2014 coup that was that overthrew a democratically elected government that was installed by NATO, by Western powers, especially United States, right? With John McCain going their hands on cookies with Newland and stuff like this, right? They put in the Nazis in power, right? And as soon as they got in, they basically said Ukraine is no longer a bilingual country. Russian is no longer going to be taught, right? The Eastern Ukrainians said, what the hell? What the fuck's going on? And then Crimea went for independence. Donbass region wanted autonomy.  NATO was arming Ukraine and they started killing off Russian speaking Ukrainians. Like, it's a fact. Like, it's not it's not anything that anyone is not is denying. Victor, Russia is the one bombing civilians on a day on a daily, not even targeting military sites, just straight out apart. No. Okay. Western Ukraine was bombing the Donbass region since 2014 15. Are we writing that off? We're saying no, it didn't happen. Everything started on February 24, 2022. Is that what we're saying?  We're going to we're going to say no, they didn't burn Russian speaking Ukrainians alive in Odessa.  Yikes. Yikes.