Good afternoon, I'm Adrienne Dix, BC's Minister of Health, beside me is Dr. Bonnery Henry, our provincial health officer for British Columbia. I just want to let members of the media and the public know that we will be doing a briefing tomorrow Saturday. It'll be here and it'll be at noon. I want to acknowledge the territories of the Musqueam of the Squamish of the Slewa Tooth, where we're giving this briefing today and I wanted to introduce our provincial health officer, Dr. Bonnery Henry. Thank you and good afternoon. So today we are announcing that we have 77 new cases tested here in British Columbia for a total of 348 people. This includes 200 in Vancouver Coastal Health, 95 in the Fraser Health region, 30 in Vancouver Island, 19 in the Interior Health region, and four in the Northern Health region. Of those, 22 people have been hospitalized or in hospital at the moment. 10 of those people are in ICU and six have now recovered fully from this disease. I did want to talk as well. We have a couple of long-term care facility outbreaks, including in our numbers, a new health care worker who's identified, associated with the Dufferin Care Centre and that centre is now being managed as an outbreak, as we do whenever we detect a case in a health care facility. So Fraser Health is addressing that issue today. In talking about health care workers, I think that is something that is always in the back of our minds as we've been going through this last number of weeks and I do want to say that we have had a number of health care workers who have been infected with this virus and today I really want to talk about my colleagues. I know how challenging it is when we are dealing with a new virus, a virus particularly for which we have no cure, we have no vaccine, we have no way of preventing it and we know that it can cause quite severe illness in people that we love and people that we care for and people that we're close to and it's been a tremendous challenge for our health care sector to watch what has been happening over those last few weeks and months around the world and I have been watching with them very very carefully and looking at what are the things that we need to do to best protect our health care workers and our health care sector here and I want my and so much of course in the beginning is very unclear but I do want to say we do know so much more now about this virus here and we know that hundreds of people have been safely cared for by our health care worker teams here in Vancouver, here in BC, here in Canada over the last few months and we know that they're safely cared for because of the dedication and the intense practice that we have in our health care workers system. We know as well that about two dozen of our health care workers here in British Columbia have been affected with this virus and I'm relieved and happy to say that none of them have had severe illness all of them have had very mild illness and there have been some very short hospitalizations but for the most part people have been managed at home and that is something that we're grateful for. It is everything that we do in the health care work in the health care setting that keeps us safe from this virus and these are things that I've talked with with my colleagues many times over the last number of weeks. It's identifying cases early and we've seen that around the world. It's when we have unrecognized cases that are in a hospital or in a care home that it gets transmitted and that's when people can get sick and that's when it gets passed on to others. It's making sure that we have administrative processes in place so we minimize the people that are in contact with somebody who's sick. We make sure that as soon as you go into the hospital and we talk about this a lot about the importance of calling ahead and as soon as you go in with the respiratory symptoms you're given a mask to prevent you from putting your droplets out there, prevent other people from being exposed. It's the engineering things like putting someone in a single room like making sure we have the right air exchanges in our hospitals and our settings and of course it's the personal protective equipment that health care workers wear that protect them as well and I know there's been a lot of angst and a lot of concern that we're not going to have the supplies that we need to best protect our health care workers across the spectrum whether it's the pre-hospital care in home care whether it's our community physicians whether it's our hospitals or our long-term care homes and I want to say that right now we have been working on this across the province and across the country and we do have the supplies that we need right now and we are committed to doing everything everything that we can to make sure that we continue to have what we need to best protect everybody in our health care system right now and that's my personal commitment as well. Finally I do want to say a little bit about the fact that you know we we have had these social distancing measures in place we've had gatherings that have to be curtailed and but we do need to we do need to make sure that we still have the resources that are available to support us we need businesses to support our health care workers in particular make sure we have food make sure we have child care and details around our child care strategy to support our essential workers will be coming out very shortly but we also need to make sure that we that there is appropriate precautions that are taken in certain settings to make sure that we are not allowing settings that allow transmission of this virus and we go back to the reasons that we're doing this the reason that we're doing this is so that we can build a firewall we can prevent transmission of this virus between us so that we can prevent the people who are going that are the most susceptible to having severe illness or dying from this disease from being exposed to it in the first place and that's the reason why we're doing all of these very extreme measures in many places and it's become clear to me that there are certain settings where this is really a challenge and we talked earlier about about restaurants and we put in some guidance around being able to to maintain a distance within the restaurant and it is becoming obvious as it has too many of the municipalities around here that this is a very challenging thing to do so from today my order is that restaurants must move to a take-out or delivery model only as a way to best protect us and to ensure that we can maintain those distances I also say that you know for many businesses they can and should stay open to support us but they need to have the important social distancing measures in place to ensure there's not that many people in the office environment for example there's not face-to-face meetings there's not congregation in groups it's also important and I've talked about this for us to go outside but like we do when we're inside we need to go outside with our close family with our small groups we're not to be outside in groups we're not to be out playing basketball we're not to be out sitting together in large groups on the beach watching the beautiful sunsets that we have because when we're out in groups like that the chances are that somebody will be exposed to this virus and then we'll bring it home to our family and our communities and that is what is going to spread the virus so right now and again this is not forever but right now we need to be in small groups we need to be with our family we need to be with our close friends one or two of us and we need to maintain our distance with others you know we must be united in these efforts and we need to build that firewall and make sure that we keep it as strong as we can and that there are no gaps thank you very much Dr Henry and I wanted to just lay out a couple of things that have changed and some new information that people have been asking for that I think people will be interested in this Dr Henry has said 77 new cases today for a total of 348 province wide to put that in context 22 of those cases are in acute care that's compared to or in a hospital compared to 17 yesterday and 10 are in ICU compared to nine yesterday and I just want to put in context for a moment two weeks ago the premier and Dr Henry and I laid out our pandemic preparedness plan we had been at stage one and dealing with acute care hospitals in terms of preparing them and opening up space and this Monday we went to stage two with the cancellation of non-urgent elective surgery I just want to briefly report out how significant these actions have been and what they've meant and what they mean in the system prior to the COVID-19 response we averaged about 103.5 capacity in our acute care system that capacity as of now our snapshot as of today is 78.5 percent that's a change in occupancy in a very short period and in particular this week of 25 percent or 2,398 beds in addition our critical care beds are at 61.8 capacity and I just want to put this in context because there's a lot of discussion of what people are doing in terms of preparing but I think in terms of preparedness as I said 22 patients currently in acute care just to put it in context this preparedness is not what happened in other jurisdictions and this shows how much we've learned from other jurisdictions we've acted and our actions have had consequences for a lot of people as you can see in these in these bed numbers but we've acted now to ensure that our acute care sector that our hospitals are more prepared for a possible surge or influx of patients and this has required extraordinary effort by everyone in the acute care sector and I wish to thank them we released and published the this morning I believe on the bccdc website the testing numbers this week which I believe are 17,912 I want to echo what Dr Henry said about the health care sector about health care workers that we must do everything possible to support our frontline workers and hospitals long-term care the entire healthcare sector to give them the confidence that they can do their jobs with the lowest possible risk of their health and well-being frontline staff are critical to beating this virus they are critical to beating this virus I know all British Columbians support the school and I want to say this directly to doctors to nurses to carers to those who keep our hospitals clean and safe to food support workers and everyone else on the front line we are grateful we are enormously grateful for all your efforts we're making headway together against this threat but we're far from prevailing everybody knows that but we also know Dr Henry has said it that you're performing miracles every day I want to make it as clear as I can we're going to spare no effort to ensure that you have the equipment you need to get the job done including the equipment to keep you safe we do have supplies in place and we're identifying more we will of course respect your collective agreements and do what it what it takes to provide the additional support child care and extra accommodation to help you work effectively we'll track down and punish anyone caught misappropriating the better word is stealing the supplies we need for your protection but our health care system doesn't exist in a vacuum it needs support from every part of society and every part of the economy and those essential services must continue they cannot be shut down they're equally critical to beating this virus of course we know when we've said it and everybody understands how important care aides and nurses and doctors are but so are pilots flying test kits to labs early education workers caring for our health workers kids grocery workers making sure we're all fed and longshoremen moving critical supplies across the dock we are grateful for your work too it must continue under safe conditions and we're taking additional steps to clarify the conditions under which important work can continue both in health care and the wider economy we'll have more to say on this shortly but we'll be calling on employers unions and worksite to develop the protocols under the supervision of Dr Henry to give those working in the economy the confidence they are doing so safely of course in addition to this and Dr Henry has made it clear with respect to restaurants and I've had occasion to talk to a number of mayors including the mayor of Vancouver today clarity about restaurants and the fact that they can only operate on their take-out basis I want to say this there's lots of discussion about all of the of the group of people that may not be following Dr Henry's orders it's our expectation that we all do so it's what we owe to one another and we must continue to do that I want to say a couple of extra things yesterday we convened the Isabelle McKenzie the seniors advocate staff of the Ministry of Health and MLAs of all parties including Shirley Bond and John Yap who are liberal members including Rona Ray Leonard and Janet Rutledge who are NDP members and Sonia First and O the Green Party House Leader to work together to build on the groundswell of community goodwill that is out there to support seniors they've come back and reported with recommendations working and using in the lines with better at home in the united way to make light to support seniors who are currently dealing with the consequences of social isolation necessary to protect their health we'll have more to say about this on Monday but I want to thank all the members of that committee and the actions that they have recommended are being put into into motion finally I just want to say we are all in this together and we're today we're going to find more ways to be all in in the fight and tomorrow we're going to find more and we need everyone to join us we've said it before but we need to say it again and again and again not just in settings like this but with to one another that that we have we know because Dr Henry has laid it out some of the toughest measures any anyone has seen that are required for all of us to keep each other safe some of it is simple soap and sleeve and social distance and self-isolate and stay home when you're sick and support seniors and elders and those who are vulnerable but starting now there's no longer any room for almost or pretty good or nearly enough starting now our individual fight our collective fight must be a hundred percent all of the time and a hundred percent of the effort and a hundred percent right I know it's a lot to ask of a lot of people I know people don't get the information all at the same time but we need to give our health care workers our health care professionals Dr Henry a whole team of people around the province who are preparing to deal with the health consequences of this virus we need to give them the flatten curve they need to provide the continued service and the continued care that will help everyone in the province right now and every moment from here on going all in on COVID-19 will save lives and the and our own lives and those of the ones we love it's just that simple the Vancouver coastal 85 a Fraser 30 the Vancouver 19 the total the cut the COVID-19 so this intensive patients we're ready to take your questions. Thank you minister Dix.


This team's had a lack of better term concentration issues when they faced some short-handed squads. I mean, the Lakers obviously without Ron, Sacramento we know, obviously with Pat and Philly. What, I guess, was your message to them today that they needed to start taking teams seriously, no matter who was messing with them? Yeah, it really was a standard that we have to have for each other. And that part of it is, you know, we talked a little bit about it is finishing plays. You didn't see as many breakdowns at the end of play tonight, the concentration level.  And I thought from the beginning of the game, you saw that a lot more talk from each other, a lot more communication, a lot more demand of each other, which was great to see also. Congratulations, and it claps today, even last season. For me seeing him, when we first drafted him to now, I think the ability to play on a nightly basis, to get your mind and your body ready to, right now he's starting for us. And so the matchups are different. Tonight you have a longer athletic guide at your garden. A few nights ago you might have been guarding a bigger guy that is 40 pounds stronger than you.  We see him get better at it. Talk about the demanding of their teamies. I mean, was Kevin an example of that? I mean, it looked like he was explaining to Joe how he wanted something time-consensual. Yeah, which was great. He's a good force.  And let's talk about some of these scenarios that are happening. And because we want that to continue on to the court. And so really trying to get this group to open up demand of each other. Talk out there on the floor. It's a huge part of us being better defensively. When the team hold each other accountable, that's all, that's what you want. What way is going to be confidence-boosting and encouraging that Kevin can only take 10 shots or 12 points and he's still going to win? And he feels totally, you know, great about it after the game, which is great to see him in the locker room excited that, you know, multiple people in double digits. He always says that's team basketball right there. And that was a prime example tonight. Coach, do you think that right now, especially with these last games from Ben Simmons, Kevin and Durant are reaching together the expected level from three to one? We hope so. You know, it's still early for this group. They really haven't played together. And, you know, Ben, as you see, is just feeling better physically, which is helping us, you know, for him to go coast to coast and get easy basket for us to push the pace for us to get open shots for everybody else. And they're learning how to play with him and play with his pace. So hopefully we're growing. Those combinations going to change as we get healthier and they'll continue to play better with each other. I don't know. Do you think a lot of times, Simmons has been criticized for his lack of shooting? In his case, in that case, with Irving and Durant, do you think that this is even better for you that Ben Simmons doesn't take a lot of shots? I think it's a luxury. Most people in basketball want to score the basketball all the time and thinking about scoring. You know, you got a guy that is not thinking about scoring, but making his teammates better majority of the time that he's on the floor. That's a luxury. He really enjoys getting other people involved. And when we can surround shooting with him, pretty tough to guard. What's quick for him since L.A.? I think it's really the body piece really coming into the game. And I think he was at maybe 14, 15 minutes at halftime and he was telling me to put him back in the last three minutes of the half. And I said, just hold off. But he said, I feel good. And then when he's telling me he feels good, I think he's.


A lot better than last night. I felt like we came out really strong. We were able to not make a lot of hitting areas, credit to Weidner. I thought they played some fantastic defense, which made for some great rallies. Defensively, I thought we played really well and I was really happy. We hit on all cylinders. All our hitters hit very well tonight and it just seemed like no matter who we had out there, they did a great job. Well, first off, it's a team sport and the girls we got in tonight are pushing to get on the court. So, even though, yeah, it was the last 15 points of that match, what we see on the court matters as to our decisions of what we make in the future because that's game experience. And tonight, it seemed like whoever went in there was able to come up with some great plays. I mean, Jesse Freeman coming across and picking up a tip, you know, coming out of nowhere and Sarah Brigham got right in there and passed a couple of great balls and it was great to see because that's how our practices are. Phenomenal, if you came to our practice, you could know who started and who doesn't start. It does, but it's a great problem because as you can see, the last two nights, all setters play the same amount. They've all played now three sets. Yes, everybody played three sets in the last two nights, so they have to be ready. All three have to be ready for us to play. The other teams know I have no idea who we're going to throw at them and so they're not going to stay sharp unless they get on the court and play. Well, I think they just spilled a little bit of momentum going into Saturday. They know that they are looking forward to Saturday because it's going to be a great match. Both matches are going to be really good. We're getting two very good teams on our home court. We'll put ourselves in position to have a phenomenal season if we can do well on Saturday.


So welcome everybody. This is one of the sessions in the block that was organized by the Federation of American scientists, and in response to their call for policy recommendations in terms of what the federal government should be doing for promoting open science. And I am getting a couple of echo notes, so I will do my best to fix that. But hopefully it's working okay for most folks. It is my pleasure to introduce our the cohost and the panels for this session. This session is about promoting transparency and rigor in federally supported research through open science practices. And we've got four great folks here to share their perspectives on that question. Going first will be Jessica Ponca. She's the open science program director at the Stara Institute. Thank you Jessica for participating, of course. Brendan Nyan, he's the James O. Friedman Presidential Professor in the Department of Government at Dartmouth College. Brendan, Wave. And then we've got Sean Grant, who's a professor in the Head Co Institute for Evidence-Based Educational Practice at the University of Oregon. Sean is also the chair of the Top Guidelines, the Transparency and Openness Promotion Guidelines Advisory Board that we support and encourage at the Center for Open Science. And then finally, here's truly welcome everybody. My name is David Meller. I am the director of policy at the Center for Open Science. And I'll be sharing my submission to the FAS Open Science Policy Sprint last. But without any further ado, I'm going to stop sharing my screen. I'm going to pass it over to Jessica. Thank you so much, David. I really appreciate the introduction. It's great to be here with all of you. While I am now at Stara, I'm going to talk today about some work that I did while I was executive director of ASAP Bio. And the title of this memo that I'll tell you about is Make Publishing More Efficient and Equitable by Supporting a Published Then Review Model. So as I'm sure everyone on the call is very familiar with the Nelson Memo, which came out in August of 2022 and called for all federally funded research to be openly available without embargo, removing the 12 month embargo imposed by the Holdren Memo and also allowing for reasonable publication costs and costs associated with submission, curation, data management and article publishing. So the Nelson Memo was a huge step forward for open access in the United States and by extension globally. But I think that it raises the question of what are reasonable costs. This is not a term that is clearly defined within the memo. Although they do note that American taxpayers are already paying an estimated $390 to almost $800 million annually for the cost of publishing federally funded research. So article processing charges or a charge levied by a journal for publishing an article has been found to be increasing and one study found a 50% increase over the decade from 2010 to 2019. And APCs vary widely, but they can cost up to over $12,000 for the publication of a single article. But costs in terms of APCs are not the only barrier to publishing. They only cost of publishing. The traditional publishing system also costs time. So not only are papers kept sort of in the dark for months or perhaps even years before they're actually published as a final, there it is, a final peer reviewed paper. This process that sometimes involves repeated submission to more than one journal, repeated rounds of review, repeated rejection, repeated requests for review, has been estimated to cause up to 15 million hours of wasted researcher time in duplicated peer review. But there is a way to help make research immediately open. So preprints, which are manuscripts posted online without formal peer review can make research immediately openly available. And in the past and in many disciplines, for example, physics and economics, preprints or working papers have been used extensively to share research prior to peer review and they allow for community feedback ideas and discussion while the paper is under review. But there's another way to use preprints, which is as the foundation to reimagine a publishing workflow where instead of reviewing and then publishing, you publish and then you review. So after posting on a preprint server, there are services which will conduct peer review and allow the author to update and revise the paper in the open. And then this results in a publicly available peer reviewed paper with all of this work happening in the open. So what are the benefits of this? Why would anyone want to do this? The first is really that research, not only the original version of the manuscript, but also that peer reviewed version, the reviewer's opinions, reviewer's thoughts, that is available sooner. And this makes peer review more efficient and rigorous. It is more open to scrutiny. It's easier to determine what actually happened in the peer review process. Experts are able to weigh in even if they haven't been invited to participate directly in the formal peer review. And it's more equitable because people who might not be able to be invited to peer review are also able to bring their expertise to the table. Also, taxpayer research dollars are used more efficiently because I think like the very high APCs that I mentioned, I think are driven by the prestige signaling that journals provide. But by unbundling the functions of coordination of peer review, of archiving and publishing and distributing these tasks to open infrastructures, researchers are better able to make decisions based on the services rendered. So preprint usage is growing in the life sciences, for example, and in many other disciplines as well. This plot from European C shows the cumulative number of preprints as well as the number of preprints posted on a monthly basis. Preprint review is growing too, although at a much smaller scale. This is really in its infancy, but there are already several services that provide open review of preprints. So where do we go from here? The recommendations in the memo are first that Congress should commission a NASA report on the benefits, risks and projected costs to American taxpayers of supporting alternative scholarly publishing approaches, including the publish and review infrastructure. OSCP should coordinate support for key disciplinary infrastructures and peer review service providers and also draft a policy that federal agencies can consider adopting. And those individual funding agencies should follow the lead of the NIH, which in 2017 put out a guide notice indicating that preprints could be cited anywhere journal articles are cited in grant applications and reports. And they should also extend these provisions to include publicly accessible reviews. So thank you for your time. I've put a link up here to the original memo and these slides can be found on the OSF project for the conference. Thank you. Thank you Jessica so much. Most questions are going to be held until the end unless there are any clarifying questions folks would like to ask at this time. If there are not, what we'll do is move on to the next speaker and then start some discussion and Q&A after the last speaker. Seeing none, I'm going to introduce Brendan to talk about promoting reproducible research in federal funding, Brendan. Thanks, David. I do not have slides. This is a pretty short talk. I'm sorry. It's refreshing. I've got slides so I can make that joke. Oh, got it. Okay. All right. So I'm going to be talking about the white paper that I produce as part of this open science policy sprint that is intended to help describe a pathway towards the federal government better incentivizing reproducible research practices. That turns out to be a complicated task. So I'm going to lay out the motivation for this effort. The evidence that I think justifies moving in this direction and then what steps might be taken. So obviously I think everyone here shares the notion that both it's both true that science is generating important actionable insights that improve the quality of life for people in this country around the world. From new treatments for diseases like melanoma to the behavioral science that's helping inform the response to COVID-19. But we're still seeing challenges with promoting reproducible rigorous credible research given the set of incentives that scientists face. And the evidence from this I think can be found from many different sources. The Center for Open Science obviously is one of the key players in this space and has contributed a lot of knowledge. But without recapitulating that entire debate, I think it's fair to say we're not getting as much value for our scientific research dollar as we could. Under the current circumstances that the current funding models do not adequately incentivize in my view the use of reproducible research practices. Most notably but not limited to pre-registration and that has in many cases led to research that is not reproducible, not just in the social sciences where I do my research, but in the natural sciences and clinical medicine as well. And so to the extent that what I just stated is true, we're losing valuable important scientific insights that might be gained for the same amount of money that we're currently spending. If we were better incentivizing these kinds of practices which would in turn help us generate more rigorous replicable results that other researchers could build on and generate all the kinds of auxiliary benefits that we would hope science would create. Now the good news is, again, I think as people on this call are well aware that the federal government has started to take more seriously the notion of promoting open science, of course, the Nelson memo was referenced. There are many agencies and other folks in the government who are thinking about these issues. There are a number of reports and studies from different parts of the government or associated institutions like the national academies, the government accountability office, etc. Getting into these issues, but excuse me, but we haven't yet changed as much of how we administer scientific research funding and deliver it to encourage these practices as we might. The most notable change in this direction is the establishment of clinicaltrials.gov and then the requirement for registration in it, but we've still found downstream problems even in that area. So there's clearly a lot more to do to extend those kinds of models across more of the scientific research funding that's being done at the federal level and to within each field think about what the best way is to promote these kinds of practices. So what the white paper lays out is a set of steps that could be taken to start a set of pilot initiatives that could be used to start this process of thinking about how scientific research funding could be, could better encourage the use of reproducible rigorous research practices. In whatever way makes the most sense, given the topic at hand, the research methodologies in use, etc. That's obviously going to vary and it's going to be important not to use a one size fits all model, but I think we shouldn't. It would be easy to underestimate the role the federal government could play in changing the norms and incentives on this issue. It's very hard to be successful in an area when other people aren't using these kinds of practices and you are. It's making the publishing and grant funding landscape more difficult in an asymmetric way. Whereas in fields where the federal government is a key or the key funder, you're now creating a more level playing field where everyone is encouraged or incentivized to use these kinds of practices. So I think it could be really powerful. So what are these steps? First, we propose creating discipline specific offices that would develop initiatives to promote rigorous and respond and reproducible science within specific NIH institutes or NSF directorates. So again, there's not going to be a single model that's the right one here, but this would be a kind of starting point within a particular area of research to think about how to promote these kinds of practices. Then the next step would be to learn in that area about what kinds of practices are being used, how often people receiving federal funds are employing them, and what approaches or models or incentives might better encourage funded researchers to use these going forward. So that could be in the criteria for the funding solicitation. It could be in how applications or the questions asked in funding applications. It could be in the evaluation process. And then one final thing is it could even be downstream in terms of post-branting reproducibility studies. That's something I'll talk about in a moment. So there's a lot of different steps at which the office or institute in question could be thinking about how to encourage these practices. So what I hope they could do is develop some of these ideas about what might be tested and then conduct some studies prospectively and retrospectively within their area about how to incentivize and reward these open science practices in their field. The final idea I alluded to before would be to do a pilot study or set of studies in which the federal government was actually partnering with an independent third party organization like the Institute for Replication to commission replication studies. And this could be both as a measurement tool to learn about what percentage of research is being funded is reproducible, but it also could be implemented prospectively in a way that was communicated to the grant funding recipients. And that would in turn hopefully encourage them to use reproducible workflows and approaches knowing that their work would be replicated at least with some probability by some third party agency. So I'll stop there. There's a lot to dig into. I hope we can talk about it in the Q&A. And of course I defer to the expertise of the folks in the room who know more than I do about the particulars of how to implement this in that process. But I think I think there's a lot we could do and I hope we start moving in that direction. Thanks. Thanks, Britton. I was taking some notes and I've got several questions that I'm going to hold off until the end because I think they'll open the door for a lot of good discussion and I want to make sure we've got time for everybody. But before I pass the baton to Sean, I'm going to unpin you and just ask if there's any clarifying questions that folks had before we moved to the next speaker. Seeing none, I'm going to give it over to Sean Grant. Great. Thanks so much, David. And thank you everyone for taking the time to attend this presentation today. It really means a lot. I know we're all very busy folks. So I put in the chat, Michael Leonard. Yeah, I don't want to finish this one today. I mean, this one is just my copy. Yeah, if everyone wants you to mute. The participant list shows who's unmuted. I think it's, yeah. Perfect. Well, thanks everyone for attending today. I've put in the chat the link to the piece that I wrote for the Federation American scientists series on open science and policy. Mind folks is on incorporating open science standards and to the identification of evidence based social programs. And essentially the motivating question for this presentation is whether closed science makes it too easy for social programs to be considered evidence based by federal state local governments and nonprofit agencies. And the background of this in short is that evidence based policy uses peer reviewed research to identify programs that effectively address important societal issues across various areas of domestic social policy. And there are several agencies in the federal government in particular that run these groups called evidence clearing houses that review and assess the quality of research published in peer reviewed scientific journals. And they use those assessments to identify programs that they think have evidence of effectiveness and therefore are eligible for literally billions of dollars of funding every year. But the replication crisis in the social and behavioral sciences raises concerns that research publications may contain a higher if not alarming rate of false positives rather than true effects. In part due to the selective reporting of positive results that closed science makes possible. So even in studies that meet standards of evidence related to internal validity like randomized experiments with low participation nutrition. The ability to conduct undisclosed multiple hypothesis testing and then selectively report positive results remain a plausible alternative explanation for the results in a study rather than true intervention effectiveness which is the assumption underlying these evidence based funding models. So the use of open science practices like study registration and availability of data and code. I think an engender trust that studies provide valid information to decision makers, but these practices aren't currently collected or incorporated into these assessments of research evidence. So the recommendation from this presentation to rectify this issue is for clearing houses to incorporate these open science practices into their standards and procedures that they use to identify evidence based social programs. So let's dive into all of that a little bit deeper. So to start, as I mentioned, evidence based funding of social programs is becoming increasingly prominent. The federal government is increasingly prioritizing the curation of research evidence and the use of that curated evidence to make decisions about which policies and which social programs to support. And a key component of this evidence based policymaking is what sometimes called evidence based funding that either requires or incentivizes research evidence of the effectiveness of social programs in order for those programs to receive government funds in the real world. And these evidence based funding mechanisms typically prioritize interventions with evidence of effectiveness that are supported by randomized controlled trials or other research studies using causal inference methods. And the most major source of finding these randomized trials is the peer reviewed journal literature. And these evidence based funding mechanisms rely on again what I've called evidence clearing houses that review the literature. So in this effort, federal evidence clearing houses, these influential repositories that curate evidence on the effectiveness of programs are widely relied upon to assess whether programs across various policy sectors are truly evidence based. And these clearing houses generally follow a set of explicit standards and procedures codified manuals to assess whether published studies both use rigorous methods to assess effectiveness. And then a key thing for this presentation, they often require them to report statistically significant and positive results on outcomes of interest and require that there are not negative statistically significant results reported. As one key example, I work in a college of education. The Every Student Succeeds Act, also known as ESSA, is what is authorizing the federal funding of our K through 12 primary and secondary education systems. And that act directs states, districts and schools to implement programs that have research evidence of effectiveness when using federal funds for K through 12 public education. The What Works Clearing House, which is under the Department of Education, is the clearing house that identifies programs that meet these evidence based funding requirements of ESSA. And so consequently, the ratings from what works clearing house on whether a study or a program meets the highest or lower tiers of ESSA or whether it doesn't has the potential to influence the allocation of billions of dollars appropriated by the federal government for educational programs. And similar mechanisms exist in the Department of Health and Human Services and the Department of Justice, the Department of Labor. This really is something that cuts across any federal agency that provides funding to states for social programs. And the thing I want to focus on today is that this approach rests on assumption that peer-reviewed research is sufficiently credible to inform these important decisions about research allocation. And in particular is reported accurately, transparently and comprehensively enough for these clearing houses, these systematic review groups to distinguish which reported results represent true effects that are likely to replicate at scale. But anyone familiar with the meta science work over the last decade knows that there are concerns that published research can contain results that are wrong, that are exaggerated or are not replicable. And one of the main concerns is that this is due to closed scientific workflows that hinder reviewers and evaluators attempt to detect issues that negatively impact the validity of reported research findings, sometimes called questionable or detrimental research practices. The main culprits are selective reporting of results and the ability to try different ways of mining, analyzing the data, key hacking the data, and then selectively reporting results. I'm not saying that that's what everybody does. What I'm saying is that that's possible. We know it happens and there are no safeguards against it right now in the system. So it becomes a plausible alternative explanation for the results that you see in the journal article. And so we contend in our line of work that research transparency and openness can mitigate this risk of informing policy decisions based on false positives, rather than true effectiveness of these programs. These practices include things like registering your study, prospectively sharing your protocols and analysis plan, and releasing data and code required to reproduce results, which would allow third parties like journals and these evidence clearing houses to fully assess the credibility and replicability of research evidence. But in work that we've done looking at the standards and procedures of both clearing houses and the peer reviewed journals that they use to find these trials over 300 journals. By and large standards related to these practices are not existent in the journals themselves and really no clearing house requires any of these practices to meet even the lowest level of a promising practice, but alone in evidence based practice. So as a result, one concern we have is that not that it necessarily always happens but again it's unacceptably easy for interventions to seem evidence based if someone has the ability to selectively report positive results and that's never checked. It's not rewarded by the system that asks for only statistically significant positive results. There are even some great tongue and cheek papers that say how to make a program seem effective, even if it's not essentially how can one gain the system. As I mentioned, a lot of funding and prestige and career opportunities are tied to these clearing house ratings and publications. So there are structural senses in place that I think make this a very real concern. So our recommendations are for policymakers to enable clearing houses to incorporate open science into their standards and procedures that they use to identify evidence based social programs that are eligible for federal funding. And to do so they need to increase the funds appropriated to clearing house budgets to allow them to take on this extra work clearing houses should be applauded for the work they've done over the last two decades to raise rigor and standards. They've said in the socio-political context they're constrained by the resources they have. So this isn't a lack of desire on their part. It's an issue of opportunity and capability to do this kind of work. So one thing that could happen is dedicated funding appropriated by Congress and allocated by federal agencies to clearing house budgets so they can better incorporate this assessment of open science practices into their evaluation of public research. Funding should facilitate the hiring of additional personnel dedicated to collecting this data. And if open science practices were used, folks who have their FTE covered to assess the comprehensiveness of reporting, like checking publications on results with prospectively shared protocols, and doing computational reproducibility checks, rerunning analyses using the study data and code. We also think it would be helpful for a group like the Office of Management and Budget to establish a formal mechanism for federal agencies to come together and develop shared standards and procedures for their respective clearing houses on reviewing open science practices. As one example, one could create an interagency working group that's been done before for standards on internal validity to develop and implement updated standards of evidence that include the assessment of open science practices. We published an example of this in the Journal Prevention Science, and we think it'd be great for a federal group to take this on board and create an official top guidelines for evidence clearing houses or something of the sort. And then as funding standards and procedures are in place, federal agencies that sponsor clearing houses can create a roadmap where maybe perhaps folks are just starting with reporting whether or not these practices are done. And then as they become more normative over time, in part based to the several amazing initiatives being discussed at this conference, clearing houses can look to increasing the requirements for higher tiers of evidence. So for example, the thing that we say needs our highest tier of evidence is something that is comprehensively reported according to reporting guidelines for results manuscripts. Those are concurrent with perspective we shared protocols and analysis plans and the results are computational reproducible. And we think that if that's the case, we will have greater trust in evidence-based practices if we know that they have computationally reproducible results and have comprehensively reported details in study findings. So we really appreciate the time you've taken today to listen to this. We think that this momentum from the 2023 year of open science in the White House of 2022 year of evidence for action creates this unmatched opportunity for connecting these federal efforts to bolster the infrastructure for evidence-based decision making with federal evidence to advance open science. And so if you want to chat more, contact detail on the left. This is a QR link to the piece from the Federation for American Scientists. And I want to thank Arnold Ventures for funding the project, my co-principal investigator Evan Mayo Wilson, and all of our findings and materials for doing this kind of work are available on our website, trustinitiative.org. So thanks so much. And I look forward to your questions. Sean, thanks so much for sharing that. Likewise, I've got some discussion questions that I'm eager to start on, but I'm going to jump right into my own presentation right now. And we're running perfectly on time. So thank you to the speakers so far. And if you'll bear with me for one moment, I will share my screen. Right. So the gist of what I'd like to say today is that there are mechanisms to improve research through existing infrastructure that the federal government uses in the grant application process. And those are through the data management and sharing plans that are integral to the upcoming policy changes that OSTP has mandated. And I'll go into a couple of details about all of that and what those mechanisms are. I'm going to start with a precise definition of how the Office of Science and Technology Policy defines data and what the implications of that are, and then to describe a certain set of policies and practices that are going to be useful for making sure that the work that goes into those are as high quality as possible and are as verifiable as possible. And I'll wrap things up by giving a preliminary snapshot of what federal agencies are planning to do around the guidance coming out of the White House and some recommendations for improvement. So as Jessica mentioned at the beginning, and as I suspect many of you know, the US federal government is moving towards open science being led by the Office of Science and Technology Policy stating that all work conducted by or funded by federal agencies needs to have openly available outputs. Those are open access papers and that is open access to data and research outputs that result from that work. And the timing of that is going to be either by the end of the grant period or by publication of results from this work, whichever comes first in a strong statement to address well known problems with non-sharing of results or file drawing. A couple of key details here that I'd like to go into is about how the Office of Science and Technology Policy defines scientific data and what the implications of that are. So they define scientific data at the objects that need to be shared and freely and publicly accessible to the extent allowable by ethics and law and intellectual property rights. And when they state that data needs to be made available, what they're talking about is an expansion of prior guidance from OSTP, and in particular, I'll emphasize that the need to replicate and validate research findings. So for the purposes of this policy guidance, scientific data includes recorded factual material that the scientific community states as being of sufficient quality to validate and replicate research findings. And just to clarify what that does not mean, it's not lab notebooks, preliminary analyses, drafts of papers or peer reviews, and it doesn't include physical objects. But it does include all the other materials needed to, I'll say again, validate and replicate research findings. And I'm going to extrapolate from that and advocate for items that that is included under that strong definition and what federal agencies should do to make sure that that definition is applied uniformly. So what do they mean or what do I think they should mean when they're talking about items that are needed to validate and replicate research findings? Those are all the digital research objects used to create the study or to conduct the data analysis processes. This is the code that's used to analyze the results. This is the information that's collected during the or that's accessed during the portion of the research lifecycle that also includes metadata and descriptors that really describe the context under which the data set was collected. And it includes research protocols and assertions of when a study is going to be conducted. As others have mentioned, registration is the process of specifying that a study is going to take place. This is common in clinical sciences, but it provides an opportunity for defining when a study is going to take place. And it provides an opportunity to state that the protocol is going to be applied under a certain context. And I'm going to ask, please, if you're not talking to keep your microphone on mute just because there is sound that does come through sometimes. The top guidelines, as has been mentioned once or twice also is a policy framework that articulates items that are used to conduct research that can be shared and archived and disseminated for the purposes of reproducing analyses or replicating previously published work. The top guidelines includes all of those items that I mentioned before and a couple of others, but the key framework that the top guidelines provides is applicable across many different disciplines designed to be applicable to life sciences, to social sciences, and other fields. It's modular in the sense that these different practices are distinct elements that are used or conducted throughout the scientific process. And the top guidelines also provides a structure for improved practices over time. So it starts with recommendations for items such as data availability statements, disclosures of whether or not an open science practice took place. And it moves up into more stringent requirements or verification steps that these practices were done well. So how and what should the federal government do to improve data management and sharing plans? I'm glad you asked. Those DMSPs need to be part of the scored criteria of grant applications. Oftentimes they're required elements, but they're specifically excluded from any sort of peer review or evaluation of their grant. And as we might surmise, that does not provide a strong incentive to put a lot of thought or effort into those DMSPs. So they need to be considered as part of the core scientific process and treated as such and evaluated as such. So those are part of the scored criteria for funding applications. Researchers who are creating those DMSPs will put a more appropriate thought behind them. Those DMSPs need to be easily accessible and publicly available, along with other information that's shared on federal websites about federally funded work. Additionally, that information includes the title of the research, abstracts, affiliations, and the primary investigators who are conducting the work. But it doesn't include information such as those DMSPs. Now, there's lots of good reasons that many of the details of those grant applications are not publicly shared on websites because that does include proprietary information and cutting edge research that is important to keep confidential for a certain period of time. But those data management and sharing plans don't include such intellectual property. They include information about how and when research outputs are going to be made available. And that information itself deserves to be publicly shared so that readers or anyone who has a stake in how the research is being conducted knows where to go to find the outputs, to find the data, to find the preprints, to find the code that's resulting from that work. Those DMSPs should have structured templates to make them easy to read, make them easy to follow, and probably even more importantly, just make them easy to create. These are new practices for a lot of primary investigators, and especially when they're not scored, not a lot of that goes into them and will often just kind of reuse information that might be a little bit dated. And there are better ways to go about it by provided structured fields, dropdown menus with selections for when and where data are going to be made available. Those DMSPs should be updatable so that once data sets are available, the DOI can be put in there, putting in kind of a generic name of data repository about where the data will be shared. It might be all that's available at the grant application process, but of course those repositories can be large and it can be sort of difficult to find where the data are specifically in those repositories. So once they're updatable, they can be filled in as appropriate. And as I mentioned earlier, it should be very clear, especially in that structured DMSP that data means more than just a single CSV file. It means all the information needed to validate and replicate research findings. So everything that I mentioned before about materials, about code, about data and metadata, about protocols needs to be made available through those DMSPs. We've taken a preliminary look at what agencies are doing so far and there is room for improvement. So the first nine that came out are available on our website and we evaluated those based on the content that I just went over. Unfortunately, none of them stated that the DMSPs would be publicly posted, so I think there's most room for improvement there. Three of those nine did state that DMSPs would be reviewed as part of the regular grant review process. And so that represents a great opportunity for interagency cooperation or for learnings from agency to agency. It's obviously possible that that can be part of the scored and reviewed grant criteria because some agencies are doing that, but a majority of them so far are not doing that. And thankfully, several of them, but not all, did define data in a fairly expansive way. We're taking guidance from that OSTP memo stating that data is more than just the precise bits of information collected. It includes everything needed to again validate and replicate research findings. And so again, the other agencies can learn from those definitions to make sure that those are being applied along with the spirit that OSTP is guiding. Those responses are being collected and curated and disseminated online. So on our website, we have responses to requests for information from agencies as they come out with their preliminary plans. And alongside that information, there's complementary information for what publishers, funders and universities can do along these same lines. And with that, I will finally stop talking. I'm going to open up the floor for some discussion and for some Q&A. I'm going to stop sharing right now. I'm going to check to see if this is being, there's anything. Okay. So what I'm going to do right now is explain the Q&A process just to make sure it's clear that there was a couple of different channels that we have available to us. And then I'll take the prerogative of the moderator and get things going with a couple of questions. As you have questions, please use the Q&A feature. It's under the menu at the bottom of the page. Or if you'd like to speak, please use the raise hand feature. And let me just find my notes right here. I'm going to bring this back to the beginning and give a question for Jessica, if that's okay. So Jessica, get ready for the hot seat. Jessica, you're talking about the benefits that the preprint process can have for the scientific ecosystem. And of course, the benefits of pulling apart the publication from the review process. This is something that I have a strong opinion on and I fully support. But one of the most frequent questions that comes from this type of process is how you compare preprints to traditionally published peer reviewed papers, especially thinking about either members of the public or journalists, or as we're talking to policy makers to policy makers. How should they view preprints and the reviews that come on to them? How should they evaluate those as compared to traditionally peer reviewed publications? Yeah, it's a very good question. And I think that the COVID pandemic really gave us an opportunity to take a case study of how we're going to deal with information that's coming out that hasn't been traditionally peer reviewed. I do think that there is a danger for potentially misleading information to be shared, but preprints are certainly not alone in that. The best defense against information that the community disputes is to put up public reviews of that material. So just to provide an example, there were certainly some perhaps questionable studies shared early in the pandemic. But the application of comments and open reviews helped journalists put that into context and become aware of these other experts who had different opinions on the paper. And really, I think helped curtail the spread of that misinformation. There have been a number of studies about how preprints tend to change prior to formal peer review. Most papers, I will say, like the vast majority of papers do not have major changes to their conclusions between the first preprint and the final review. But that's not, I think, universally true. That's why I think it's very important. And we've recommended in this memo that the paper preprints be updated with each subsequent version so that there is an opportunity for researchers to share publicly what they're revising in response to the reviewer comments. I hope that answers the question. Yeah, I think that's one of the real, you know, key benefits that preprints have. They're simply less static as better evidence comes out. They are quickly updateable.  And I also, you know, look to see evidence of what's folks who compare preprints to published articles and how credible or reproducible findings are from one to the other. And I think there's a lot less difference than folks might might might might assume by giving a lot of a lot more credence to the peer review process than perhaps it's worth. And then can I ask you a question? Please. Thank you for your presentation. You were mentioning a lot of the open science practices that ORQ the National Institute for Neurological Disorders Office of Research Quality advocates for or promotes. Basically, you meant your memo you talk about registration and replication as two practices are those the two main practices that you think there's that they should be promoting and advocating or were you thinking about kind of all the open science practices or how would you rank a couple of these different best practices and what they should be spending their time promoting or creating policies around. Yeah, it's a good question. I value all the different practices that are on the table. But if we're trying to think about getting the greatest bang for our buck in terms of discovery, I think we need to think about the integrity of the research itself. So making the output, the papers more accessible, accessible sooner via preprints, etc., is valuable, but it doesn't necessarily change the scientific output. Similarly, having the data be available helps other people build on it. But if that data is generated through some non credible process again, we're not we're building on a foundation of sand in a way that I think is best avoided. And so the idea that that I'm promoting is that in areas where it's applicable pre registration can help us ensure that researchers are transparent about which hypotheses were truly tested in confirmatory way and which results that are being reported are exploratory or in some way unexpected. You know, maintaining that separation pre registration is the best tool we have to promote that and that will in turn I think, and it will help us see also when people are, when there are lots of file drawer nulls on particular topics. So in both those ways we can get a lot of value. The replication idea again, making public data available allows for replication the sense of you can download the data push the button and make the results come out. That's kind of baseline expectation, but I'd like to see, excuse me, replication in the sense I described that people know that with some probability their work will be reproduced downstream. And that in turn creates a kind of positive reputational incentive to be more careful upstream. So I really like to, you know, encourage the best incentives for researchers because right now they're off quite the opposite. I really agree with that. I'm going to ask one more question and I'm going to encourage it for everybody listening, both our panelists and listeners to use the raise hand fake function or to use the Q&A box for any discussion you'd like to have or any of these points. But Sean, you've been putting a lot of thought and effort and push it behind bringing higher standards to those evidence clearinghouses or essentially to the evidence used prior to making policy. Can you talk about the trade off a little bit between kind of, you know, the need to improve education, for example, versus the need for better evidence behind initiatives that are going to have money put to them. They often think of like the clinical trial world where there's an urgency to get patients to the best available promising interventions. But sometimes it's going to be that balance between false hope and getting to the most cutting edge treatments available. What's the status quo for how that's going right now and how do you see this balance evolving with higher expectations and standards. Yeah, thanks, David. I think it's a terribly important question and the things that keep me up at night working in this space about this on the face of it positive mood toward evidence based policymaking. There are one, the weaponization of this to justify gutting social welfare, basically like gutting the social safety net saying we have a positive things that meet high standards. And therefore that means why are we even doing this space and that just betrays a bad faith effort and a lack of understanding of the kind of relative evidence that trials provide on whatever's been tested. What makes it to the point of having the funding to be tested, etc. So that's one thing I think is always important to be mindful of. Walking back from that extreme, I think another one is the some of these are set up where they really mandate the use of practices on these clearinghouses to the point where you might be asking communities to abandon things that weren't the comparison in these trials with populations and settings and outcomes that weren't represented in this body of evidence. And then on top of that my concern is with evidence that isn't really that strong. I think if you're going to tell people to abandon what they're doing, it really needs to be rigorous and applicable evidence. I think the best ones out there, and this is what I'd love to explore more with this really see these as evidence building mechanisms. So by curating the evidence and then just being honest about what the high standards can or should be in the field and what meets those standards. We can see where we're at and then we can incorporate into these cured evidence mechanisms funding for promising practices. So not just evidence for delivering the service but building up the evaluation capacity for things that seem like they have a potential replicated scale. And give money to groups who deliver those interventions to those who have evaluation capacity to make the evidence base stronger over time. That's what excites me, but I think it's really important to be mindful of unintended negative consequences of which there are several in this space. Yeah, I think there's ample opportunity to rebalance towards finding better evidence and taking the sort of time and energy it takes to follow up with those kind of promising leads. Would any of the panelists like to ask questions to their fellow panelists or any of the members and I should double check. I think we've got six minutes, but we could also be putting up against the next session. So if there are any follow up questions or final thoughts that anyone would like to give, please do so. I want for you, David. Would you consider if it's an NIH funded study on a social program. So it's a someone's gotten an R01 to do a randomized trial on educational intervention or like a health prevention intervention. And the, as part of the grants developed intervention implementation materials like training manuals or scale success fidelity of intervention implementation. Do you think that falls under the Nelson Meadow as things needed to carry out the study and replicate results, or is that more in their list of things that they explicitly say don't count. I'm, I'm pretty sure it counts because it has a pretty expansive sort of definition about work that's supported by or conducted by the by the federal government so my understanding is that it will. I also don't want to speak on their behalf and spread any misinformation. So if anybody else has an opinion on that I'd be also happy to update my priors, as they say, hearing none. That's now the law of the land. So it's good to have that, that opinion is power. I was beginning at it. It has a quite an expansive definition in terms of not not just the type of grants that are traditionally thought of in the academic research space, but work conducted by federally employed scientists, work that's conducted as part of evaluations as long as there's not a compelling justification to, you know, keep the work confidential, which many times there are going to be exceptions to that. But the default expectation is that it falls under those policy guidelines. I can imagine ongoing assessments or evaluations. I could imagine carve outs coming for that on a case by case basis. I want to say one more time, thank you to our esteemed panelists for responding to the FAS open science policy sprint, putting your thoughts and suggestions into writing. I think us getting those opinions in front of the audience that FAS has available to it is a step in the right direction. I think there's a lot of work to be done to make sure that that these opinions are heard and are echoed and are shared and this is obviously a part of that process. And so thank you for being a part of this session to work towards that mission. And for all of our audience, thank you for coming and participating. And you've got about three or four minutes I think until the next session starts. So go into the lobby, mingle with your friends and colleagues have a drink and go to the next session.


Let's talk about how to replace a jet engine of your system in flight. Today, we'll talk about zero downtime migrations and how we can make that happen. I'm my solo Greenberg. I work at Google on Google's infrastructure. My current project is Google Drive infrastructure. But today, we'll talk about the project that I work on as part of the development infrastructure. I worked on the distributed belt system. And we'll talk about how we did zero downtime migration for that. So let's take a brief walk through the history of belt systems and how one belt system can become a distributed belt system in a company. First of all, let's do a show of hands. How many of you work with MAKE, MAVEN, Grado, Rake? OK, I see a few hands. OK, yeah, more people. So these belt systems allow you to execute the build. So they automate the compilation and the build process of creating binary artifacts automatically. And they do this on a single machine, on a single desktop. And the system I'll talk about today is a distributed belt system. Your company may not have the business need to use this distributed belt system, but generally, every belt system goes through a similar evolution of the following. So we'll start out with a project where one engineer is working on it and they work on some project that they need to build. And so they use the belt system on their machine. And the belt system will detect many dependencies, some of which could be stored somewhere on the cloud. So say, our project depends on TensorFlow. So the belt system will be able to download this. Now as the project gets more and more popular and a whole team is assigned to add more and more features to it, the team will have to set aside a machine and use the build system on that separate machine to be able to execute the builds and build their binaries, test them, and deploy to production. Now a company may have many different teams. And if it's a machine learning company, then maybe every single team will have a dependency on a machine learning library like TensorFlow. And so now every team will have to set aside its machine and which will execute the build. So we run the test, build the binary artifacts, and then deploy them. And so as the team grows and as the company grows and there are many different teams that have to do this, what they have is they have an overhead where they need to allocate the time and energy and resources to maintain that separate machine on which they do the build and test. And to be able to make sure that it's up to date running and that if there are any bugs, they can fix them. And so now this takes away time from their primary project, which would be their machine learning project in this example. And so this is where the team like my distributed build system team comes into play, where now instead of having every team set aside their own machine and have to worry about maintaining that, we'll create a shared infrastructure. Where now the team will maintain the shared infrastructure all of the machines and we'll ensure that the user interface, user experience is the best possible. And we'll make sure that we add new features and we maintain the availability of the different teams in the company. So the system I'm talking about today, the one you saw in the boxes, it's called BuildRibbit and Google. And it's a distributed build system. It sits very nicely in the middle of the test and build stack. So continuous integration system will send its builds and its test request to BuildRibbit to be able to execute the builds and run the test on a remote machine. And this is similar to some of you might be using Jenkins, right? So this would be what the system does. Now release infrastructure will use BuildRibbit system to be able to build artifacts and then deploy them to production. Individual Google engineers and other teams will use BuildRibbit system if none of the existing infrastructure suits their needs. And integration testing infrastructure will use BuildRibbit system to be able to build all the different individual services then bring them up, then run integration test to ensure that the system as a whole works correctly. Now BuildRibbit system itself depends on many different components of the build and test stack. First of all, you need to figure out what the source code state is for a given user and for a given commit number. And so the source system will give us an overlay of the source code state. And then BuildRibbit will go into Blaze, which is the single desktop build system similar to Maven, Gradle, that would execute the build. And then it would put the build artifacts into storage. And then BuildRibbit will have a way to retrieve those artifacts and serve them back to the user. So Blaze was open sourced as Bazel. And so if you're interested in checking out, it was open sourced last year. And you're welcome to check it out by its name Bazel. OK, so as we see here, BuildRibbit system sits very nicely in the middle of the build and test stack. And so as you can see, because of the numerous number of people relying on it and lots of requests coming in, the luxury of downtime is not an option for us. We can't just ask our users to stop building and testing and stop releasing their code. And so for us to be able to turn down our system and then swap out for a new architecture and bring it back on, that's just not possible. So we need to do zero downtime migrations. And to give you a better sense of the scale I'm talking about, there's 30,000 engineers in over 40 offices that use the distributed build system. There is 45,000 commits that go through the system every day. So that's 15,000 commits made by other engineers. And then 30,000 commits made by robots that is automated scripts. So anything we can automate away, that would be a repetitive task for the developer, we will automate that away. And that would still go through our build system. The source code is two billion lines of code. And half of that code base changes daily. And so every time they change to the code base is issued in a commit, we'll rerun all the tests to ensure that all the integration still works and everything still works as expected. There is roughly, on the order of five million builds and test requests going through the distributed build system called build rabbit. And it produces petabytes of output artifacts. And so when the project started out, it was quite small. It started out as an experimental project. We didn't know exactly what the niche for it would be. We didn't know exactly how our users will be using it. But as the system became popular and finally found its use and a lot more engineers and teams started depending on it, we started hitting limits of our architecture. So we started out with something very beautiful like pagoda, very beautiful architecture. And it made a lot of sense for many, many years and it worked really well for us. And we were able to have this project be up and running and make it very useful to the company. Now as more and more users started using it and more people found it reliable enough for their use cases, then we started hitting the limitations where it no longer supported the metropolitan style of demand. So we needed to re-architect it. We needed to build it to be able to support the existing demands of thousands of engineers as well as the growth for the build system. And so for the remaining of the talk, I'll talk about how we were able to do the architectural upgrade and how we did that with zero downtime. Again, just as you can see here, Build Admin System sits in the middle of Build and Test Stack. It has a lot of dependencies on the build system components, on the various source system and the build system itself as well as it's used by many, many different infrastructure projects and individual Google engineers and teams. So this task really felt like we needed to replace a jet engine of our system in flight. And I tried to find pictures that would be able to demonstrate what the task was about. And apparently you can refuel an airplane in flight. So you see here is a helicopter refueling an airplane. But apparently we cannot, as a civilization, yet replace a jet engine of our system, of our airplane as it moves around. So I had to default to my drawing skills. And this is what it felt like trying to launch this with zero downtime. So we needed to keep the system up and running, keep the airplane in flight, and we needed to replace all the inner workings of it so it was a lot more powerful of a system. So let's talk about how we did migration with zero downtime. For the remaining of the talk, I will continue using the same example with the distributed build system. But the takeaways are the same for your system if and when you need to do zero downtime migrations because the lessons learned and the main considerations are still the same regardless of the system. So to make sure that we are on the same page about the task at hand, let's look at the old architecture and the new architecture and where the difficulties lie. So we started out with a very simple client server architecture where we distributed the client library to our users and now the client will send a request to the build-out of the scheduler which will just access the load balancer and then it would return the result for which of the workers our client should talk to. And then the client would send the request directly to the worker and get the result back from the worker directly. So now as you can see here, this works fine and it works fine for many years but we started hitting limitations when if the connection broke between the worker and the client, client lost all of the progress. It wasn't able to figure out if we started from some point in time that it made the progress so far and so it needed to restart the whole build. It resulted in a very thick client library where now the client had to understand a lot of the internals of how build system works and had to interpret it whether the build system had some sort of transient issue or whether the request is bad and the client needs to give a different request that would be able to succeed. So the client library was in charge of a lot of utilization and routing logic. And this was very difficult. It became very difficult as the use case grew as lots of users relied on this because that meant that a lot of the moving of the infrastructure and developing and evolving it to become a better, more resilient system was a lot more difficult for us. And so we converted something we call build service where now we had the same client library so we used the old facade, we switched the internals all of the how they build is processed by our system. And so the client would send the, will put the request onto the persistent queue. And then when the worker is available and has some capacity to execute the work it would take the request from the queue. Now there's two different types of outputs that is generally produced by the build system. One of them is build artifacts. So these are your binaries and another one is build progress information. So this is which test succeeded, which test failed. And so the build system will, the worker would output the build artifacts as it builds them and compiles them and puts them to storage, which are accessible then by the build artifact service. And also it would start streaming the results of the build progress information. So some of our users only ever cared about the build progress information. They only cared to know which of the test failed so they could go back to their source code and figure out how to fix it. And some of our clients needed the artifacts. And so they could choose which of the output information they wanted and request that specifically. So all of the green boxes are part of the build system. While we kept the orange box, the client library the same. So it was putting the side by side. There is the older architecture and the new architecture. And as you can see, it wasn't very easy to translate the outputs and the workflow through the system from one system to another. There's a lot more different boxes. The flow of the data through the system is now very different. And so we needed to figure out what it meant to transition it from the old architecture to the new architecture with no downtime. And so the first of all, what we needed to realize is that we knew what the old architecture looked like. We have been operating it and developing it for many, many years so we understood it very well. Now we understood what the new architecture is like because we built it specifically to solve a lot of the problems that we've encountered that we want to solve for our users. What we didn't understand, what was the intermediate steps that we had to take? As an example of the intermediate steps, if we wanted to migrate our architecture from using the scheduler to persistent queue, the way it request went is the client library sent the request to the scheduler, then it got the response back. Then it sent the request to the worker and then it got the response back about all the outputs of the build request. So now what happens here is we want to bring up the persistent queue and we want to start sending all of the new requests to the persistent queue. So once the persistent queue service is up and running and it's ready to accept the workload, we would stop sending the request to the scheduler and after a while, all the new requests will start going to the persistent queue. And the reason for this is because we want the system to be available for the users. So we want them to continue sending us requests. It's just we want to change how and where they send us requests. So now there were a few in-flight requests that went into the scheduler and so after they've completed, we'll stop receiving the responses from the scheduler. Now after a while, because we stopped hearing from the scheduler which worker they should talk to, the client library no longer needs to talk to the worker directly to send it to request. And as soon as the worker comes up, has capacity and is finished with the previous request, it can dequeue from the persistent queue. And now after a while, after all the in-flight requests that we're talking directly to the client library from the worker completed, we can stop that communication channel altogether. And now for the two different outputs is not shown here, but we had to come up with other ways to serve the request results to our users. And so I let the launch of the dequeuing aspect of the architecture. And it entails not only all of the steps that I mentioned, but also we work with five different teams in three different geographic zones across two different time zones. And so understanding very well what intermediate steps will we need to take in order for us to migrate to this new architecture was very important to us. Now we also needed to focus on ensuring that we can bring up our backhands and they can serve the new load and they can withstand the existing load. And so what we needed to do is first of all, focus on migrating of our backhands. We need to ensure that the new backhands, the new surfaces that we bring up are able to serve the existing workload well and do no worse than the old system and also ensure that we can scale it up for growth. Now it wouldn't have been possible without also understanding of how to decouple this thing. So not only do we need to migrate our backhands and figure out what the intermediate steps are, but we also needed to ensure that we can launch some of the architecture before all of the backhands are ready. And so we needed to figure out how we can break up the launch, how we can break up the migration so we could decouple the launches of services. And then that being split something along these lines where the built artifact service could be independently launched from the built progress information service. And then they took care and they persisted that they needed to be launched together. And so what this allowed us to do is first of all, not be blocked by any one of the services being not ready for the launch and still proceed without launch. And also what this allowed us to do is we needed to consider very carefully about what the system is going to look like if only one of the backhands is up, if all of the backhands are up or if partial backhands are up, if partial states of readiness and what that means for our system. And what this really meant is we needed to consider how our system would look like and write a lot of throwaway codes. The way I visualized it is every single different state, intermediate state that we had, it was like an island and then the throwaway code was like bridges connecting them. And so what we needed to do is we needed to consider all the different intermediate states that we would have and how they can be connected together with the throwaway code that is not necessary after we're done with the migration but it's essential for us to be able to do this without any downtime. And also as I mentioned earlier, we had a client library that we distributed to our users and we couldn't ask them to follow our release cycle. So we released weekly but at the whole point of having a shared infrastructure is that we didn't have to rely on our clients always having up-to-date clients. And so what that meant, we needed to support our clients months back so that they could focus on their work and we could proceed with making changes and evolving and improving our system. And so what that meant is that we needed to support months old clients. So we needed to consider what the state of the system was like months ago and what the state of the system is now and how to translate between the two. And so there was a lot of throwaway code involved but that ensured that our old clients that haven't rebuilt their client yet that haven't used the new client yet could continue using the system and get the results that they expected. Now we of course couldn't just turn off the system and then turn it back on or just turn on all the new code paths immediately for our users. So we needed to do an incremental rollout and this wouldn't be possible if we didn't have access to the system like Mesa's or what we use internally at Google Borg. So the incremental rollout looks something like the following, so we'll have two different zones and then instead of launching to all of them we'll do one by one first to one machine in one zone then to the rest of the machines in the zone then comparing the zone side by side. And so what this allowed us to do is we could utilize a control of individual machines. We could compare metrics side by side and see if we have worst latency through the new code paths or we're doing just fine. If our resource usage is a lot higher in the new code path and whether we can explain it or if it's a lot lower and that's what we want. And so it was very important for us to be able to launch to very specific places and control which of the zones, which of the machines are running the new code to be able to have a better confidence in our launch. This also meant that we needed to focus for which clients could benefit most. We needed help from our clients and so we asked them, we asked some of our clients who could dedicate their time and energy to help us verify that the launch works as intended and it solves their problems and we needed to find clients for whom the benefits of the new system would be highest and so it was worth it for them. And so it ended up being roughly along the boundaries of how backend was split, how backend launches was split. Where some of the clients, the client that I'll launch for the de-cure aspect so that's the persistent queue and the build-up in worker. That was our integration testing framework and continuous integration testing. Infrastructure and so we needed to make sure that we can work with them very closely to ensure the success of the launch and ensuring that it is what we are launching is correct and will work well for the rest of the clients. And so this also allowed us to isolate which clients are heard. So the higher risk clients, those who are working on with us directly, we could isolate the impact we have just on those clients from the rest of the clients by ensuring no user-visible impact for all the clients not participating in the initial launches. And in order to be able to roll it out incrementally, we also needed to have maximum visibility into our system. We needed not only access to best metrics, we could have to ensure that we can compare side by side the latencies, whatever metrics we used for user experience and the resource usage and the health of the system. We also needed to make sure that we had good access to all the different laws and we could tell whether the system is healthy and is up and running and is doing what it needs to or we could detect early whether something is going wrong before we affect lots of users and we can roll it back safely. And also having really good visibility to how our system is doing was essential in ensuring that we could debug it afterwards without having to reproduce it in production again. So if we were to think about the launch as a performance because we have one shot at doing it well once and getting it out to production, then it's very important that we practice that. And practice was the essential component of our successful launch for production. So as I mentioned for the intermediate steps, we needed to ensure that we understand all the different rollback steps. So as we send through the old code to our clients and then the new service comes up and if it starts failing to accept new requests coming in, we want to ensure that we understand what the rollback steps are to be able to roll it back quickly and efficiently. And this also meant that for every single intermediate step that we needed to take during our transition, we needed to understand well what the rollback steps would be in order to get back to the correct state and produce a few number of unwanted side effects. And if all goes well, then we could just proceed with the launch. Now, in order to be able to practice this, we found that checklists were very, very important and very helpful for us. There is a book by a former surgeon called Atokawande. The book is called Checklist Manifesto in which he talks about the value of having checklists. And the checklists are not there to educate us, to teach us something we didn't know already. The checklists are there to help us prevent the simplest mistakes. So when the surgeon or the nurse go into an operation, this is not for them to learn how to do the job, it's for them to prevent the simplest of mistakes. And it was helpful for us as well because as we were getting closer to the launch, a lot of us were tired and excited and it was a lot more likely that we would make some simple mistakes that would push the launch back into it and we couldn't launch it immediately. And so by preventing the simplest mistakes, we could get through the launch and make sure that we don't make silly mistakes along the way, we could just go on with the launch. Now, the checklists were also very helpful for organizational resilience. On the date of the launch, if somebody has to take off for family emergency, for lunch break, for the go on vacation, that didn't matter because anybody could step in. They didn't have to be trained or have to have practiced with us, but the checklists were detailed enough that it allowed us to be able to truncate this knowledge and quickly run up anyone who was stepping in to help us out. In order to be able to launch with zero downtime and no use of visible effects, key considerations that we needed to ensure we understand and we work on and we focus on and we ensure that we are prepared for. We needed to ensure that we understand all the different intermediate steps for our launch. So we needed to ensure that we understood how to get from our old architecture to the new architecture and everything in between that needs to be done. That meant writing lots of throwaway code that wasn't useful after we launched, but it was able to give us that support that we needed during the migration. That meant that we needed to migrate our back ends first because we needed to ensure, we needed to prove that our new architecture is actually solving the problems and our design and implementation of the new services is able to support the existing workload and also can be scaled for growth. And so a lot more energy was put into understanding all the different intermediate steps and refining our plan. It's important to be able to roll out incrementally, especially if you're going for zero downtime migrations and it would have been possible that systems like Mesos or what we use internally at Google Board because it allows us to have a lot of tighter control over which machines are running workload while that was rolled out seamlessly to our users. And we were able to compare side by side how our new code path side doing, how our new system is doing and how our old system is doing and see if the user is experiencing any issues with the new system. And finally, practice made it very important and made it very, that was the key component to our success of launching to production with no downtime. It ensured that we are well prepared for anything that could go wrong, that we know well how to roll back to the old state of the system and try it later. But in the meantime, we can go back and provide a good function of service to our users. And this meant that we used a lot of checklists to be able to coordinate across many different teams throughout different time zones and it enabled us to launch this new architecture to our systems with zero downtime. That's it for today. Thank you.