Good afternoon, everyone, and welcome to the Software Engineering Institute, or SEI's, Research Services, HI-T 2020. I am Sheila Rosenthal, Manager of Research Services. And before we begin our HI-T program, I would like you to please welcome Palma Buddles-Valdes, here with us to honor the importance of this very special day, Juneteenth. Make sure my video is up there. Okay. So I want to say thank you for allowing me time to commemorate the 155th anniversary of Juneteenth. While Juneteenth does celebrate the end of slavery in the United States, I believe it's important to use the day to also recount the more than 250 years before Juneteenth, during which our country practiced and condoned the enslaving of human beings. There's no way to do this topic justice in the short time, but I do want to take this opportunity to inform you more about what Juneteenth is about and the context in which it happened. So right now, I'm sharing with you a segment of a time-lapse video from the Trans-Atlantic slave database from Emory University. It illustrates the number of slave ship voyages from Africa to the Americas. The larger the dot, the greater the number of captive Africans on the ship, and the color of the dot indicates the flag of the country who own the vessel. And so I would really encourage you at the end of this presentation, I have some links, and I would encourage you to take some time to look at this site. It has a number of other things that are I think are important to share. So in 1591, the first slave voyages from Africa began setting the stage for slavery in North America. So the full time-lapse of this video shows over 36,000 slave voyages from Africa, 11,000 from South America to North America, and according to the database, there was some 12.5 million Africans who were forced to make these voyages, and 10.7 million survived that voyage. This journey alone took the lives of someone's father, son, daughter, mother, or grandparent. This alone represents a grave loss of life, the journey of itself. And okay, thank you, Shea, for the slides.  If you jump to 1790, that's when the first U.S. census was conducted and it recorded nearly 700,000 slaves, but they were not counted as their white counterparts were. For the slaves, there was no notation by their name, age, sex, or origination. What was listed was the number of slaves under the owner's name. Then we move through time into 1808, and through an act of Congress, the African slave trade was abolished. However, the widespread trade of enslaved people was not prohibited and freedom was not achieved. On January 1, 1808, the day that this act went into effect, Reverend Peter Williams, Jr., an African American Episcopal preacher, and abolitionists celebrated the passage of the act with an oration on the abolition of the slave trade. And he delivered this in the New York African Church. At the end of this presentation, I also provided you a link to Reverend Williams oration, and I encourage you to read that as well. By the 1830s, the abolitionist movement in the institution of slavery in the U.S. was underway. The movement demanded the immediate and full emancipation of all slaves. An abolitionist included Frederick Douglass, who was born into slavery. His mother was of Native American descent. Harriet Tubman, who helped escape slaves, reached the North via the Underground Railroad, and Sojourner Truth, who was both an abolitionist and a women's rights advocate. Harriet Beecher Stowe, who authored Uncle Tom's Cabin, Susan B. Anthony, and William Lloyd Garrison were also well known abolitionists. On September 22, 1862, Abraham Lincoln signed the Emancipation Proclamation freeing 3 million enslaved people in the state of the Confederacy. It became law on January of 1863, but all were not free. It wasn't until 1865 when the Thirteenth Amendment, which abolished slavery in the U.S., went into effect that the Emancipation Proclamation became law. And that leads us to Juneteenth. It is the oldest nationally celebrated commemoration of the ending of slavery in the U.S. And on June 19, 1865, Union soldiers landed in Galveston, Texas, which is actually three hours away from where I am now, with news that the war had ended and that over 250,000 enslaved blacks in Texas were now free. So I'd like to note that this was two and a half years after the Emancipation Proclamation. Texans began celebrating Juneteenth the next year in 1866. Do the next slide, please. They celebrated with community-centric events, such as parades, cookouts, prayer gatherings, historical and cultural readings, and musical performances. And it is thought that Texans who moved to other parts of the United States brought this holiday with them. Today, Americans across the country celebrate Juneteenth Freedom Day and today, Juneteenth 2020, people are uniting for justice. Generations of African-Americans and other people of all backgrounds who knew that slavery and discrimination are wrong have argued for, protested for, and given their lives in an ongoing push for equality. From former slaves turn abolitionists to the tireless organizers striping beneath the Black Lives Matter banner, demanding that our nation live up to the principles expressed in our founding documents that all humans are created equal. And 155 years later, we are asking ourselves how far we are together from realizing that.  On this, the 155th anniversary of Juneteenth, steps have been taken at the city level across the United States, Portland, Austin, state level, Virginia and New York, and at the federal level to elevate Juneteenth by making it an official state, paid in many ways, in many places, holiday. Organizations across the U.S. are also taking steps to recognize the importance of this day by providing holiday and in some cases closing their organization in observance. Here in Austin, there are a number of protests, peaceful protests, and I know that's across the country as well today. There's also a petition that was put forth today in Congress to make Juneteenth federally recognized. That just hit the news, I think, a couple of hours ago. I end asking that you take time to learn about the importance of this day and to use the knowledge that you've gained for action. And the last slide, please. Yes, and these are the links to some of the things that I've referenced and I've also included some links to some activities that are going on within Western Pennsylvania at the National Museum of African American History and Culture, and then also additional places that you could go to learn more information about Juneteenth. Thank you. All right, thank you so much, Palma. Just want to welcome everyone to the 2020 high tee, CMU Atlas of AI. I'm Shae Badalado, your event planner, and I want to thank all of you for attending today. And I want to thank those of you who registered for the special tee kit, the Won't You Be My Neighbor. The idea from the kit came from today's title of the program, CMU Atlas of AI. The word Atlas sparked questions of where do we call home, what do we love about our neighborhoods, and what makes us feel like we're part of a community. We wanted to make sure that the kit represented Pittsburgh, a place that most of us call home. The tee that you have is from Love Pittsburgh, which sells local gifts and goods from local artisans, and they join forces with Tupelo Honey in Millvale for the exclusive Love Pittsburgh blend that you have with you today. The honey stick is from Pittsburgh Honey, which is a family-owned apiary located in Squirrel Hill. And then the tee infuser kit that you have is from Marco Promo in Wexford, and he works really closely on most of our swag items at our events with communication design team on branding. The sweet treats are from our very own cyber cafe working alongside Potomac Bakery, which is located in Mount Lebanon in Dormont. I hope that everyone enjoys the tee and the kit, and that you have your favorite mug ready. I'm about to hand the program right over to Sheila Rosenthal to get us started. Thank you. Thank you, Shay. We are always delighted to be teaming up with the University Libraries for high tee, and this year's event is even more special because it's our very first virtual high tee. Since all attendees are muted, we hope that you will use the chat feature to submit your questions and comments for our speakers. Please stick around at the end of the program for the networking portion of the event where we will randomly place you in one of three Zoom rooms. Importance of honey bees? Won't you be my neighbor? And high tee versus afternoon tee. Now I am honored and delighted to introduce our first special guest speaker, Tom Longstaff, SEI's Chief Technology Officer. SCTO, Tom is responsible for formulating a technical strategy and leading the funded research program of the Institute based on current and predicted future trends in technology, government and industry. Tom rejoined the SEI as CTO in 2018 after 10 years in a variety of senior roles at the Johns Hopkins Applied Physics Lab and National Security Agency in Maryland. In Tom's prior role at the SEI, he was one of the early members of the CERT organization, developing many of the institutional groups still in place today. Tom received his PhD in computer science from UC Davis in 1992 and now lives in Pittsburgh with his wife, Andy. He has two sons, one grandson, four years old, and is grateful everyone is healthy and thriving. Please welcome Tom Longstaff. Thank you. Thank you so much, Sheila and Shea and the entire team for putting together, you know, really this wonderful high tee and thanks especially to Palma for really kicking us off by correctly observing the importance of Juneteenth and the importance of that day that we have today. But I really am grateful to the tee and I know I am violating the Zoom etiquette by slurping my tee in the middle of a Zoom call but I'm going to anyway. Really delicious, really delicious. Look, we're living in a world with regard to our research collaboration that it's hard to believe could even happen to us only three months ago. Collaboration in January for me and for most people that I know was walking down the halls or walking from the SEI building to campus or walking to another building in the CMU campus, running to colleagues along the way, running into offices dynamically coming up with ideas, not even knowing ahead of time what it was necessarily we were going to talk about. But just really collecting thoughts with people and personal interactions and, you know, being very close socially, not at all distanced. I call that sort of academically close and right now in many ways we feel academically distant in the same way that we're socially distant but that doesn't mean that collaboration is not every bit as important as it always was. So, well often what happened before is in January, we would talk to people, we'd get ideas and where would we go next? We would turn to library science or turn to the various resources we would have to then start to educate ourselves and look up lots of details on the important areas of the events of the important areas of study that we were thinking about doing. So, while library services were incredibly important, they were kind of secondary to that first, you know, getting a hold of people and doing the interactions that we would collaborate with. How different now collaboration is? So, between Zoom, Skype, Teams, WebEx and FaceTime, one thing they all have in common is they're focused. We know what we want to talk about when we get into these calls. We call for purpose. We call for reason. Even happy hour calls and high T calls have purpose and reason. They're no longer serendipitous. So, what things start with in this new normal is how important library science now is than ever before. How important things like Kilt Hub that we're going to talk about for allowing us to get insight and new ideas and new sparks, new ways to start collaboration are even more important in this new normal. And even in a world when we all come back together again, when we're all vaccinated and we all come back, that world is still going to be important to us in terms of finding information, finding colleagues and finding purpose in the interactions that we want to have. We have a key example happening right now in Palma. I'm going to turn back a little bit to your talk because it's so important. Following the murder of George Floyd and the recent killings and the protest movements that are there, I personally am spending a lot of time and reflection around how to become a better anti-racist. In the world before, this would have revolved around wandering the halls and talking to a lot of my colleagues and having hard discussions, heart-to-heart discussions that I would start up from that. And one particular key learning that I have right now is this is not the time to be reaching out to my Black colleagues or to the African American community or to people of color and asking them to educate me. This is not the time for that. So I have to do my own homework. What's the best way to do that? It is turning to the resources. It's turning to library science. It's turning to the best learning that we have to begin to focus my efforts so that when I do reach out in Zoom and when I do reach out in teens and when I do reach out on all of the other collaboration, I can do it with purpose having been somewhat educated. So Palma, your resources at the end of the Juneteenth discussion are especially important. And the idea of Kilt Hub is especially important to begin to focus our thinking so that when I reach out and when many of us reach out, coming into knowledge of this space and all of us becoming more aware of both our racist tendencies and how to be a better anti-racist, how to actually come out with that. So I think the idea of Kilt Hub, the idea of library sciences and the idea of this event is even more important now than it's been in the past. It is more important now for us to be able to focus and learn and really begin to bring this into into sharper relief in terms of where we want to go. So my hope today as we begin to explore Kilt Hub and as we explore the other areas on our agenda and hear Keith, is that we can apply what we learned today to the very real situations that are happening in our lives right now, right here in the middle of COVID-19, in the middle of trying to become more aware of how we can be better in many ways, how we as organizationally, as organizations can be better, how our policies can be better, how we as people can be better and how we can use the resources to do that. So I'm grateful, I'm grateful to the resources, I'm grateful to colleagues like Palma that can really help point me in the right direction, I'm grateful to the various resources that are now becoming prevalent and are out there that we can all see to sort of move in that direction. So with that, for the next 15 minutes or so in our program, I really want to focus a little more tightly on the SEI's Emerging Technology Center's goal of collaborating with new research partners across the SEI and CMU, primarily in the world of artificial intelligence. The goal is to make it easier for anyone at CMU to discover faculty and staff and AI, to aid in the creation of resources that showcase the benefits of partnering with the SEI. And as you will see, the intention is to describe the role of Kilt Hub in achieving this and so much more in terms of collaboration. So with that, I'd like to introduce Carol Smith, Ritling Gupta, Andrew Mellinger and Holland Browner in order to help us understand better some of the resources that we would have with Kilt Hub. Please. Wonderful. Thank you. And thank you all for joining us today. We're going to be talking about, as was mentioned, the Atlas of AI project. And unfortunately, Jay Pallet wasn't able to join us today, but he also helped to initiate these discussions. And first I'm going to turn to Holland and she's going to tell us, I hope, a little bit about what the Atlas of AI was, what this project is about and what we're trying to do at the ETC. So sure. The Atlas of AI describes our effort to understand the breadth of AI research being done on campus. It's an informal project, so it's not like the level of a line or a lens project, but it's still important. And the title is catchy and it kind of stuck because we're talking about a sort of mapping of that AI research landscape. And a specific part of this that relates to the theme of today's high T is Kilt Hub, as Tom has mentioned. And you're going to hear more about the specifics of Kilt Hub, especially in the panel later on. But I'll just say briefly that for anybody who's not familiar with Kilt Hub is that it's CMU's institutional repository for scholarly works and research publications and other materials. And it has some really great features like collections and tagging. And it's available to anybody on campus, including us. Great. And how do you see Kilt Hub helping us to tell this the AI engineering story, something that the whole SCI is interested in working through? Sure. So there's, I guess I should talk a little bit about the goals. So first, the main genesis and the motivation behind the ALIS was that we always have collaboration in mind. And this is how the ETC and really the SCI in the bigger picture increases what we can do. We like the phrase that we scale through collaboration. And campus is a natural place that we find those collaborators. And we already have a really strong network of campus contacts. But we try to keep up with what might be a new direction. Who don't we already know? And what might those researchers be doing that relates to our projects and our larger objectives? So of course, Kilt Hub wouldn't be a substitute for that strong network. But it's more of another tool that we could be using. And a bonus is that since it's our university's institutional repository, researchers are adding material to it all the time. And we just have to find it. And another thing about Kilt Hub is that I think about what might we add to it. And that's something that's near and dear to my heart as our team's strategic communications manager. Because I think of Kilt Hub as one more place we can tell our own story. And that's especially important as we lead the creation of the AI engineering discipline, which you were asking about. And there are a few key parts of that story, which has its foundation and DOD needs. We know that there's a DOD need for certain qualities in AI systems. And we're working towards scalable, robust, robust and secure and human centered AI. And our work in AI for mission enables us to learn lessons that feed into a discipline of AI engineering. And then that discipline in turn helps us work toward those DOD goals. So I guess to get back to your question, it's that I can see opportunity for us to tell or reflect that story using Kilt Hub. And maybe an obvious approach would be to maybe create some collections related to those goals, as we have more and more material. And we can certainly pull in campus publications. And this would work alongside our website, our blog, and all of our different media and content streams that we have already on YouTube and LinkedIn. So it's another powerful tool that we can use. And I'm excited about the possibilities. Awesome. Excellent. And then we're going to have Andrew join. And Andrew is going to talk to us a little bit about the broader efforts on artificial intelligence that are happening with the ETC, and then also how we've been able to collaborate with the rest of the SEI. Hi, everybody. So there's a paper we put out. It looks like I just looked at the copyright on the paper about a year ago. I was on June 6 of last year. I think the link is going to show up in the chat here at some point. And it was on 11 foundational practices for AI engineering. And this is a paper that we did in response to a custom request. It was done by, well, it was supported by each of the different divisions, but the people kind of representing each division was, EPEC was for SSD, Angelo was for CERT, and I was representing ETC. And we kind of pulled all the just the very, very high level information directed towards decision makers about kind of how can you describe the key areas, the key practices, foundational practices of AI engineering, because they want to be able to, as they acquire more projects, kind of have some understanding of like what they should be looking for, what characterizes a program that's kind of addressing all the areas. And again, as this is made for decision makers, it's only a handful of pages. It's really three pages, I think. But the goal of this was to kind of sketch again, sketch out those high level statements of like what in very pithy ways, what's important. Our goal then, it's kind of stalling right now with obviously COVID and other things, is to be able to take each of those areas and produce more material materials under each area. So for example, Carol's moving forward under 0.11, which is about ethics as both a software design and consideration and policy concern. So meaning how do we bring ethics into AI? And I think we had a conversation about that not too long ago podcast, I believe somewhere. Was it actually about that? Human factors, maybe? Those kind of things from the AI, yeah. And maybe they'll get posted somewhere too. So we're going to continue to try and build this map out and hopefully get those things either in Kilt Hub or our own materials or elsewhere, right? Regardless of storage issues, those things are really important, right? And they're kind of the concerns, I think, that all engineers and all AI folks face and the government's going to be facing. So we're hoping that people, if you see something they're of interest, feel free to email any of us with things you'd like to produce or things you know of that could help fill out that area. Obviously that's not all inclusive of everything. It's just kind of a starting point, but some way to get our heads around the AI ever increasing AI problem. Excellent. And we're going to have Ritwik then talk about some of the actual projects, some of the work that we've done, including Project Hater and then also with the GVSC. Ritwik? Whoops. Yeah. Hey everyone. Glad to see everyone here. Glad that everyone can see me. Best with the mute button. Yeah. So, I mean, Andrew gave a very, very good overview of kind of the stuff that we have. Some of the stuff we have in our AI pipeline, but specifically some of the work that we work on focuses on humanitarian assistance and disaster response. So we've been working on this stuff for the last about a year and a half now. And the idea is that again, there's a lot of problems when it comes to the world with regards to human rights abuses. How do we properly get people the aid that they need when they need it in a fair equitable fashion? How do we respond to disasters again in a timely fashion? More preventative than reactive, etc. And so these are all areas that we've been engaged in. We recently did some work with the Defense Innovation Unit where I currently sit focusing on how do we properly do building game assessment after massive, large-scale disasters across all areas of the world. Highly successful. Something that we really worked on for about a year. Worked with, again, I think the key there was collaboration. It's nothing that we can do in isolation. We don't have the experts here. I'm not an expert in this world to kind of speak authoritatively in any of this. So work with over 40 different agencies around the world, the UN and Nepal, Cal Fire, the city of Walnut Creek, surprisingly, very tiny, but it's close by Amnesty, AAAS, etc., to basically build this AI toolkit which can take satellite imagery from multiple different sources, but primarily commercial satellites, and automatically within seconds start classifying which buildings are damaged, to what degree they're damaged, and how should responders then allocate resources. It's been super great. We've already deployed it to the Australians, the Australian Geospatial Intelligence Organization, to help with their bushfire response. We're currently working with the California Air and Astro Guard for the California fire season. I can see some of the smoke already rolling in out here. And we're also working with the New York Fire Department and other interested partners as well as the Jake, the Joint AI Center, to make this a reality. One thing that we're really focusing on now, something that's coming up and certainly very relevant, and as we see the political climate unfolding in front of us is human rights abuses. How can AI be used to counter human rights abuses? And also, how can AI be abused to basically aid and abet human rights abuses? And so we're working now with OpenAI. We're working with a lot of different partners to start surveying the field and what's the state of the art in both good uses and bad uses, and hopefully start doing something about it to make sure that, again, the AI that we're building is equitable, free of bias, fair, and easily accessible to anyone who needs it. Yeah, so there's a lot of stuff going on. I know, Carol, you mentioned the stuff we're doing with GVSC. Again, that's a pretty cool collaboration. That's not really necessarily related to Hater or anything like that, but again, the Army needs vehicles that run autonomously. The Army obviously has a large investment in it, so we're helping them basically understand how does autonomous behavior fail? What are the anomalies? Where does that behavior arise? What do we do about it when it arises? And also, what are all the different edge cases as to how it could fail? So that's been a really successful project, working with our colleagues, Eric Hyme, working with a lot of collaborators on campus at the Robotics Institute. We're working with, specifically now with Dr. Jean Oh, she's awesome. Her work kind of focuses on in-risk reinforcement learning. There's a lot of disaster use cases for that, but basically she's helping us figure out how to basically understand a robotics anomalous state as it moves from state one to state two instead of once the mission is done and looking at it retroactively. So she's really helping us make a huge leap in that world. And we're also collaborating with some of our colleagues at the Institute for Software Research, which the SEI at large collaborates with in a very large fashion. I know Andrew's work with them a lot, but they're helping us understand, and this is actually critical for a lot of things, but as different AS systems interact with each other, there's a lot of errors that go in. We've all seen AS systems. My system may be 98% accurate for certain things, but then I start adding another thing at the end of it. So now my planner now goes into my actuation. The actuation then feeds on something else. And as we compose AS systems together, errors can propagate and get bigger and bigger and bigger, and we don't even understand how that happens. So Christian Kastner and some other collaborators at ISR are helping us figure out how does error propagate through AS systems? How do we do something about it? How do we understand and then visualize it so that an operator knows what's going on? And then finally, applying it to this robotics use case. But again, I think the point to be emphasized here is collaboration is key. We're not the experts in the space, and we have to very much be aware of that and understand where our shortcomings are so we can augment our capabilities and work with our colleagues both in the SEI and outside of it. And really, again, just making sure that our knowledge is distributed as widely and as fairly as possible. So if anyone else has any ideas or knows of any other AI collaborations that they want to start, let us know. We're here to help everyone kind of force those connections. So yeah, happy to help. Excellent. So as you can see, we've got many, many existing partnerships with external organizations as well as customers. And here with CMU partners within the SEI, we've been able to work across the different departments to be able to publish work that is helping to add to the knowledge base around AI. And we're hoping that the use of Kilt Hub and other tools are going to really help us to be able to do more future collaborations and be able to further the research in AI and make sure that we're doing ethical work as well. And I'll turn back to Sheila. Thank you. Oh, they're in the questions. I forgot to check. I'll actually add some of our colleagues who are human computer and action institute reached out to us to collaborate on our work because they found our paper on Kilt Hub. So already seeing a lot of good use out of Kilt Hub. That's awesome. Well, thanks everyone on the ETC team. That was an excellent presentation. And I hope for our attendees that if you have questions, be sure and put them in the chat and we'll address them definitely. And now it is with great pleasure and enthusiasm that I would like to introduce our next special guest speaker, Keith Webster, Dean of Carnegie Mellon University Libraries. Keith has spoken at a number of recent SEI HITs and will be known to many of you. Keith joined CMU in 2013, having previously held leadership roles in publishing and in universities in a number of different countries. Now please welcome Keith Webster. Thank you, Sheila. Good afternoon, everyone. I'm truly sorry that we can't be together in person this year, but that's a great test of the technology. We're already planning next year's HIT and we've said that we do hope we can join each other in person in the SEI for that event. I echo Tom's appreciation of Palma's opening remarks and also his recognition of the importance of responding actively to recent events. Today we commemorate the anniversary of the delayed but welcome news of the end of slavery. Collective action followed as equality and justice for African Americans advanced slowly, frustratingly, gradually on this nation's journey toward a more perfect union. I think it's particularly significant on this Juneteenth that we remember that struggle as we reflect on how far we've come as a country, but also come to terms with just how much remains. Next slide, please. Open science can be defined as any type of scientific research effort that is freely available and publicly accessible. I use open science to reflect both traditional research products such as journal articles, conference proceedings, and research data, as well as other artifacts of the research process such as blog posts, code, social media posts. There has been a considerable uprise in interest in open science in recent years because there is a growing sense of expectation that those who pay for research will have access to the products of the work that they fund, and that can be taken to the highest level of taxpayers looking for access to the articles and other products of the research that their taxes have funded through the NIH, the NSF, and other grant agencies. There's also growing expectation of reproducibility and accountability, striving to minimize errors and reduce scientific misconduct. We know that in so many parts of our lives, the internet has democratized what we do, and we expect science to be the same. We want it to be visible and accessible. And we also recognize that open science can help increase visibility and play an important part in building a scientist's reputation. This model is taken from the European Union's open science framework. I'm thrilled to hear so much positive feedback about our Kilt Hub repository, and you can readily imagine how its role as both a research data repository and as a publication repository can help accelerate this university's interaction with open science. And I'll say a few more words about that in a moment. Next slide, please. The overarching principle of open science is that research conducted openly and transparently leads to better science. Claims are more likely to be credible or to be found wanting when they can be extended and reproduced by others. All phases of the research process provide opportunities for assessing and improving the reliability and efficacy of scientific research. Integrating open practices at all points in the research process eases the task for the researcher who's committed to open science. Making research results openly available is not an afterthought when the project is over, but rather it is an effective way of doing the research. And planned from the outset, it is a natural part of the process. Researchers can take advantage of robust infrastructure and tools to conduct their experiments and use open data techniques to analyze, interpret, validate and disseminate their findings. And indeed, we certainly are seeing a growing sense that researchers believe that open science helps them succeed. I'll mention in a few moments our annual open science symposium series where we've heard so many success stories from members of our research community. Next slide, please. Open science in the broadest sense is being encouraged by agencies engaged in advancing research, whether it is institutions like universities and research institutes, agencies and charities that fund research, libraries and research offices that provide the necessary infrastructure and expertise. And collectively we are doing that because just to reinforce my messaging, it drives greater attention to the research, the research, the researchers, the products of research, all are in the public eye. I have a firm belief that an attitude of openness leads to greater open access, which leads to greater science. If ideas can be freely shared, exchanged and built upon without paywalls, without having to try and figure out who might have a relevant data set, we can do so much more. We can view this also as a huge infrastructure cost saving. If an experiment can be run once and the data reanalyzed by anyone who might have fresh insights, fresh ideas, we can achieve so much with increasingly limited research funds. There are emerging a whole new series of communities in the open science environment. Many researchers now are turning to those as the place in which they can build their peer networks. And I do get a sense that perhaps this is for good or for bad and having an impact on the traditional learning and scholarly society, they perhaps are losing membership to communities that are being built on open science platforms and in social media, particularly amongst early career researchers, graduates, students, postdocs, assistant professors. In recent times, a number of intergovernmental agencies have set out challenges and questions about open science. And at the forefront in recent months has been work from UNESCO in trying to build a series of recommendations on open science. Next slide please. This week, coincidentally, was the deadline for public comment on the UNESCO recommendation on open science. And at an event to encourage the submission of feedback, the UNESCO director general built some clear links between their thinking on open science and the pandemic through which we have been living in recent months. And I've put some of his remarks on the screen, but let me read them. The pandemic raises our awareness of the importance of science, both in research and international cooperation. The present crisis also demonstrates the urgency of stepping up information sharing through open science. The time has come for us to commit all together. So just as we've been hearing in earlier remarks this afternoon, this notion of collaboration, whether between colleagues in a single institution or between the global research community is instrumental in advancing the economic, technological, social and health well-being of our climate. Next slide please. Some broader points raised amongst the UNESCO community exploring open science just tries to tease out some of that thinking more systematically, that sharing knowledge reduces the gap among countries. Undoubtedly, there are phenomenal inequalities amongst research, enterprise, research infrastructure in different countries. But that is not to say that the best minds are automatically found in the wealthiest countries. And what better way to level the ability of anyone to contribute to this global pandemic than by sharing knowledge so that anyone, wherever they are, might contribute. The mobilization of everyone is allowing for free access to scientific data, to sharing research findings, educational and public health resources and research facilities. The open science environment is reinforcing the links between the scientific enterprise and the policy and decision makers who collectively are trying to meet societal needs. And also recognizing that the ability for the research community to exchange ideas and skills is made more difficult whilst national borders are closed. Open science inherently is an online activity and that transcends any physical border. So next slide please. Responses to these sentiments have been strong. The left hand panel on this slide reports on activities at the end of January when the coronavirus, I think by then it had been named COVID-19, still was in its early days of global impact. But at that stage, in January, 117 research funders, public health organizations, centers for disease control, academic journals committed to immediate open access as a concerted effort through which results of scientific research into COVID-19 could readily and openly be shared with the World Health Organization. Fast forward a few weeks to March of this year to the right hand panel which reports on a commitment made by the chief scientists of 12 countries including the United States through which they called for open access to publications and machine readable access to research data. Next slide. Building upon these sentiments, technology and other commercial companies recognized that they too had a part to play. This wasn't all in the hands of university and charity researchers. The corporate world had much to contribute and they committed collectively to sharing intellectual property, setting aside commercial competition to fight COVID-19. Next slide please. The Allen Institute for Artificial Intelligence partnered with many research groups to pull together relevant research articles as well as pointing to free and open analytical tools and data sets to allow anyone to begin to mine for new insights into COVID-19. This truly is a blend of open science and citizen science through which anyone can participate in this opportunity or this fight. This is a perfect example of these activities in action. Next slide please. We heard earlier about Kilt Hub which is Carnegie Mellon's institutional repository. That is built upon a platform called FiXHR which is a commonly known research data sharing platform. A parallel product to FiXHR is dimensions, a relatively new tool that allows us to look at scholarly output, research publications, data sets, clinical trials, patterns and so on. I appreciate this slide is a little bit messy but if we could jump to the next slide, I've pulled out a few of the relevant panels. Thank you. What I did here was a simple search for COVID-19. I reckon that that would be a fairly easy one to limit by time given that COVID-19 was named this year and what you can see in the top bar is highlighted in the darker text 52,262. That is the number of publications that have emerged this year about COVID-19. I have no doubt that this must be the biggest explosion of scientific literature on a single topic ever. You can see the timeline in the panel just below that to the left. No publications in preceding years and then a sudden jump to 52,000 publications in four or five months. You can see just below that a substantial portion, roughly 84 percent I'm guessing there abouts that are available in open access format. You can see in the bottom right hand panel the extent to which those publications have been supported by different research funders given the early iteration of the virus in China, not surprisingly, research funding being represented from Chinese agencies quite particularly. Just above that panel, so I'm now in the top right one, a point that I'll return to in a moment's time, but the number of those that are those publications that are present in pre-print servers and similar types of outputs. I'm grateful to my colleague David Shearer who you'll hear from in a few moments who's pointed to a couple of links in the chat to the dimensions platform. Ritvik has pointed to a question about DoD funding. I believe DoD is marked as a research funder in dimensions and we can look back with you offline. I'm going to show my terrible ignorance here in Nick's question about different types. Broadly speaking, Gold Open Access which is at the bottom of the slide is one where the author or their institution has paid an author's fee to make articles freely available in journals that are typically commercially licensed through subscriptions to libraries and I'll say more about that also in a couple of slides time. Green Open Access is where a copy of an article has been deposited by the author in the institution's repository and Bronze Open Access is one that has been deposited in a pre-print server and I'll say more about that in two slides time. Bear with me. Closed means it is not free to read but rather is behind a paywall and means you need a personal or an institutional subscription to that content and thanks to the pointer to Wikipedia. I wish I'd looked at that rather than ad-libbed but bear with me thank you. Next slide please. Clearly 52,000 publications never mind the associated research data I skipped over that but there were 574 data sets indexed in dimensions alone is a phenomenal challenge. Nobody can keep up with that amount of literature and we are seeing innovative uses of AI from groupings like Kaggle and SciSite and there are certainly other tools emerging to support that sort of analysis. Anyone who has an aptitude and interest in AI I'm sure will find opportunities to put their expertise to good use in mining this content. Next slide please. The peer review process for traditional scholarly journals has often been held to move at glacial pace and some of the best journals have months of peer review process and in an effort to expose research output as quickly as possible in the current pandemic. Many researchers have turned to pre-print servers like Med Archive and Archive and Bio Archive as a way of getting their content into the public domain as quickly as possible and that is what constitutes the bronze open access that I mentioned a few moments ago. You can see in this Med Archive article that it has been pinpointed as being a pre-print and it has not yet been peer reviewed and people read that with a degree of understanding that experts have not reviewed the rigor of the science or the reproducibility of the findings and it is understood to come with that warning but we have seen some unfortunate consequences of that even in peer reviewed journals like Lancet where there was a high profile retraction of a study a couple of weeks ago so there is certainly a sense of concern about the extent to which non-peer reviewed literature has surfaced in recent times and I do hope that that in some way points to the importance of reading research carefully and thoughtfully and relying upon the expertise of librarians and academic colleagues to help understand that more closely. But in this open access and open science environment please do appreciate the support that your university libraries and your colleagues in research services and the SEI can bring to your interest in joining that effort. Next slide please. You can find a lot of content on the CMU Library's website on the tools and services and training opportunities that we can provide to help with open science and data collaborations and my colleague Wajin Wang who will be moderating the next panel is the person to approach with any questions that you might explicitly have on open science and data collaborations. Other colleagues on that panel are also experts in different aspects of our work in this area and will help you with Kilt Hub and other services. I know that my colleague Hannah Gunderman is present at the IT and she is our consultant in research data and can help you plan the curation management and sharing of your research data when you need her expertise. Next slide please. As a way of keeping up to date my colleagues produce a regular newsletter on our efforts and initiatives in open science and we do hold a number of training courses and events. Next slide please. For example last November we held our second open science symposium in partnership with the Mellon College of Science and the presentations and slides from that event are freely available on the open science framework site that we have licensed as a tool for use across the university. I mentioned when looking at the dimensions platform the opportunities afforded by gold open access. Next slide please. Typically anyone wishing to publish an article as a gold open science article has had to do so through the payment of an article processing chart which typically costs something between a thousand and as much as five thousand dollars per article depending on the journal and the publisher with whom the article is being placed. We have embarked upon a process to try and institutionalize gold open access so that when anyone at the university including the SAI is the corresponding author on an article they can do so by taking advantage of comprehensive agreements with individual publishers. Towards the end of 2019 we announced our first agreement with Elsevier the world's biggest commercial publisher such that articles submitted for publication after the first of January 2020 would be published open access when they were accepted for publication. Effective the same date our largest society publisher the association for computing machinery announced an agreement with Carnegie Mellon with MIT and with the University of California and Iowa State University. Those four groupings had negotiated together for about 18 months to arrive at the first phase of an agreement through which the ACM intends to make all of its academic journals open access and just a couple of weeks ago we announced our latest agreement which takes effect on the first of July with the public library of science our largest completely open access publisher with whom our faculty publish. So we have sequentially negotiated with our biggest commercial our biggest society and our biggest open access publishers and those are services that we hope are going to be of benefit to the entire university community. Just in the last few days I know that Matt D'Amelio the head of the university contracts office has worked with Elsevier to negotiate a special form of warrants that explicitly revises the agreement with Elsevier to reflect the specific government interests that many of you must respect in publishing your work. So I hope that that will be of particular benefit. We haven't stopped there there are negotiations underway with some other major publishers and societies watch our website for further good news in the months to come but next slide please. Very quickly let me just say a few words about how the libraries have been coping and continuing to support education and research since we vacated our campus facilities towards the end of March. There are a number of interesting case studies anecdotes interesting news items on our website under the tagline online by but never remote which we borrowed from our colleagues at CMU Qatar. I'm going to go through the next few slides quite quickly so please have your finger hovering over the advanced slides button. So next slide please. In many ways we've been preparing for this for years and we have a collection now of more than a million electronic books that can be downloaded or viewed online across all fields of scholarship and I couldn't help but do a search for pandemic and bring up 487 electronic books about pandemics. Next slide please. Our move towards online journals started almost 25 years ago and we have a tremendous number of scholarly journals available for you to view through the university library's website and next slide please. Those who are interested in non-textual media such as musical recordings and performances, operas, theatrical productions, all of which are important to our students and faculty in the College of Fine Arts can find those online if you would like to view those as a way of relaxing listening to soothing music in the background while you work please do they're freely available for you to use. Next slide we also have streaming academic video resources through services like Canopy. Think of this as a bit like a Netflix or a Hulu for academically relevant documentaries and films. Next slide please. All of that points to just the vast array of digital content that we have been building towards for many years and it reflects truly the way in which the vast majority of our colleagues interact with libraries these days. We have also been able to benefit from the goodwill gestures of publishers to help us continue support for those who have been disrupted by the closure of our libraries and the loss of access to their personal collections. We know that many students were disrupted because they had gone home for spring break or gone away on research trips or a vacation and they left their textbooks behind in their dorms in their rooms and apartments and then went home to their parents in many cases and left their textbooks behind. Textbooks are the last bit of the puzzle we haven't solved yet but many publishers have freely opened up the textbook resources made them free to access and we have worked with students and with departmental liaisons to ensure that students have been able to access at no charge the online version of their textbooks. We also are beneficiaries of a service called the Hathi Trust which is a community resource through which many universities have pulled their scanned versions of print books in university libraries, many from the Google Books Project. In normal times these are not openly accessible because those books remain in copyright but there is a particular provision through which we are able to provide access to these books whilst our libraries are closed. The broad argument is that libraries have invested in these print collections nobody can access them these digital surrogates represent a fair approximation. So through our website we can provide access to digitised versions of the print books held in our collections where they are available and the Hathi Trust through the emergency textbook access service or some such thing it's known as E-Tas. Let me just keep going next slide please I saw a question in the chat I'll come back to that. Of course not everything existed in digital form and before we went offline or off campus we were actively engaged in digitising textbooks and copies of chapters using fair dealing provisions so that those who we knew would require access to materials normally available in print would be able to do so digitally whilst we were operating online only. Next slide please so as far as possible we made every effort to ensure that those who required access to academic content to continue learning teaching and research would be able to do so but we knew that there would be cases where people required access to things that could only be provided in print either because of the particular content and highly visual materials or books that simply didn't exist in digital form and we established a service through which faculty and graduate students could alert their liaison librarian and in turn if the liaison librarian determined that a book was required in print we would arrange for that to be delivered from amazon in most cases to the faculty or graduate students home address so that they would be able to continue their work using those print resources and we trust them to bring the books back to us when our libraries reopen more news about that on our website. Next slide please it's interesting just to pause and reflect back to what happened in 1918 when the Spanish influenza pandemic swept the planet and looking at the newspapers in our archives from that time we see news that Carnegie Tech as the university was then called suspended football because of the pandemic the army training corps who were housed on campus due to the first world war arrangements were placed under quarantine these are important parts of the university's history and my colleagues in the university archives have set out to catalog the lived experience of everyone on campus where they wish to share their stories during the current situation next slide please so they have called for anyone who has been touched by the pandemic to share their stories what is it like to be learning online what is it like taking a music course and trying to teach somebody to play an instrument online rather than next to the student next slide please so we have developed a toolkit through which anyone who has an artefacts information stories to share to please do so so do contact our colleagues in the university archives if you have things to share and next slide please my last slide there are many more questions that we have been asked and have been answered by the experts in the libraries than I have touched on so please turn to that frequently asked questions section of our website I see a question in the chat from Caroline about the data sets attached to publications in our repository that points to a very important thing that I'm sure my colleagues in the panel will talk about which is that we deliberately set out under David Shearer's leadership to view our killtub repository not as something exclusively for a particular type of output whether it was publications or data sets or other types of artefact but rather to view this as a comprehensive resource for any digital product of research and that has allowed us in a single platform to associate different research objects with each other so for example it is possible to deposit and killtub a journal article and to deposit alongside that data sets that underpin the research that is recorded in the article the data sets themselves can be given a DOI which can be cited in the article and in turn those data sets can be specifically picked up by services like Google Data Set Search and that helps raise even more aggressively awareness of the underlying data and allows for the broader sharing that I have been referring to in the open science part of this presentation so and David has put a link with an example of that in the killtub repository so please do check out the COVID-19 stories there are some wonderful and moving examples here I must say just how deeply proud I am of the manner in which everyone in the university libraries has turned to support the university community during unprecedented times and the fashion in which they have been able to support everyone who has continued to teach and learn through half of a semester delivered and target online we know that some people have increased their research activities during this online operation and others have struggled to undertake some of their fieldwork and we've done everything we can to help that continue so with that I am going to pause I am happy to take any quick questions but I really do want to turn things over to my colleagues on the panel so what I am going to do they are seeing nothing new in the chat is introduce my colleague Wajin Wang who I mentioned earlier Wajin is one of our faculty in the ways on librarian team she is a cell biologist by training and also serves as our program director for open science and data collaborations Wajin is going to introduce our panelists and facilitate a discussion on the benefits of sharing projects in killtub so Wajin over to you thank you Keith thanks for everybody for joining the panel so we have seen from Keith's presentation just now very eloquently he showed us how open science has helped us to share the research products help the research to be seen by the broader community and especially during this COVID-19 crisis we have seen it in action so like help us to disseminate the research product really fast and raise awareness for research and for knowledge and Tom has also mentioned in his presentation that so in this new norm collaboration is the key and with the library science and all these platforms we have we can help you tremendously to find information find collaborations so here we have the panels so let's welcome back first the panels from the SEI team Carol Smith Richard Gupta and Holland Barmer to share some of their experience and also I want to introduce my colleagues from the library university libraries who are basically part of the killtub team David Sharer who is the scholarly communications consultant and Katie Berman who is the institutional repository manager for killtub so first David I would like to I want to ask you for maybe give the audience a little overview of what is what exactly is killtub and what can you deposit into killtub sure thanks watching for questions so um hi everyone so as Holland mentioned earlier the killtub repository is the comprehensive institutional repository at CMU and it's provided by the university library an institutional repository is a repository for an institutional community and its collaborators to publish and share the research and other scholarly outputs traditionally an institutional repository or IR houses textual materials such as these using dissertations presentation slide decks academic posters various forms of great literature like technical reports or white papers in versions of peer reviewed publications as Keith mentioned we say versions of publications because authors may be permitted to share a version of their published work in an IR but it depends on the author's rights retained in the copyright transfer agreement in most cases we can deposit the peer reviewed author's final version of an article but it again it depends on each publisher's copyright transfer agreement and in some cases the agreements of individual journal title we call killtub a comprehensive institutional repository because it can do more than a traditional repository can do killtub can also accept audio vision materials and recordings as well as a wide array of formats that research data may come in these roles are sometimes put into different repositories but with killtub we have one single easy to use repository that captures the gamut of materials produced in the course of one's research cycle as the comprehensive repository is CMU killtub also differs greatly from using services like google drive or box killtub is an open access repository which means that all of the materials are freely available to anyone with access to the internet all content in killtub can be downloaded by anyone and much of the content can also be viewed or sampled natively in the user's browser thanks to built-in players and reviewers a part of the repository once published materials can be discovered in a variety of places including with searches engines like google and also google focused search engines such as google scholar and google data set all content is published with a digital object identifier or doi which is included in a recommended citation if the user wishes to cite the content the repository will track the views downloads and citations to the material and will provide additional data on the alternative metrics or alt metrics that the content may receive from attention garnished across dozens of different types of platforms and venues which includes policy documents mainstream media social media reference managers and mentions in venues such as post peer review sites like academia to edu and blogs killtub is powered by the platform known as fig share as Keith mentioned earlier it's a publicly utilized repository platform so many of our campus community members have already used it it's also the same platform used by many disciplines as their pre-print service such as kim archive and it's also utilized by many publishers and funders to publish their data sets and other related materials and this includes the public library of science or plos and the nih along with a complete record that increases discoverability all content is published with a known copyright status this includes the full list of publicly known copyright options from the credit commons as well as licenses that are tailored towards software such as the mit apache and gpl license killtube can also accept custom copyright licenses such as those that are in use by fdi overall killtube mission is to collect preserve and provide stable long-term global open access to the wide range of research data and other scholarly outputs created by faculty staff and students at cmu we hope it's going to become an important resource for the fdi and we're happy to be here today to talk about it thanks all right thanks david uh that's a very nice introduction to the killtube and i hear from what you said there's a lot of i guess motivations for researchers to use killtube because like the discovery the discoverability the licensing it's it's central so carol i want to ask you i i know you have been um worked with david and katie for a while on this atlas of ai project um so can you tell us a little bit what's in that collection and how's your experience so far yeah yeah so we've been uploading a variety of different types of documents so presentations from conferences papers that have been published and other types of information that we want to make sure is shared not only across cmu but available to to the greater community and uh in so far it's it's gone really well we've been able to to upload things and to be able to uh to make sure that they're attached properly and since there's a review process i don't have to worry so much about uh making sure everything's exactly right although i try to make sure it's right the first time it's always the goal um and they really the only um frustration i've had at all is is just trying to make sure that we can find things across departments so that that is a an existing challenge um mostly because we are in an academic institution and there's a very strong adherence to departments which totally makes sense um but it does make it a little bit more challenging to to be able to connect the work that that everyone is doing across the entire organization um in artificial intelligence which is really the area of interest that that i have so um but otherwise it's been very easy to use um and uh the librarians have been extremely supportive so thank you okay yeah thank you very much so like you you have you did mention a little bit about like discovery within of the university um yeah can you tell us a little bit about like what's your envision if if everything like in the perfect world how this would work yeah yeah ideally um we would be able to have a home for artificial intelligence topics and it's just such a broad area and so many people are doing work in this area across so many different domains and it would be really lovely to be able to look for a particular topic such as ethics such as human machine teaming within that set of information and be able to see all the people who have done work on that particular topic within artificial intelligence and right now you can do searches but just the other day i was doing some research for a proposal i'm working on and i was really digging through a lot of different things that had nothing to do with with what i was actually trying to find and so it's a little challenging but it's definitely a great tool and i'm very glad to have a place to put things that that is broader right thank you um so now i want to turn into uh richwick um so i i i really enjoyed your presentation about your research with all these disaster release uh relief data uh so like so as a research i know you have you produce a lot of uh research outputs your research like what what do you exactly make like do you make data set up you make code and how like when you think about sharing this research output with the world uh what do you look for what would you take into consideration yeah so i mean we're we're producing all of the above we're producing data sets pretty big ones especially for remote sensing they still scan up into terabytes we're producing code certainly data processing scripts the actual modeling itself any any code that takes to deploy the models we're ship we're sending out docker containers so some sort of repository to host those containers and those images is essential and then we're also publishing papers so that's kind of our kill tub and other like place like archive and your church on journal process comes in handy um it gets tricky i'm managing all that together um because you know while i can put my code up on github i can't put my docker images up on github and also github is not the place to share my paper so like compiling all those things together gets gets really difficult um it's certainly i think still a good problem um it almost sounds like a place where the problem of data discovery and reuse uh should be should be researched further so this could be a good plug for the workshop the conference that hwajin and some of us are planning um coming up in 2020 for uh data discovery and reuse for um for ai um everything all of that is a problem um um so i'm part of this i'm part of this working group at fema the remote sensing innovation working group and we're facing this right now so maybe somebody could help us solve this problem where uh we want to share our research with with each other within the within the working group um but some of us are publishing multi extabyte lighter point clouds for the entire united states and some of us are probably publishing like 30 kilobyte python scripts that then process those those lighter point clouds and like there's no one platform which which fits both of those needs um and and so this is something that we're trying to figure out right now as well it's it's very much an open problem where are the where do they live now right now uh one of them lives on lives on a doe supercomputer on their uh and so you have to you have to go to the doe get an account onto their supercomputing cluster uh wait to get access and then once you have access use like something like the pitchers and supercomputing center to get that transferred over to something close by to you um and then with this python scripts they just live on github so anyone can access them but then some people in the dod can access them because they block github there's all sorts of different concerns and everything so yeah um any help we could get yeah i hear you yeah i guess the data we use and how to share data on the streamlining all these workflows is a big challenge for everybody and i'm glad we're working on this together um so now i want to turn the question to katie who's uh actually doing a lot of the curation and the work of the data that coming into the kilt hub so katie do you want to like so like for example if i'm a researcher i want to share my data on the kilt hub how do i like get started what do you uh recommend researchers to do uh before they want to deposit their data well i think collin said earlier um every cmu affiliate has a kilt hub account under their andrew address so you can log in to kilt hub get familiar with the interface see what your account looks like you can customize your profile you can add an orchid account and put profile picture your social media links then to create a record you just drag and upload files into the interface and create some metadata um to get started first i'd say take a look at your materials get all your files together as david mentioned you check your author and copyright agreements to make sure that what you want to upload can be uploaded because i know you if you have a paper in i triple e you can't really just download it from i triple e and upload that pdf into kilt hub um think about how you're going to describe your work in the metadata that you'll type in um we're talking about the atlas of ai so we want to make sure that that artificial intelligence research has a kind of descriptive metadata and keywords and abstracts that will allow people to find that in a search and it will come up yeah thanks hedy i also want to add like our data uh data creation data management consultants henna is also here she can provide some consultant consultation if you are not sure what you're doing uh so i encourage you talk to her after this um and so i want to turn to uh back to sei uh so holand um you're the communications manager for the etc program can you tell us like um what is your role in supporting the etc program and then the atlas of ai project and also what's your broader vision for this sure um so on a just a very basic level um i i see myself as helping our team tell our stories and that includes uh describing our projects and our impact and our expertise our future directions and that's to many different audiences uh so stakeholders customers and collaborators um and the atlas of ai is is part of that um and so my vision for that is that that we would continue to build those relationships across cmu and for kilt hub specifically um i hope that it becomes a place where we put our stuff uh one one more place where we put our stuff um in addition to having it in all the places that we have it now um and i i think back to one of the questions that i asked during some of our first meetings um with your team uh and i said doesn't it get confusing having stuff in all these different places like if it's on the sei website and it's on kilt hub like what is what's with that and i really liked your answer and that answer surprised me um because you said lots of copies keep stuff safe and uh i i like that answer and it does make a lot of sense um and i think uh taking it one step further is that lots of copies get stuff found also um and i do know that that we're going to have some some efforts with uh from research services at the sei and the communications team and i i think there will be a more concerted effort to get some of our publications on kilt hub and and sort of um harness that uh lots of copies get stuff found if i could add to what horn said as well it'd be really useful if uh kilt hub had something like an import from archive feature something like that where like we could just seamlessly upload all our preprints into kilt hub yeah so david do you have comments for that i do uh so kilt hub it does have some built-in functionality where it does integrate with other systems in the ecosystem so i just mentioned actually in the chat that uh you know what you were you had mentioned earlier about working with github uh so kilt hub does have an integration where you can sync your github account to your kilt hub account and transfer materials once they're ready to be made publicly available from github to uh to the repository and as you publish new items or new versions of content that content will also be versions in the repository this is something that we're always talking about and focusing on as librarians is how to shrink the gap between various portals and tools in these authentication ecosystem and thinking about an integration between preprint services like archive and others that sync to your repository are great but as as holland mentioned um you know having something shared in multiple different places is great uh one of things i would point out that we offer as a service along with uh the research team at fdi with rachel and sheila and others is that one of things that we can do is help researchers and and cme community members understand how they can share things uh the right way and the appropriate way and the best way to get it out there so that's an additional service that we can offer and um we're happy to help when anybody that's interested yeah thanks uh david uh that's very useful information so for the sake of time i see there are some questions in the chat there i think a lot of them are has already been answered one of them is to create your own sort of a kilt hub package and then Keith has mentioned that sounds like a my kilt hub a service maybe on the way right so uh let's continue this discussion in our like break in our break out break out rooms and then after this event has completed so i'm now going to thank again the panelists for this great discussion and now i'm going to turn back to sheila sorry i had to unmute um well first i want to thank Keith and tom for their wonderful presentations um this is going so much more splendidly than i could have ever imagined and it's all because you're all doing such a great job so thank you hajin thank you everybody who's on the panel if i mention all your names i'll take up too much time from our schedule but i just love you all you did such a great job everybody um and uh so now um please i oh i i want to welcome back tom longstaff and keith wefster they're going to give their closing remarks so tom keith welcome back sure um keith you want to lead off through the echo um sheila's thanks to everyone who's participated either in making today's event possible or in oh keith got muted there you go is that any better yes yes okay i don't know my ear pods dried up of juice so i was thanking everyone who spoke everyone who helped technologically or organizationally in making today's event possible we very much do value our relationship with the sei um as many of you know we have two of these events each year one your traditional high tea in the summer the other an event towards the end of the calendar year we look forward to having an event of that sort um in november or december um enjoy the um tar dislike experience of going into your breakout rooms um tom over to you uh thanks keith i i just want to echo my thanks of course to to sheila and shea and the whole team it takes an incredible amount of work to put on and prepare something like this and i'm tremendously grateful um you know everything from trying to put the program together to the small logistics like who's going to package the box to send out the tea um i mean these are these are all things that they you know that have to get solved during covid and um you know i really appreciate both the panel and keith the you know the description on open science um you know remembering back in my early cyber security days how difficult it was to share and curate data how difficult it was to actually find and identify related research um it it really is kind of a new world of being able to do this and in a much more open fashion i'm tremendously grateful to all the work and things that are going on so don't leave yet um we're all going to uh try to extend this experiment to what we would normally do if we were sitting inside of a large conference room and being able to have tea and biscuits around uh around the table and sort of chat with each other so uh so i think we are going to from here welcome you into breakout rooms where hopefully we can have an open discussion thank you sure