Good afternoon. Awesome. Perfect. Welcome. Thank you all for being here. I'm Katherine Klosack, Director of Information Policy at the Association of Research Libraries, ARL. I want to thank CNI for hosting this important conversation. And I want to thank you all for being here as you find your seats. In just a moment, we're going to hear from representatives of foundations and federal science funding agencies here in the U.S. that support research infrastructure, humanities research, higher education, and cultural heritage initiatives. And as a panelist share their insights on trends and priorities that they're observing, listen for common trends around working toward equity and taking new approaches to addressing global challenges. We're going to hopefully save some time for Q&A at the end. And we're going to kick things off with prerecorded remarks from Martin Halbert at the National Science Foundation. I'm Martin Halbert, the NSF Science Advisor for Public Access in the Office of Advanced Cyber Infrastructure. And I'm delighted to share a few thoughts with you today. In this very brief set of comments, I'd like to just observe a few things in the environment and what they are, how I'm tracking them in terms of intersecting trends and questions concerning scientific communication. I'm going to talk briefly about the CHIPS Act, NSPM 33, and the 2022 OSTP Nelson Memorandum. The CHIPS Act was a major piece of legislation that has interjected a very large amount of new research funding into the scientific ecosystem with an investment over a period of years of more than $280 billion in new funding that's going to be made available through NSF, NIST, NASA, and other federal agencies. It is going to have enormous effects in terms of boosting domestic research, especially in the manufacture of semiconductors. It was spurred by global semiconductor shortage and widespread concerns that the United States was falling behind in semiconductors and science generally. Now, the point, in addition to just the basic impacts that this act is going to have on scientific research across the nation over a long period of time, I do want to highlight some parts of the act that are less well known. In addition to allocating all that money, CHIPS made many statements about the importance of research reliability, reproducibility, and public access. I draw your attention to section 10344 and invite you to look at some of the statements there. I'll come back to CHIPS, the CHIPS Act, in just a moment. NSPM 33 is a very complex and somewhat controversial national security presidential memorandum that directs agencies to strengthen protections of US government supported research against foreign government interference and exploitation. Now, it was motivated by concerns that the US government needs to take steps to protect the intellectual capital and discourage research misappropriation of US taxpayer dollars and the resulting research coming out of that. Now, an interesting sidebar to this is an effect it is having of making persistent identifiers, for example, ORCIDs required for individual researchers to obtain federal research grants in coming years. Now, while the overall gist of this memorandum was around protecting US research investments, it is actually having an interesting synergistic sidebar effect of moving forward federal agency mandates around ORCIDs, which is going to have an interesting strong effect on better sharing of research results in terms of tracking researchers uniquely through ORCIDs. And I'll come back to this in a moment. Finally, the 2022 OSTP Nelson memorandum that has gotten a lot of attention recently, which was issued just on August 25 by Dr. Alondra Nelson in OSTP, has obviously made a number of new public access requirements, notably the emphasis on free, immediate, and equitable access to research results. It calls for default zero embargo of peer-reviewed articles after publication together with data undergirding such publications. Now, you've undoubtedly heard a lot about this memorandum, and you're probably going to hear more about it in this panel. So I'm not going to go through the details. Let me just make a few comments about these three developments in the field and how I see them as interconnected in various interesting ways. First, a trend that I'm tracking of renewed attention on the United States as part of the global milieu. The CHIPS Act and NSPM 33 highlight in very different ways the scientific and economic anxieties of the United States with regards to the rest of the world. Now, conversely, other nations do watch developments in the United States nervously at times, anticipating the impacts that developments like the Nelson memorandum may have on science in many other areas internationally. I can attest that groups like Coalition S and Science Europe are anxiously following the developments of agency responses to the Nelson memorandum because they see the impacts of decisions made in the United States on the overall global scientific communication ecosystem. So my questions that I've been thinking about a lot lately is what's the right balance here? To what degree should we seek to integrate or separate US scientific communication systems with or from the rest of the world? My second trend is a renewed attention on open science and public access results of federally funded research, perhaps an obvious one. But the Nelson memorandum has set really a new bar for scientific information sharing. It called out the importance of immediate and free accessibility in order to respond to threats like the COVID-19 pandemic and climate change. So my question here that we think a lot about in the office of advanced cyber infrastructure is what investments and alignment of policies and practices are needed to realize the full potential of open science? I'm coordinating the NSF response to the Nelson memorandum and I'm thinking a lot about this these days. The other thing I'll highlight is just a competition that you may have heard of, the Ferros RCN competition, NSF 22553 that I competed earlier this year around findable, accessible, interoperable, and reusable open science research coordination networks. Quite a mouthful, but I'd encourage you to look at some of the awards that we made in response to this solicitation. Finally, a renewed attention on sustainability of interrelated systems. There are many interrelated tensions in the extended scholarly communication ecosystem that's centered on sustainability today. Research programs, libraries, publishers, and other entities in this ecosystem are struggling to survive in constantly changing circumstances. So my questions that I've been wondering about a lot is how should funders prioritize research investments to maximize social benefit and what are the best strategies to improve scholarly communication systems? These are obviously very broad questions but I wanted to set a kind of a higher 30,000 foot view or to start the rest of the panel's discussions. Thank you very much. And I hope I'll be able to join you for the Q&A. Thank you so much. Thanks, Martin. So Martin is planning to join us for the Q&A. We're gonna hear next from Bob Hanish.  Do you want to? You're welcome to. Well, thank you so much. Let me tell you a little bit about what's going on around open data, open science at the National Institute of Standards and Technology where I'm the director of the Office of Data and Informatics. I wanna talk a little bit about our public data repository, then a little bit more about laboratory information management systems. And again, I thank Cliff for teeing up this presentation for me about metadata capture at the source. An update on our research data framework, which I spoke to you all about a year or two ago. And then in following up sort of Martin's introductory remarks here mentioning the CHIPS Act, a word about data that cannot be open. So for open access, again, the conformance with the OSTP, Nelson Memo, the good news at NIST is that we've just about done all of this work already, already stemming from the Holdren Memo in 2013. And I came to NIST at about the same time and felt that it was really important to modernize the way that NIST was managing its research data outputs coming as I do from the astronomy community where we had made major strides in sharing data from major research facilities. And so we started a program already eight years ago to improve and open up access to the outputs of our research. And I should just note, NIST is primarily an intramural research organization, but we do fund a fair amount of external research as well. In terms of additional work, it's really around the persistent identifiers for the people that we fund and also persistent identifiers for our own staff. We actually passed a policy now a year ago that requires our researchers to have an orchid for all their publications. The CHIPS program is going to tweak NIST in a big way of that huge amount of money that Martin talked about. We expect around 11 billion will pass through the doors of NIST over the next five years. Our annual budget is less than one billion. So you can imagine that that is a huge lever arm on the system and putting it politely or not so politely, it's created kind of a feeding frenzy around the campus. So we're trying to deal with that and at the same time assure a fair life data lifecycle. And then I'll give just a quick update on the research data framework. Our public data repository is located at data.nist.gov. One silver lining of the pandemic was that the amount of data that our staff put into the public data repository increased by an order of magnitude. They had lots of research results sitting in their labs. They couldn't get into their labs. And so they wrote up those results and published them and share the data. Along with data, of course, is the software that is used to analyze it and reduce it. And we now have online our public source code repository which is available at code.nist.gov. We have over 600, might be 700, now public repositories on the NIST GitHub. And we can now, you can search across all of those and also get download and usage statistics, including links to any associated data sets and publications. We've invested a lot in laboratory information management systems at NIST, which is a way to basically capture metadata, the conditions of an experiment, the sample being studied at the birth of that data. So we want to preserve this information, track samples and automate conversion of the data from sometimes proprietary forms to open formats and basically make the case that all data can be born fair. So it helps in the translation of data across space and time from one researcher to the next in terms of publication, but also from postdoc to graduate student, so forth. You don't have to write down on a little sheet of paper that you stick onto a thumb drive. What happened that day, it's all collected for you automatically. We built our own because we found the commercial systems like this are really designed for production work in areas like pharmaceuticals and are not very adaptable to the very dynamic research environment that comes with, in our case, running electron microscopes. I'll mention briefly the research data framework. This is a project that we started in late 2019, is ongoing now, in which we are characterizing the players, the roles, responsibilities in the research data lifecycle. And again, I thank Cliff for teeing up this conversation about preserve and discard, which is of course a very important part of the data lifecycle. And we have been now talking with hundreds of representatives from involved in research data in one way or another, from management to bench scientists, to curators, to librarians, to publishers to understand where people spend most of their time and energy and how we can best help people to understand where they are in the research data ecosystem. We've done a lot of work in the past year. We had 15 stakeholder workshops. We did a lot of restructuring of the topics and subtopics. And we are now developing what are called profiles, which is where you identify your role, perhaps you're a research librarian. These are the typical functions that research librarians have. And here's who you can go to to understand how your role intersects with other people's roles in the system. We hope to bring this to a conclusion by the end of this fiscal year with version two of the research data framework. And the last thing I'll say is about data and the CHIPS Act. Martin already set this up. This is going to involve relationships with the private sector, with industry, and it means that some data will be private or at least restricted. We already have systems for supporting restricted but public data, where you have to basically register with us in order to get access, but we're going to have to look at further steps, such as encryption and integrity validation in consultation with our information technology security experts at NIST. And this is really critical for establishing trust in this research data. Industry needs to rely on us. We need to characterize this data so they know exactly in what circumstances it pertains, that its uncertainties are properly characterized. And we want to be, of course, consistent with fair principles, which are as open as possible, but as restricted as necessary. So thanks very much. Thanks, Bob. And we'll hear from Terry next. All right, is this on? No, yes? Can we use the table mics? Can I, I think I can, yeah. Unless I have very large lungs, great. Good afternoon, everybody. I'm Terry Taylor from Lumina Foundation. And to be honest, I don't spend a lot of time with people like this network. For me, orc IDs are possibly World of Warcraft options. But I feel like we're in a time where building new relationships, understanding what's happening to higher ed as a field, and how we can all sort of work together to sort of develop more strategies. Funders sit in these tensions where we're so careful to design our strategies, to meet the moment, to meet our grantees and other partners. But there are so many externalities and unpredictable events that can disrupt that. And it feels like right now we're living in a particularly disruptive time. Democracy is backsliding globally. Despite the midterms, we still have a rise of white supremacy. We still have a lot of hate happening on campus. We see libraries and educators as battlefields, not as sort of neutral goods. And it's hard for us, for those of us that were trained to be brought up in systems who take a lot of personal meaning in providing access to information, in teaching, in wrestling with complicated ideas, it's hard when that gets politicized. And so I think all of us have skills to build and relationships to build. I'll just offer a couple of data points. 40% of Americans think that higher ed is part of the problem with our country. Most people across the political spectrum think that when faculty members teach a controversial topic, they're mostly providing their own opinion and they're not providing space for debate. We can dispute that, but it's what people believe. So I think it's important to think about some of this as we're figuring out our own strategies. It's also worth noting that 39 million people with some college-known degree, a population that Lumina has spent a lot of time on, that's sort of a democracy problem. That's a societal health problem. And so how do we help make sure people who come through our doors are well-served? And frankly, how do we help faculty and administrators see libraries and other resources as part of the solution, not just the folks doing research over there? If you know Lumina at all, since our CEO came on board in 2008, everything we've done has been designed around a national attainment goal of 60% of American adults having some kind of quality credential by 2025. When that was set, 2025 felt like a long, long time from now and now it's two years away. And so part of what we've been doing is something that philanthropy doesn't often do, which is think about its second act. And so part of my job as the Director of Innovation and Discovery is to think about, we're defining that particularly in terms of what is education for and what systemic risks is our country facing, sort of outside of our core work of increasing enrollment, increasing retention, increasing graduation rates, particularly for students of color and adults and other underserved populations. What are the things that might get in the way of that? Both systemic risks that are threats to our country and to overall systems, but also to our own mission around attainment and equity. We're also trying to figure out how to talk about those risks in terms of opportunity for how we talk about quality, how we talk about equity, how we talk about relevance. I think when we talk about value of higher education, that's more than just how much somebody pays. It's also what is society getting back for all of the investment that we pay. So let me give one example, and I think this is why I was invited to be on this, about divisive concepts. So if you all know, so quick history, the Trump administration in their last six months in office put out a policy saying that the federal government will not pay for any training around certain defined divisive concepts. I can go into that later, but basically it's a list of eight or nine things that then got rolled up into a model legislation that's now been passed by several states. The federal policy went away, Biden's first day in office, but since then I think something Penn America has estimated, it's like 121 million Americans now live in a state where there's some restriction in K-12 or higher ed, and there's eight or nine states now that have specific restrictions about higher ed, what can be taught, and there's also, I'm sure you all know, a bunch of book bans happening. And so all of this really limits what we can do. And to be honest, I am deeply concerned that a lot of faculty don't really understand what's happening. They're getting mixed messages from their counsel, they are reading stuff in the news, and so I think this is a place where again, how do we help people understand what's going on? How do we help make meaning of this and design collaborative strategies a big thing? Lumina's role in this was, I'm a lawyer, I was doing this innovation work and I had this legal lab at the time, and so about 18 or 20 months ago, I was like, well, let's do an analysis. These things have to be unconstitutional. Let's ask.  I will not say who they are. And everyone at first, about 14 months ago, was like, eh, this probably isn't our fight. Like if we talk about it, walnut just add more fuel to the fire. It's only eight states. At the time, it was places like Idaho and Oklahoma. So, please don't do that. But we ended up being able to get a little bit more traction. And partly because a lot more bills got introduced at the beginning of this year, including specifically higher ed. There's been a blurring of boundaries between higher ed. There's a lot more consequences now. One of them has now been knocked out of court. You may have seen coverage of the Florida Stop Woke Act, which was almost so bad that it was easier to challenge in court. But again, this takes resources. And what I've seen in that specific issue, that's not the only issue that we're talking about, but studying, paying attention to it early helped us in a couple ways. It helped us have better conversation across the foundation when we heard grantees coming to us with like, hey, you gave us some funding for equity training and equity decision making in our system or in our college, and we can't use those words anymore. So, or we are trying to figure out, how to support our grantees working with legislators. Are there places where if we do work in this state, we're going to have our own reputational problems because of some of the threats to higher ed. So, it's really complicated and hard actually. And those of us that really put a stake in the ground on behalf of racial equity, I think it's particularly important for us to think about how do we bring all these dynamics together. So, thinking about just how we meet the moment, I think funders can be helpful because we see the field in an interesting way. We have a lot of individual inflows with our grantees. We get to sort of see trends sometimes happening before our eyes. We get to talk to other funders and hear what they're thinking about. So, that kind of helps us identify key issues. I really hope we can work to help build and expand networks. Libraries in my world for a while were like, a partner to reach adult learners in rural areas because that's where adult learners were getting online. We did a little work on OER. But that's an overly limited view of what you all do. And so, I think that there's a question of how do we all sort of work together. There's also funders can give space to try new things. I'm excited to hear about the NEH programs they are announcing. And also really think hard about capacity and who's best positioned to do what now and in the future. I think we also have to think about controversy. It's hard for funders sometimes to be seen as controversial at all. Our boards can get nervous about that. Our grantees can get nervous about that. But at some point, what do we stand for? I think there's also a really important time for us to listen. And not just design strategies in a vacuum or what makes sense with our own stakeholders, but also to listen and try and understand what the field needs and where capacity needs to be created. So I was really happy to hear some of those similar ideas from other funders. And with that, I'll turn it back over. Thanks so much, Teri. Brett, I'll tee up and run your slides if you wanna. I'll come over. It's up to you. If I can't see my slides, I'm doomed. Thank you. Thank you, hey, everybody. Brett Bobly with the National Endowment for the Humanities. I'm the Chief Information Officer and I'm also the Director of our Office of Digital Humanities. I wanna tell you about, you know, I've been with the NEH for about 25 years and this is probably the boldest, most remarkable new initiative we've released in the time that I've been there. Our Chair, Shelly Lowe. This is kind of her signature initiative which we're calling American Tapestry. And as I said, it's a big deal and let me talk to you a little bit about it. There are three pillars to this new initiative. The first one is called Strengthening Our Democracy and it is very much about funding projects that are investigating democracy in the United States in its current state, potential decline of democracies around the world. So not just looking at what happened to democracy 200 years ago, but how does it reflect today's challenges? The second pillar of this new program is called Advancing Equity for All. And this is, again, as you can imagine, it is about funding projects that really look at studying equity, both internally and externally, I should note. So funding projects that are exploring equity in politics, in life, economically, but also internally within the humanities. We're taking another look at what does a pool of our peer reviewers look like from an equity perspective? What does a pool of our grantees look like from an equity perspective? And the third pillar is called Confronting the Climate Crisis. So which I think is about the most direct way we could say that we're interested in projects that are looking at confronting the climate crisis. I say that and I'm only half joking because I think that it's fair to say that in the past, the NEH has kind of shied away from advertising that we fund contemporary projects, projects that are attacking, addressing, contemporary issues. But I think most of you would agree that if you look at some of the biggest challenges in the world today, a lot of them involve humans and they have a social dimension, right? Cliff alluded to some of this earlier, but obviously science can make amazing vaccines, but maybe they can't understand why so many Americans don't wanna take them. We really want to ensure that we are willing to address these sorts of issues. So let me tell you about a couple of the new grant programs, which we have just announced that go along, that fall under this umbrella, is American Tapestry Umbrella. The first one I'm gonna talk about is one that's actually coming from my own office, the Office of Digital Humanities. It's called The Dangers and Opportunities of Technology, Perspectives from the Humanities. So the idea here is that I think that in the digital humanities, I founded the Office of Digital Humanities about 15, 16 years ago, the lion's share of the projects that we have funded have been about turning the lens of technology onto the humanities, right? That is to say, how do we do philosophy research or humanities or history research or literature research using computers? How do we use computers to help us do research? That's the way we've almost always done it in the past. But the idea behind this program is to turn the lens around. We are now saying, what do the humanities have to say about technology? Technology, there's all kinds of problems with technology. You know, a lot of the thorniest problems we have today have a technological element to them. When it comes to things like disinformation, medical research, AI, you know, almost every major thorny issue, cryptocurrency, democracy, there's this huge technological element to it. But it's not just a computer science issue per se, it's not a science issue per se, there's this human element to it. And we want to get scholars from the humanities bringing their voices to the fore, writing articles, writing books, being in the public eye to talk about what are the ramifications to people of these new technologies? That's kind of what this new program is about. Here are just a few random examples of the sort of research topics that we might want people to look at. Again, things like data privacy, wealth inequality, crypto, supply chains, streaming economy, all these things that really have a humanities element to them. But they are also things I think it's fair to say the NEH did not call out in the past. You could always technically come to the NEH and try to get a grant for one of these things, but not very many people did. I think there was a perception in the field that, oh, the NEH isn't gonna fund that, it's too controversial, it's too hot button of an issue. And maybe it was, I don't know, but our new chair under the Biden administration really wants to ensure that humanities research plays a critical role here. And by the way, Bob, if you guys are having any trouble spending that 11 billion in chips money, I'd just give me a call. Yeah, all right, we're good. This new grant program has two levels of application. If you are an individual researcher in the humanities, you can get up to 75K. If you are a research team, you can get up to 150K. And by the way, I should note, you don't have to be a professional researcher in the humanities to apply. You could be a scientist, you could be anybody, but you have to be taking a humanities perspective in your research. Deadline is February the second, it'll be an annual program that'll go on forever or until we are shut down. A few other programs I wanna tell you about again under this umbrella of the American Tapestry program. This is one in our division of education programs. It's called Spotlight on the Humanities in Higher Education. And this is really about supporting smaller organizations. So this is not something for the Harvard's and the Gales. This is specifically we're looking at community colleges, smaller HBCUs, regional comprehensive universities, organizations like that that need extra support who are interested in a variety of different types of topics that are open. This particular program had its deadline last month and the next deadline will be next November. Another new program in our division of public programs is one that'll be coming up in June. This again, this is looking at smaller organizations in the public domain. You know, in addition to funding research, the NIH also funds a lot of public programming, things like museums and historical associations and libraries. And this particular program is aimed at smaller organizations across the country. Our division of research programs has made a few changes, important changes to some of their grant programs. For example, the Summer stipends program, which is a program to help scholars get like two months of pay over the summer to do research. We're making some changes to make it easier for certain organizations like tribal organizations, HPBI's, HBCUs, and others to apply for summer stipends. Similarly, in our collaborative research program, we are making some changes in particular to encourage universities to partner with smaller organizations, including HBCUs, tribal colleges, community colleges, and the other to try to do research bringing in these other organizations that are not as financially solvent as them. Our Office of Challenge Grants has a new program called Climate Smart Humanities, and this is specifically for strategic climate action and adaptation plans. So this might be, for example, an organization that has an archive or a library, maybe in a flood zone. And maybe they haven't had a flood yet, but they realize that in 10 or 20 years, they might. And this is a program to give them money to bring in consultants to help put in a long-term plan for preserving and addressing those sorts of issues. Our Preservation and Access Division has a new program called Cultural and Community Resilience. And this is about specifically looking at both COVID-19 and climate change and helping to document and safeguard the culture of people around the country. So as you can see, this is pretty unprecedented for the NEH to release like six new grant programs in this course of a year. It's kind of a big deal for us. So a very wide variety of things there. I'm gonna stop here. I'll also note just for the record, since there's been a lot of talk about the Nelson memo. In my role as CIO, I'm also sort of one of the point people for the OSTP memo, and we're looking at how the NEH will be able to comply with and take part in the open science arena. So those are some things I'm working on. And that's all I got. Thank you. That's awesome. Thanks so much, Brett. Patricia, I can pull up and run your slides if you're ready. If you want to start with Shelly Lowe. Oh, sorry. Sorry. Well, it's always a pleasure to be on the same panel as you, Brett, because you are a hard act to follow. I know the NEH is doing some really cool stuff under the leadership of Shelly Lowe, and so it was really inspiring to get that update. I'm Patricia Sway, the program officer of Public Knowledge at the Mellon Foundation. And I'll just be providing a high level overview about the foundation and our program, especially its strategies. If there were slides, we would be advancing to the next one, which would show a quotation that is on our website. Mellon is the, this is not the quotation, so you were supposed to read the quotation and I was supposed to give this script. Mellon is the largest private philanthropy addressing needs in the arts and humanities. Since it was established in 1969, we have made more than 18,000 grants, totaling $8 billion in support of the arts and humanities. To get a sense of the grants we have made, you can do searches in our grants database at our website at Mellon.org. On the, okay, so slide three, please, Catherine. Thank you. The arts and humanities are expansive domains, so it may not come as a surprise that we have several program areas that focus on specific areas in the arts and humanities. Through its grant making, for example, arts and culture, bolsters creative practice, scholarship, and conservation practices, while also fostering an arts and culture ecosystem that is representative of the wide variety of artists and organizations represented in the field. Higher learning is the program area that works with colleges, universities, and other organizations in higher education. Among their goals is to increase our understanding of American history and culture, develop the tools and methods for interpretation that researchers can use to create meaning and design opportunities for those seeking to lead these institutions in transformative ways. Humanities in Place is the foundation's newest program intended to amplify and care for the places and cultural sites and structures necessary to a fuller, more complex telling of American histories and lived experiences. And finally, public knowledge. My program supports those who steward and make available to the public the knowledge embodied in humanities, scholarship, and cultural record. We care largely about the stuff, the primary source materials, for example, that enable the evidence building and interpretation that much about teaching, research, and learning and the humanities pivot on. We care about the infrastructures and tools that support access to and use of the stuff. And we want to ensure that there are partnerships and collaborations to help ensure, to help make sure that the created infrastructure is robust and able to endure. Next slide, please. So in addition to our four program areas, our president, Elizabeth Alexander, has created two initiatives that her office makes grants to support. One is an effort you may have heard of called the Monuments Project, which is a $250 million grant-making initiative over four years to transform the nation's commemorative landscape through public projects that are representative of the multiplicity and complexity of American stories. And then the other initiative is for Puerto Rico, and that initiative is accomplishing three things. It is creating sustainable cultural opportunities for artists, curators, educators, and other culture workers. It is building capacity by strengthening local and diasporic arts, culture, and humanities organizations, and it is helping to elevate voices and fortifying networks by lifting up the scholarly and cultural contributions of previously marginalized voices in Puerto Rico. Next slide, please. So prior to 2020, public knowledge was the Scarlet Communications Program. That year, the foundation adopted an explicit social justice framework to our grant-making, which essentially means that we aim to support those efforts that are solving problems of inequity in order to create a more just and fair society. And for public knowledge, this change has meant a broadening of the types of organizations we make grants to and the communities that these organizations help nurture. Next slide. Before I proceed further about the program, I did want to give you a little list of the people who work in this program. So Michael and I are primarily focused on the pre-award side of grant-making, while Celia and Chandra are focused on the post-award side of things, and we have a research associate, a new role, who through her assignments and projects is helping the program get a finer understanding of gaps and needs in the sectors we support to inform our social justice grant-making. And that's Julia. Next slide. The public knowledge program cares about equitable access and on the other side of that coin, access that may need to be controlled for cultural sensitivity purposes. And so to this end, we make grants in accordance with three strategies that are listed at left in order to solve problems and challenges that resonate with areas and activities on the right. And I did go there with the word cloud, which I know is a rather dated way of representing information, but I did want you to see that there are emphases that the program is addressing. Next slide. Through our grants addressing preservation, for example, we strive to ensure the authenticity and value of original sources in the process of documenting and preserving them in all formats, including web-based content. So web archiving is still important to the program with focus on materials from historically underrepresented cultures and populations. Next slide. Through our grants that address maintenance and sustainability, we aspire to maintain and improve the technology, the tools, and infrastructure needed for the public to readily access knowledge, resources related to the foundation's social justice mission. It's important to us that while we are continuing to innovate with new tools and new technologies, that we also pay attention to those tools and technologies that have been existence for a while and have communities that use them and need them. In addition, we're concerned with financial health and resilience for organizations that are central to the sectors that we care about and that our grants encompass. So to this end, we've made change capital grants to several organizations, cultural heritage organizations in the last couple of years. Next slide. Through our grants that address networks and sharing, we aim to establish and strengthen interdependent networks and services for exposing and sharing hidden or little known information, resources, and collections. And this strategy shows our commitment to nurturing collaborations and partnerships when it comes to solving system-wide problems because we know distributed solutions are likely to last longer than standalone ones. Next slide. So about our process, Mellon advice grant proposals, but there are a variety of ways in which we've also been lowering barriers for our grantees, and some of them are listed here. So we do have an online inquiries portal where you can submit ideas for consideration. And from that point, if we accept the sort of two paragraphs of I'm expressing your ideas, then we may invite a concept note. We hold open calls from time to time. Public knowledge was, when it was the Scarlet Communications Program, conducted the Foundation's first open call for proposals, and that was to community-based archives. We've also held what's known as a limited submission concept note call and a higher learning program experimented with this year, focusing on sending out this call to hundreds of institutions, asking them to submit calls or concept notes that addressed either civic engagement or race and racism or literary imagination and social justice. And it was quite successful in terms of the submissions that they received. We've done something similar in the past. We have collaborated with the Arts and Culture Program to do a similar call to academic institutions that have campus museums and libraries and that share mutual challenges or that have mutual challenges in need of solutions. In addition, we've been doing a lot of outreach and engagement to new organizations that include public libraries that are embraced by the tribal library community and that are nonprofits that are infrastructure-focused and community archives and so on. Next slide, please. So lowering barriers for grantees means being transparent about our practice and our processes. If you go to the Mellon website, we now have a section that details our proposal process, complete with this flowchart graphic. Next slide. And a final point to make is that the foundation revised its proposal guidelines over the summer so that the guidance across all program areas are similar rather than starkly different. And to save time, I won't go into details about the changes, but I will say in sum that the various mechanisms that Mellon is using to attract proposals have increased in complement to the convention of inviting them. We've become more transparent about our process and we've streamlined the proposal development process so grantees are not burdened with excessive document requirements and excessive information requests. To give you an idea of where we have streamlined, we also now have page limits on our proposal narrative. So for smallish grants that are in the range of $35,000 to $150,000, we require a proposal narrative that is only two to three pages for grants that will be considered by our Board of Trustees. That range is like five to 10 now. So that is what I have to say about our program and thank you so much. Thank you, Patricia. And Daryl, we'll give us the last word. Well, thank you very much for joining us this afternoon and thank you to Catherine for the invitation to serve on this panel. I've really appreciated hearing the presentations by my colleagues. I'm Daryl Meadows, I'm with the National Historical Publications and Records Commission. For those of you who are unfamiliar, we are a small but I say mighty funding agency associated with the National Archives. We were created right alongside the archives in 1934 and we've been making grants since 1964. You can find us at archives.gov forward slash NHPRC. I wanted to talk about sort of where we are as an organization. We're very much in a learning and reflective mode. We fund primarily on the record side, that's the R in our acronym, small and large archives who are largely processing archival collections, making access happen in various kinds of ways. On the publishing side of the house, traditionally we have funded what I would call historical edition projects, like the Founding Fathers Projects, which some of you may have heard of. So we've been making grants along these lines since 1964 on the record side since the early 70s. And recently, I think we have begun to really think about our grant program portfolio and the degree to which it is or is not meeting our equity goals that are embedded both in our strategic plan, but as well sort of charged to us from the presidential order on racism and federal government. So I wanted to talk about a couple of areas that we're currently engaged that are informing very much where we think we are headed without sort of previewing answers. We're seeking opportunities to listen. One of the ways in which we are learning tremendous amount is through several, actually four programs that are very explicitly trying to foster what I like to call mutually beneficial and respectful collaboration. So on the record side, we have what used to be called the Major Initiatives Program, it's now called the Major Archival Collaboratives Program, which fundamentally seeks to fund collaborations with an eye to digitizing and making accessible important collections that are geographically dispersed that are not easy for researchers to get to. We think there's a lot of work and thinking around creating virtual collections that are important to be thinking about, and it's not just the technological infrastructure for that, but in all of these programs we're concerned about what is the human infrastructure that we're fostering here. Another program called Archival Collaboratives is a smaller program that has a planning track and an implementation track. When we first conceived of that program, we thought maybe it would work in a regional sort of way where maybe an R1 would take the lead and collaborate with smaller institutions. For those institutions perhaps to have access to digital infrastructure that they lack, to gain expertise, perhaps training, to be able to do a number of things that smaller institutions cannot readily do on their own. I will say, but that's not exactly what we're seeing. We are seeing really interesting collaborations come together, but they tend to be collaborations among smaller institutions. There is room in this program to see some R1s and other sort of better resourced institutions get involved, but I think there's just interesting things that we're seeing in learning through this program. On the publishing side, much of my own thinking in this area actually comes from a program that I developed and that is funded through the Mellon Foundation that's a re-grant program called the NHPRC Mellon Planning Grants for Collaborative Digital Editions. That program explicitly is about fostering collaborations that are about creating robust edition projects that center historical materials that tell the stories of African-Americans, Latinx communities, Native Americans, Asian Americans and broadly the field of digital ethnic studies. So through these kinds of collaborative programs, we are seeking to foster whenever possible collaborations that help to bridge longstanding institutional inequalities by promoting resource sharing and capacity building at all levels. In the planning grant program that I'm overseeing, I've now spoken to somewhere in the order of 30 to 40 potential planning teams and many of the folks that I'm talking to are at small institutions. They are at community centers that have because they are a trusted place, they have in effect become archives, but what I'm seeing through these conversations is frankly a whole host of need that we I know in our programs are not currently meeting but that raise really important questions for us to be thinking about. So one of the things we also are trying to do through these collaborative programs is to encourage projects that build in meaningful community and user engagement into their plans. In a lot of my conversations with folks perhaps at a leading university somewhere, someone in the library or scholar at that university is proposing a collaboration with particular communities. We want to see those applicants engaging with those communities from the very beginning. So through our interactions with them, we're trying to stimulate this kind of collaborative behavior. The other area that I wanted to talk about is so the Mellon program came along a little bit for this, but the president's executive order which mandated all federal agencies on the executive side to create what we are calling at NARA, our equity action plan. Those plans were approved by OMB and the White House last April and they were made public. NHPRC played a pivotal role in the creation of NARA's equity plan and I was very proud to serve on that committee and work toward those goals. For NHPRC, the outcome of that was two very explicit action items, one of which we have now more or less completed. That was an action item to engage in listening sessions with practitioners of color, diverse folks across the United States in a variety of organizations and institutions to really put on the table, how can we ensure that our professional development programs are serving a more diverse audience? So we have two longstanding programs, one what was traditionally called the Editing Institute or the Institute for the Editing of Historical Documents which goes back to the 1970s and the Archives Leadership Institute which we've been running since about 2008 or 2009. In both of those instances, we heard from archivists of color and scholars of color, really important insights around our grants, the way they're structured, the way they're advertised and thinking about these particular professional development programs, frankly whether or not folks really felt at these programs for them. So that process which took place over about the last year and a half or so, really informed some rethinking. We are now entertaining applications for both of those programs for their next iteration and we are looking forward to seeing those programs in their new transformed state in 2024, 2023 and 2024. And finally, and this is a bit of an appeal to you as well, if there are folks that you know and you're engaged with who are similarly engaged in this kind of equity outreach, I'd love to hear from you. Our second action item is really just in the planning stages. We're gearing up for what we're calling a program-wide listening tour. We expect to sort of learn from the process of that first action item to really refine and be a little more thoughtful about the rollout of this. So we're just really beginning to think through how we want to plan this out. But we're expecting that at the end of this process, perhaps a year and a half or so down the road, what we will learn in this process, we expect to shape discussions with our commissioners, with staff in our thinking and decision making around our entire portfolio. Do we need to substantively revise programs? Do we need to create new programs? What about archival training and other kinds of training support for community archiving? For projects that are linked to community scholarship efforts. What's our role in that? We expect that this effort will push us to be even more reflective than we've been in the last year or so about our application process, about our peer review and its purposes, whose interest is it serving? Is it serving a gatekeeping role? And in general, the role of program officers in really providing upfront advising to folks at smaller institutions who've never before attempted to apply for federal funding. And it should really revolutionize our outreach and communication strategies. So whether through new or substantively revised programs, how we do, how we better serve and support diverse archival institutions in communities of practice, including, again, folks in the emerging field of community archiving whose historical records and access to them are essential to those communities as well as to telling more complete American histories. Thank you. Thank you so much. And we are right on time. And I think there's a break next, but if folks are willing, I think we're maybe willing to hang out for a question or two. Of course, if folks run for the door, that's our signal. We can wrap that up. Does anyone have a pressing question? I think panelists included contact info in their slide decks, which will be made available as well if you wanna reach out directly. Then please join me in thanking the panelists and CNI and the tech heroes in the back. Thank you guys. Thank you all.


Thank you for coming to Retrospectives for Humans. Before we begin today, I want to acknowledge that this has been a really frightening and difficult few weeks for a lot of people, with tragedy and violence touching a lot of people that I know. And if that's you, I'd like to offer my sympathies. It can be difficult to function normally and pay attention to everyday business at times like these. It can be even harder to learn new concepts. So please do what you need to do during this. Take care of yourself. I don't think this talk requires any content warnings. If it does, and I've missed it, I apologize, I would appreciate hearing from you afterwards. All right, I'm here to talk about retrospectives. These are incident retrospectives, but most of the things I'm going to say today will probably also be useful to you in an agile retrospective, should you choose to do that. The subtitle of this talk could also be words mean things. Let's talk disasters where no one was hurt. Seattle has the two longest floating bridges in the world, and in 1990, one of them sank over the Thanksgiving Day weekend in a storm while it was being repurposed. The public reporting to this day is extremely simplistic, but the official investigation found that there were five factors involved, and all of them were required for the bridge to fail.  They were going to change the direction of traffic on the bridge because they were building a span nearby, so they were going to expand how much traffic the bridge could take. The EPA told them that they were not allowed to let the water run off into the lake, that that was a serious environmental problem, so they had to figure out a way to store the water so that it could be trucked off and treated. You can't store water in piles, so they needed to find a container to put it in. The thing that occurred to them after much study was that they would store it in the hollow pontoons of the bridge itself. Floating bridge is made of these hollow sections like a ship's hull, it's made out of concrete in this case, and this bridge was actually floating higher than it was designed to float, and had lots of extra floatation ability that they decided to use in order to store the water so they could truck it away. When this happened, no one was on the bridge, it was, over the holiday weekends everyone was off, and one pontoon sank slowly and dragged seven more down after it, so this was a slow cascading failure, extra fun. This is a video of some of the coverage of what happened. I'm going to play it for a few minutes. In 1940 in Seattle, a Lacey V. Murrow Bridge was named after the director of Washington State's Department of Transportation. The bridge stood for 50 years, but in November of 1990, a rehabilitation and maintenance operation on the bridge led to disaster. In a short but deep section of Seattle's Lake Washington, the Murrow Bridge floated on a row of 25 hollow concrete pontoons, each 350 feet long. The pontoons were bolted together for stability and anchored to the bottom of the lake. The conditions are just right for use of such a structure because the water conditions under which they're installed are perfect. It is very seldom is there any ice, very seldom is there a heavy current, and the water is very deep, 200 or more feet. In the fall of 1990, the Murrow Bridge was undergoing a facelift by means of hydro demolition, a process in which highly pressurized water is used to break up the roadway. Since environmental regulations prohibited dumping the contaminated water into the lake, it was temporarily stored in the empty pontoons via access holes which were cut into the tops and sides of some sections. The water level inside the pontoons was carefully monitored so the structure would not be compromised. On Thanksgiving weekend, with bridge crews shorthanded, a heavy storm hit the Seattle area. Rainwater drained into the open pontoons. Winds from the storm whipped up the surface of the lake, propelling splash water into the pontoons as well. On November 25th, eight waterlogged pontoons sunk into the lake, taking those sections of the bridge down with them, and severing 12 anchoring cables of a bridge that was being constructed nearby. Luckily, no one was injured. The cost of damage was estimated at $69 million. An investigation of the accident came to a sobering, if unsurprising, conclusion. After a very extensive study and very sophisticated computer modeling methods, the conclusion was that concrete pontoons simply don't float when they're full of water. A very unsophisticated motive. The Lacey v. Murrow bridge was rebuilt two years later. Though it looks the same from the outside, the pontoons that support the bridge have been divided into smaller sections to minimize damage from leaks, a precaution that underscores the structural similarities between floating bridges and floating ships. The pontoons were designed with multiple cells, not unlike the multiple cell super tankers, so that failure in one place would not bring down the entire pontoon. So he got the laugh he was looking for. Step in here. That's not what I wanted to have happen. Apologies. Where is my mouse? OK, so this was a giant disaster. They checked a lot of stuff. I'm not even going to read all of that off to you. It was an incredibly thorough investigation. I was able to get a paper that was published by the people who fronted this investigation and read the original text of what they determined the failure to be. This part I am going to read. This is what they found was the problem, several problems. The loads that created significant leakage were the combined effects of all accumulations of water, including rain after the windstorm, longitudinal flow on the surface of the bridge, and pumping through November 24, 1990. These loads caused static moments, that's to say, pressures, that exceeded the threshold for leakage. Existing cracks were open sufficiently to allow water to leak into the pontoon. Progressive and accelerated sinking began at this time. So there's a lot of things that we've got going on here. We found that there were existing cracks in the bridge. The survey did, in fact, find that this bridge was built in the 40s, 40s, 30s. He told you at the beginning of the video. Now I can't remember. So the bridge was in excellent shape for its age. It had cracks that were expected in concrete of that age, but not as many as they would have expected. The cracks were smaller than expected. The bridge was floating higher than designed. That's the extra flotation capacity that I talked about before. So the bridge was not just in decent shape. It was in spectacular shape. It was dramatically over-engineered. And so they found they needed to put the water someplace. They did very careful surveys of the bridge and how much flotation there was, and they calculated an error, like a buffer for their error, and worked really hard to make sure that they were being careful when they did this. It was very thorough. But still somehow this happened. The official investigation missed something huge. The crews were having trouble pumping the water out on the schedule that was set by the engineers. And there is a mention of that in the paper, but there's no explanation as to why. I don't know if they didn't do an investigation or if they simply didn't document the result of their investigation. But there was a social problem here. There was a schedule set. It involved trucks that were pumping water, carrying it off the bridge. It involved water being stored evenly, hopefully across the pontoons that they were storing it in. And for some reason, the crews couldn't stick to that schedule. We don't know why. And so we don't truly understand what happened here. We know that the water was higher in some pontoons and lower in others, which was causing torque on the bridge, which helped open those cracks. We know that the trucking crews were behind. There could have been not enough trucks. There could have been not enough drivers. There could have been a strike. There could have been pay problems. Any number of things could have gone into this and we're never gonna truly understand. So as thorough as this was, as really good as the true investigation was, and as different as that is from the soundbite that made us all laugh, we still don't truly understand what happened here, which is just such a good representation to me of the things that it's easy to miss in a retrospective. So we all laughed.  But no one does things that they think are going to blow up the world, right? It's easy to look at that and be like, oh, you stored water in the bridge. Oh my God. But they stored water in the bridge very carefully with a very firm plan for how they were going to continue to have this be okay. And it still didn't work. So the lesson here is, remember that even as you have that feeling of like, you know, face palm, here we go again, it's probably more complicated than that. Asking about what a person knew and what the conditions were at the time that they made a decision will give you good insight for effective change. So let's do better than that. Now this is gonna be a crash course. I condensed like double the amount of content into this time. So we're gonna run fast. If you have more questions, come find me afterwards. Facilitation, we're gonna talk about how to have a good meeting. Other ways to think of this, servant leadership, psychological safety. As a facilitator, your job is to keep the conversation blame free, not let people make bad jokes and to run a pretty decent meeting that people are willing to sit in instead of checking out of. So let's talk about how we talk about all of this. First thing, foundational Miller's law. In order to understand what another person is saying, you must assume that it is true and try to decide what it could be true of. You're being asked to imagine what it is like to be another person, probably at another time, at another place, in another context. This is the only way to truly understand a system or a person or an environment or a happenstance that was not one that you personally lived through or even one that somebody else lived through next to you because no two people ever have the exact same experience even in the exact same moments. I learned about this from Suzette Hidden Elgin in the general art of verbal self-defense which is a fairly, yes, which is a fairly old book at this point. I do recommend it, it talks a lot about language and how we talk to each other and the ways that we can create conflict and de-escalate conflict. This is a critical tool to use when you're conducting a retrospective. So keep this in mind, started here, everything else is gonna build on this. Now let's talk about blame. English is a pretty blamy language, right? I might say to myself or to another person. You knocked over that face. Why did you knock over that face, right? Those are both statements that are about blame and fault.  Don't just walk away from me. That feels awful, right? That's terrible. Starting a sentence with you draws a line between yourself and the person you're speaking to. So does using it too often. It creates an oppositional conversation. There's the person saying you and the person who's being you'd at. Why? Why is another part of blame? Why did you do that? Why did you do it like that? That also feels terrible. Asking a question that starts with why is a request for a justification. It immediately puts the person you're addressing on the defensive. Why questions get answers in a attentive language because I knocked it over? The face is broken because I knocked it over. A attentive language has blame in the very grammar. It talks about a person doing a thing. It's strongly remembered by English speakers. So then, after you've asked this question and you've gotten this answer, you've ensured that everyone who heard it remembers that I knocked over the face. Other things, not to say to people, especially in a retrospective, but often at other times too.  Every time we do this, you. You should. You should just. You should only do it like this. There's more, but these are the least wanted. If you have a sibling, anything your sibling ever shouted at you when they were mad, that made you see red, goes on this list. Anything that your child, if you have a child, has said to you that just made you wanna completely blow your top because kids are really good at that, goes on this list. You might, if you're talking about an API or an RFC, use these words, especially like RFCs have this very prescriptive language. Sometimes we can't avoid it, but remember that when you describe the behavior of a system, you are describing the decisions that some people made when they built it, right? It's more upsetting if those people didn't know the consequences of the decisions that they were making, which they didn't, no one can be perfectly prescient. So you are saying things about decisions that people made that they probably didn't even understand that they were completely making. Why didn't you just fix it the last time this happened? How many times have we heard this, right? This is terrible. I'm gonna take this apart because this is not compassionate and it's not effective at helping us understand the problem. This construct assumes that it's happened before, maybe more than once, that it could have been fixed the last time it happened, easily, that's the just, that you was the right one to fix it and that there was no good reason not to fix it. That's a lot of assumptions in one very short sentence. Here's some better things to say. How could you or how could we? What, what happened? What do you know? What did you know? What if, what if we tried this? What if we tried that? Could we next time try this? What do you think about this proposal that I have? What would you have wanted to know in the moment where you did that and you made that decision? All of these are designed to get more complex answers when used in a question. When you're asking a question, think about what you can say that will get you the longest answer. It's a habit that a lot of us get into in business communication, right? We try to be short and declarative and stick to the point, especially when you're sending emails to an executive who only reads three sentences. Sometimes that's your manager. It's not my manager, but sometimes, right? So we get into this habit of being really concise and that is an extremely useful habit in a lot of circumstances, but in a retrospective, you want to ask complicated questions that get complicated answers. This is a creative process, both for you and the other people who are participating. One of the things that all of you are trying to do together is imagine a better world. When you ask these better questions, what you're looking for is called a remediation item, an idea for a change that we can make to code, to documentation, to playbooks, to processes that will give a different outcome in the future. At Heroku, we call this contributing factor discovery because complex systems have complex failures as we saw it with the bridge, right? There's no one specific thing that caused that bridge to fail. And if we talk about a single root cause, that constrains our thinking. It keeps us considering that there's only one reason that something broke, which is never the case. We don't deal with spherical cows on frictionless inclined planes. We deal with the real world and it's very complex. We often hear that something went wrong because of human error, but human error isn't where you should stop your investigation, it's where you should start it. People make the best decisions that they can with the information they have at the time. So how did the human make the error? What allowed the error to happen? How did the error take the system down? How long did it take the human to notice the error? The answers to those questions can give you things that you can change so that next time if the exact same thing happens, you'll get a different result. Fallacy, well now that we know about this kind of mistake, we can just not do it again. That doesn't work. Who's gonna get more sleep next week? We all think we're going to, are we? Probably not. Depending on humans to avoid errors is unreasonable. People can't be perfectly vigilant, they can't be better arrested, they can't just like not have a screaming baby next time. They can't have a cat who's like not trying to climb on them when that's what he always does every night.  Okay, let's talk about how to have a great meeting. Who's with me? All right, if you're facilitating, have an agenda, stick to it, stay on time, stay on topic. This helps keep you from missing things by running out of time. It lets participants know what you'll cover and when. It helps participants stay with the group because people who have fallen behind are self-conscious and are less likely to participate and you won't be able to hear what they had to say about the incident and their perspective on it. Digressions can be boring for the group or can inhibit group participation, especially if one person is filibustering or has that hobby horse that they drag out every time. That discourages other people from sharing. Staying on track helps make sure everyone stays engaged. If you're a participant, your job is to help the group stay on track. Before you share something in a retrospective, ask yourself if what you have to say is on topic and needs the entire group's attention. If one of those things is not true, save it for a different context. Find a different way to bring it up. Try to work with the facilitator to keep everything moving forward. Keep your eye on the agenda. Stay engaged. Try not to respond to people slacking you. It's really hard, but try. Even if you have nothing to contribute right now, because if you are paying attention to someone else and what they say, you might have a response that moves the conversation further along. It's not just about what each particular person says. It's about the interaction between the participants and the conclusions that you can all come to together. Finally, note-taking. I don't have time to go into my very long rant about note-taking. Try to make sure it's fair. Try to rotate it. Pitch in if someone else is taking notes in like a Google Doc or some other kind of shared format. Try to make sure that it's not always on one person and it should never be on the person who's facilitating because you can't pay attention to everyone else in the room and be taking notes at the same time. It's not a possible task. As a facilitator, you're gonna need to practice interrupting. It's necessary when people start to rant, complain, get off topic, it can be super, super uncomfortable. Remind yourself that you can't achieve the goals of their retrospective if behaviors that take the meeting off track go unchecked. The best way to practice this is to think about a time that you experienced when someone needed to be interrupted and they weren't. Remember how unpleasant and frustrating the result was. Everyone's just sort of sitting there, seating, you're pretty sure everyone else around you is really unhappy too, we're not getting anywhere. This person is like taking out all the air out of the room and we're not coming to any useful conclusions here. Remember that feeling wherever it feels in you. Then imagine yourself in that moment having said, thanks, teammate, but we're really far afield and we need to come back to the topic so that we can finish on time. Or everyone working this incident did the best they could and we're here to figure out what we need to know to help things go better next time. Or even, wait a minute please, if you need time to think. Do this gently but firmly. The person you're interrupting may even come to you and thank you later. That happened to me with a large billing outage that we had a few years ago and the senior exec for billing was really upset and I had to shut him down. He came to me afterwards and said, I was really angry because I didn't realize that you were actually on it. I thought that because everyone was so calm and so good at this, it meant that you didn't care and I was wrong so thank you for telling me. It's not always gonna be like that. But sometimes it is. As a facilitator and as a participant, look at who's talking and who isn't. One of your jobs as a facilitator is to make sure everyone gets a chance to speak even if they choose to decline that chance because they don't have anything to say. Don't assume that just because a person is silent, it means they have nothing to contribute. They might be shy, they might be having a hard time interjecting, they might be having a bad day. Keep an eye on the participants to see who might want to speak and be afraid to interrupt. Things to watch for. Wiggling, leaning forward, looking lips, biting lips, waving their hand a little bit, lips pursed as if they're ready to speak, frowning, nodding, other really engaged behaviors. Sometimes those are the behaviors of someone who is excited to say something next. Sometimes it's active listening and you ask them if they have something to contribute and they don't in that moment, that's also fine. Try to take time, at least twice, during your retrospectives to stop the discussion and invite anyone who hasn't spoken to speak up. When you do that, you need to wait. Wait until it gets uncomfortable and then wait a little longer. Most times when people ask, do you have anyone have any questions, anyone have anything to say? They wait, okay, let's go, right? And that's too quick. It's too quick for someone who needs time to marshal their thoughts or to marshal their courage or whatever it is that they need internally in order to be able to speak, you need to give them time for that. Okay, let's talk about humor or we could not talk about humor but let's talk about humor. It's really easy to mess up when you're making jokes in retrospective. You might have heard the saying, comedy is tragedy plus time. In a retrospective, there has not been enough time. Furthermore, someone, maybe everyone in that retrospective feels like they screwed up and they cost the company money. They cost their teammates time and maybe sleep and they made everyone present have yet another meeting and maybe even their manager is there. So it's critical to avoid making people uncomfortable by making jokes that are inappropriate in this context. Jokes you should not make. Anything that makes anyone in the room feel even a little bit uncomfortable even if they don't really notice it or they won't share it with you, right? A lot of times when people are uncomfortable they're not ready to share that they're uncomfortable with the person who is the proximate cause of that feeling. Bad jokes. Anything your parents say when they're mad or they think you haven't been home recently enough. Anything you say back. Anything a manager or other employee ever said to you that made you feel bad. Anything you ever said to a co-worker that made them laugh uncomfortably and maybe once a little bit. Jokes about getting fired or firing people. Really a bad plan. Jokes about politics, religion, or current events. Here's some things I shouldn't have to mention and I'm gonna say them anyway.  Jokes about disabilities, any joke that contains the word crazy. Jokes about any one person or their decisions or their wisdom or their plans or their face. You can probably make jokes about Murphy's law, entropy, computers being terrible. You can be really serious about silly things. One of my favorite examples is Las Vegas, colloquially known as Fabulous. Maybe you can make a joke with a person about something that they are proud of about themselves but this is tricky. If you knew that I liked making jokes about myself as the girl with her hair on fire, which I do, you might be able to make that joke. It might be okay. The issue with that joke is that if somebody else in the room doesn't know that I think that's fun, it can make that person uncomfortable and so I would usually still try to avoid it in a retrospective. And definitely, if you didn't know that I made that joke about myself, don't make it. If you can't make jokes, how do you listen to the mood? You can be kind, thoughtful, caring, call out successes. You can thank someone for their honesty and their contributions. You can be warm and welcoming. So what about when things go wrong? What do you do when other people blame or make bad jokes? What about when you make a mistake as a participant or a facilitator? If you mess up, you can tell you've made someone uncomfortable, you made a bad joke, you said something, blame me by accident. Apologize, correct yourself and move on. It's important to be sincere, but matter of fact. Don't take time to wallow. Don't be performative. Just say, I'm sorry, let me rephrase that. Let me ask you what you would've liked to know at the time and that's it. If you take time to feel shame and embarrassment and guilt in the retrospective, you're taking everyone else's time to feel that with you. It takes away from the mood of gentle inquiry and curiosity that we're trying to have. It's not effective if you're apologizing in almost any context to put your guilt out to the person who's asked for your apology or to the person to whom you are apologizing. It's much better to process that yourself with close friends, with your manager, with your mentor, whatever and keep the meeting moving. Sometimes people will blame themselves and you might need to intervene. This is particularly hard. You just need a short request that they stop and then you need to move on. If the person is embarrassed or wants to explain, if they're having that moment where they want to have their guilt with everyone, give them just a moment, thank them sincerely and then move on again. Try things like this is a blameless retrospective and that means not blaming anyone, including yourself. We're here to make sure no one gets blamed, including you. Please don't blame yourself. That's not what we're here for. Critically, you don't have to be witty. You don't have to be clever. You don't have to be on top of your game. You just need something very simple that asks someone to do something different than what they just did. Don't try to win, don't try to one up anybody. Your goal is to remind people of the ground rules and keep things moving. If someone asks a blaming question, remind them that we're trying to avoid blames, this is a blameless retrospective. Can I help you rephrase that? Here's how I would suggest that you say what I think that you meant to say, does that sound right to you? And keep it going. Someone makes a bad joke, this is all you gotta say. Please don't make jokes like that in a retrospective. Very simple, move on. You don't have to define it. You don't have to have a conversation about what the boundaries and the edges are and was this inappropriate or was that inappropriate. All you have to say is please don't make jokes like that in a retrospective. During the retro is not the time to have a teachable moment or to try to stage an intervention. You don't need to convince anyone of anyone, you just need them to stop as quickly as possible so that everyone else can get back to business. If their joke is something like I fat fingered it, again with a self-blame, you can gently say one of the things I said before or you can just ask them to not make a joke like that in a retrospective. That's also reasonable. Once you've gotten them to stop and the group has moved on, don't bring it up again. It's done, as far as you're concerned, it's finished. We never need to talk about this again. If they come to you and ask why it was inappropriate, you can choose to explain if you feel like you're in a place where you can have that conversation or you cannot, you can just reiterate that it's not appropriate for retrospective and ask them to please not do it again. All right, so what's the point? Your retrospectives are part of the communication structures of the organization which produces the systems that you're running. It's turtles all the way down. Probably many people in this room are familiar with it but I didn't know that Melvin Conway is still alive. I didn't know that he coined this in 1968. This is just such a brilliant insight to me that you can look at a service or an application and you can see the fault lines and where people didn't know how to talk to each other and where they weren't able to figure out what the UI should look like compared to what the API looks like before it all went out the door, right? So retrospectives will also, the results of your retrospectives will be reflected in the product that you produce. Let's go make bigger, more interesting mistakes. Some acknowledgments, I won't read them all out. I am indebted to a lot of people for a lot of the sort of linguists and social concepts in here, some of the technical concepts but also in how to teach and how to address people when I'm trying to show them and not just tell them what I'm trying to do. Thank you all for coming. It looks like I have a little bit of time for questions. Okay, so the question is, how do you approach it when someone is senior and they are blaming or making uncomfortable junior folks in a retrospective? Have I got that right? Yes, okay. So that sounds to me like the kind of cultural problem where you need probably an executive sponsor, right? You need someone near that person's level who is on board with the idea of a blame-less retrospective who can talk about it and who can sort of help you spread that word to those folks. Sometimes depending on the relationship that you have with the person who's doing the blaming, you might be able to take them aside afterwards and talk to them yourself. There are some people in my company where I think I have a good enough relationship that I would be able to do that but it still always helps to have someone at the high level who is on board with what you're doing and can help evangelize. That is super hard.  How do you interrupt someone who does not take a pause between their sentences? It's super hard. What I will do sometimes, all of our retrospectives are on Hangouts because we're a remote company. And so I will try to break in a couple of times. In some ways, I have a slight advantage in that I'm usually the only woman in the meeting and if I'm speaking as the facilitator, like I don't have to remind them why I'm speaking, the sound of my voice reminds them. As a facilitator, you might have to remind them. If you try and break in three times, you can't get in. I have not yet gone to the point of non-consensual muting but that might be a thing. In a physical environment, I might even stand up and sort of physically take the floor and say the thing that you need. Please save that for later. Okay, so the question is, where do I find the knowledge base? I am not completely sure. If you come up to me afterwards, I can give you my card and I can give you the slides for the longer version of this, which has at least some of that and that offers for anybody in the room. I don't know, but I also haven't been watching like what other people have written. So there might be more out there that I'm not aware of. Recommendations for a manager book club. So the book that I recommended before is the Gentle Art of Verbal Self-Defense. It's a long title. And there's a bunch of different versions of that book which were written for specifically different contexts. I think that the original is actually quite good. I didn't think that the others added a lot in terms of contextuality. I always recommend that book with a little asterisk which is that the author wrote it and she is a pacifist. She was an absolute pacifist and so she believed that it was always better to redirect, reframe, to take the hurt herself than to defend. And so there's, you can see that in the way that she recommends handling some of these things. I don't always agree as an adult with her tactics. But I think that it's a really excellent way to start thinking about language. Other books. I feel like there's some, I need to, I will try to collect a list of books and tweet them. Yeah, okay. So the question is what do you do when things get a little bit off track? They're still related to what you're talking about but too in depth. And the shorthand we use at Heroku is like you're off in the weeds. Let's come back to the point. Let's schedule a follow up if we need to talk about the technical details of this further. Most companies that like where I've talked to my peers, they have some shorthand like that. Like you're really involved in the details that are not appropriate for this moment. So I would look for that shorthand within your company if it exists.


three two one rest three two one work three two one work three two one rest two one rest work three two one rest three two one work one work three two one rest rest three two three two one rest three two three two one work two one rest three two one rest three two one work


I'm originally from Mexico. I'm a platform engineer at GitHub and today I want to share with you some thoughts about database performance. So why do we think it's important about talking about performance? Active record, normal frameworks make it so easy to talk to a database so we can concentrate to build an amazing product. However, as we grow and we have more users on load and our code target is more complex, we find ourselves finding queries that are duplicate in different parts of the code or optimizing for reads, pagination, counts, big tables, et cetera. Performance issues will affect your development cycle, your team morale, and ultimately your customers. But the real reason I care about this is because in my years of experience building back in applications, I've seen many fires and this is me when I have to fix one of those. Most of the times, if I have some information upfront, it could be prevented. So that is why it's important for me and I want to show you a little bit of how we do it at GitHub. So there are three aspects that build performance culture in any organization or team. So if you want to build that in your team, you need to take care of these three points. The first one is that it has to be the entire team's responsibility. It's not just the engineers, the DVAs, the QA team, or it's from management, everybody's on the hook. It requires process and tooling and you require infrastructure to support that. Automation tests, monitoring of development and other environments. And you need metrics and not metrics are one thing like page load time or things like that. And you need to ensure everything. IO, cache, individual queries, writes, reads, as much as you can. So enough talking, let me show you some numbers. GitHub is in the rails and my SQL application, I should say, my SQL, it's 16 million users. We have 19 million repositories. This is at least the public repositories and about 15 million issues. We do about 250,000 issues per week. So all this plus all the data that these actions generate will make up for a very large database. And this is just public information or public repositories. So I'm going to share with you how we do process and tooling to be on top of application performance, database performance problems in our application. My favorite coworker, HuBot, no offense to anybody, but he's super cool, is deployable everywhere, it's open source, it's built in Node.js and you can use it too. So one of the things that we use with HuBot, for instance, is graphing directly in Slack. This is the Graphite command. We can actually go to Graphite, copy the command that can run in HuBot. And this is great for onboarding people and actually teaching or showing when you build in a feature, how it's going to be, you know, you have metrics right there. Anybody can see them and because we're a remote company, so it's just available for everybody to use. One of my favorite aspects of HuBot is that you can customize anything and we have my SQL commands running HuBot. So for instance, if I'm building a feature and I want to know, you know, if I should add a new column, I can see if the size of the tables, I don't need to ask a database administrator or anybody in the infrastructure team, I can do it myself and I can make a decision whether it's a good idea or not to keep adding columns to the table, probably not. So I'm going to be using two examples of the most common problems that are, it doesn't matter what size of database you have or application, but these are the most common database performance problems that we probably all have experienced them. The first one is the famous Mplus1 queries. Everybody knows what Mplus1 queries. Well, an example of Mplus1 is when you have an association and a parent-child relationship and you issue one query to get the parent and then end being the number of children in the association. So the fix for this is very easy. We just need to eager load the association upfront. We can do that with includes and we have multiple, I'm not going to talk about the solution because we can figure that out, but I'm going to talk about how we actually find them at scale. One thing that we use when you are developing a feature as an application engineer, we use PIC. There are other tooling available, but we use PIC in particular. It's also open source. It was originally built at GitHub. And what it does, it puts a little top, a bar at the top of your page so you're developing a feature, you're working on this page. You can see helpful information. We have, in this case, elastic search, how much time it takes to call cache or features, but I'm interested in the SQL section because that's what I do. So we can see how many objects are being loaded. So this will probably give you an idea that there may be some smell here. And it also shows me the SQL that's being executed. So this just requires a little bit of instrumentation. It's super easy to do. And as we can see here, we have issued many queries against the label. So we have an N plus 1, so we just need to find where, because we're in the page that executes the code, we can just go and fix it. There's also other things we can do, like instrumentation on the API stabilization. You can count how many queries are executed at the beginning and then how many queries are executed after. You can graph that out so you can be creative. The point is that there's tools available to do that. And the second most common problem is poor indexing. And this is my favorite one. It comes in different flavors. First, lack of indexes. So if you can guess where the index was added there. This is another tool that we use. It's not open source. It's Vivid Cortex. It is like, I don't know, maybe you use New Relic or any other tool. Also, unused indexes. So when we change how a feature works, how a page behaves, it may be that we no longer need an index. And they will hurt our performance when writing. Sometimes we tend to optimize for reads, but also we need to take care of writing. And having too many indexes in the database, it's not good. So we have this another command running Hubert. My SQL index stats. You pass the name of the table. And this is wholly underneath the performance schema for MySQL. But I'm sure there is another thing for any other database. So you can just customize it. So we have here that this index issues and the table issues on x column is no longer used. So we can go ahead, whatever is the process, to remove it. And too many indexes. And I'm looking at you, Postgres guy. Sorry. But we can add this index merge on Postgres available. And it works. And it's great. But it's not a best practice to add an index on every column just because we can do it. As I said, eventually you will see the penalty on writes. So what is a good index strategy? Build indexes for performance for your critical queries. Don't try to fix everything. It's going to be impossible to do it upfront. Either way, see what are the most critical issues that you have. That's why you need the tooling so you can identify which tables are the most critical in terms of reads and writes so you can fix one by one. Also try to fix or add indexes that will help more than one query. So in this example, I have, because I'm suggesting a multi-column index, I'm choosing repository and user ID because I have an equality condition in the repository. But I don't have, in the second query, an equality condition for user. So it will stop working as soon as it heads the non-equality condition. And this index will help me use for other variations. If I only query on repository, it will still work. Prefer to extend an index instead of adding a new one. As I said, too many indexes is a problem. So try to take a step back. There is another command that we have that you can see as I show it, all the indexes so you can see it right there and try to do extend a new one. It's our extended system one instead of adding a new one. And favorite multi-column indexes, impartial indexes if you can. So how do we identify when we need a new index or when we have slow queries? Usually that's something database administrators do or the tooling lives in the infrastructure side. We use Haystack, it's our exception tracking tool. There is an engineering blog post that I have linked at the end so you can read more about it. And we use it for everything. And this is application side, which means that we can customize what a slow query means. So we can say half a second is a slow for us or two seconds or 10 seconds. So we grab the query, we can see where it comes from, and then we can execute the explain in Cuba. So that's why Cuba is my favorite. So we have the execution plan right there. We can see in this case that I'm querying on close at, but I'm having an index on user ID and created at. So I do need an index. So it seems like I do need an index or I need to write my query. So once I have the exception plan and I have a plan to add an index, it's really hard to actually test it if you don't have the data that is in production. You cannot never pretend that you will have the same execution plan at your local. Many things will depend for the execution plan. So what we do is we have tools available. Again, Cuba comes to rescue. We can clone a table to staging. So we can do testing there. And maybe this is not available for you, but that is a point about having an infrastructure to support the tooling. Once we clone the issue to, in this case, the issue is stable to the staging environment. We can test it. We can experiment and validate our plan. So in this case, it's successful. And well, finally, if you don't have this available to you, you can use science. And what I mean by that is scientists, which is a Ruby library that is for the factor in critical paths. And in this case, I have a, it's used in the index hands because my SQL supports them.  So it's going to use the first one, and it's going to try the second one only for comparing the performance of the index because that is my use case. And scientists will allow you to put this code in production if it's available to you and enable it for a percentage of your users. So you can test only in a percentage of your use base, and you have a graph like this, what it shows. Well, indeed, the new index that I propose is much better, or slightly better, but better after all. The key to database performance is to have limits to be on top of what you're doing. So limit round trips to the database, limit pages on your pagination, and do as much limited of your users but that you can do. And also be mindful without any columns. Now what, we have this information. I hope some of this is useful for you. Try to incorporate something in your team. Talk to your team about performance. It's not just engineering work. It's everybody's. It's for your good. And set a typical goals, start small and simple. You don't have to do everything, and sometimes you need to slow down to speed up at the end. The most important thing, oh, everything's fine. Thank you. Awesome. Thank you so much for seeing.


Yeah, it's decent. Of course, of course. Are you all right? I'm good, how are you? I'm not too bad. It's just cute. Come on.