It's a pleasure to welcome all of you here to Georgetown as we gather ahead of this first convening of the Public Interest Technology University Network My name is Jack DeGioia. I serve as the president here at Georgetown. It's an honor to welcome you here to Riggs Library I'm here quite a bit and I Usually don't get an applause. So I know there are some hoys in the audience tonight I'm grateful to welcome you all back to campus But also it's just a great honor to have all of you here and I wish to express My deep gratitude to Darren Walker president of the Ford Foundation and Anne-Marie Slaughter The CEO of New America both of whom we'll hear from later this evening I want to thank you both for your leadership and for your many efforts to bring us together this evening It's an honor again to be with all of you and to be joined by so many colleagues who are engaged with one of the Most pressing issues of our time how we will harness the power of technology for the common good We are honored to have all of you here for this convening and to play a role as a founding member of the Public Interest Technology University Network as we deepen the efforts here at Georgetown of our initiative on technology and society a Collaboration of nine programs and centers across our schools this question of how to draw upon our resources as Universities in order to advance the common good is one that has a special resonance for us here at Georgetown as A Catholic and Jesuit University were animated by a four centuries old tradition a tradition that has since its founding embraced a responsibility to the common good one of my colleagues here a Jesuit historian John O'Malley Has identified one of the early influences on this tradition as Cicero's Deofesius Cicero was a favorite author of the early Jesuits. They knew his work by heart The work is often translated as on public responsibility and Father O'Malley recalls this passage of Cicero as having particular resonance and I quote We are not born for ourselves alone We too as human beings are born for the sake of other human beings that we might be able Mutually to help each other We ought therefore to contribute to the common good of humankind by reciprocal acts of humankindness By giving and receiving from one another and thus by our skill our industry and our talents Work to bring human society together and peace and harmony close quote Well in the in the tradition upon which this university is built we acknowledge that we share a Civic commitment to seek the common good and at this moment in this network of universities We understand that there's never been a more urgent opportunity for us to animate a commitment to the common good Then where we are right here right now We face an increasingly interconnected world a world transformed by the rapidly power evolving power of technology It's a moment full of opportunity as well as profound challenges a Commitment to the common good changes everything it acknowledges that there's a truth to be discovered and good to be realized When we engage in this work together that there's a good we can achieve together that we could never hope to achieve alone So we're honored to play a role in this work and to welcome each of you to Georgetown for this inaugural convening of this network And I'm pleased now to welcome a leader in this work That's Darren Walker the president of the Ford Foundation and Darren's work at Ford and previously at the Rockefeller Foundation at the Abyssinian Development Corporation of Harlem has helped to shape social justice philanthropy across our country around our world under his Leadership the Ford Foundation has refocused its efforts to address inequality and new and innovative ways Putting in Darren's words a new gospel of wealth that engages with the root causes of injustice In addition to his leadership at Ford Darren is co-founder and chair of the US Impact Investing Alliance co-chair of New York City's Commission on city art monuments and Makers and a member of the boards of Carnegie Hall in the National Museum of African-American history and culture Among others Darren. We're deeply grateful for your leadership your vision and your commitment to building and strengthening the field of public interest Technology through work like this and we're honored to welcome you here tonight. Please join me Thank you Jack. Thank you for that and good evening everyone this indeed is a Historic moment You don't get to look at what I get to look at which is all of you and on this occasion this historic occasion The first meeting of the public interest technology University Network. I Am putting down a marker That history is being made and when history is recorded This gathering will be memorialized as a seminal moment. I First want to thank a few very special people Who have made it possible for us to get here tonight and are going to propel us forward a number of special friends like Anne-Marie Slaughter and everyone at New America. Let's please thank my friend Anne-Marie We have our distinguished FTC commissioners Phillips and Starks in the room. Thank you FCC commissioners our friends Tom Friedman and Friedman Consulting and all of the funders and we thank you funders because We need your money So Siegel Foundation Mastercard Foundation and Mike from and thank you Hewlett McGovern Schmidt Foundation and my friends Jeff and Tricia Rakes all who have supported this effort with enormous generosity. I Am incredibly Grateful to you Jack and your team here at Georgetown because it has been at Georgetown that we have through your law school and Through so many facets of this university come to understand the full power of this idea of public interest technology For us at the Ford Foundation This is not just about Ideas because we love ideas at the Ford Foundation It's not just about solutions because we love problem-solving at the Ford Foundation this is about Who we are going to be as a nation and a world We care deeply about justice at the Ford Foundation and While some of you engineers may not be engineered to necessarily think about justice as the first thing that your work is about Your work is critical To the kind of justice we are going to see in this world and The way in which justice is going to be achieved in this new digital era is going to require not just engineers it's going to require humanists and public policy and legal scholars and People working on the ground Who can help us understand the real implications of all of this? remarkable mesmerizing dazzling AI and all of that stuff This is serious business and you all are here tonight Because you mean business about justice and you know that the work you are doing is Essential it is essential to our democracy It is essential to giving us hope because that's the other thing you all do for some of us These are dark and trying days It is hard for many of us to not Be despondent and depressed and dejected by what we are subjected to every day And yet there is light That is what all of you represent during days of darkness When we need light Light seems to find its way So thank you for providing light. Thank you for making it possible for us to be hopeful about Tomorrow and all that you're going to do to make it better. Thank you. Thank you Jack for having us so grateful to be here Hi, everybody. My name is Jenny to me Yay to you yay back at you Yeah, so I'm Jenny to me and I'm the International Director of Tekken Society at the Ford Foundation And God I really want to thank I was gonna say Dean President DeGioia For inviting us into this beautiful room to have the first ever PITUN conference Thank you so much for doing that. It really sets a stage. What what a high bar, right? I Think I might have been a little bit of that Hoya energy just a small amount So tonight we began a new chapter to try to solve One aspect of this incredibly complex and timely and important problem Our big goal is to build a robust field of public interest technologists So that bright kids coming out of college or even those who are self-taught and there's a lot of them some in this room Can have more choices about what type of career they have and they bring their incredible technology skills to Today and tomorrow we're focused on one critically important set of questions, and that's just the role of academia There are many of you in this room that are working on other aspects of the problem and the problem is big I heard this this afternoon a lot of people were saying like how are we gonna deal with the supply side And and the demand side and yeah, we have to solve all those problems But tonight and tomorrow we're talking about academia And I also want to just thank the folks who have paved the way and civic tech and Tech for good and the other areas people like Mika Seafrey who's here from Civic Hall and we invited Jen Polka, but she was unavailable But lots of people are doing really important work and this is just a piece of all of that and Tonight we're just drilling down on the academic stuff because we're at a university and And and we have a little story to tell about why we chose this work And so if you'll give me a few minutes, I'll just tell you that story You know tech has transformed Every aspect of our society and yet within government and civil society organizations, and I can even say philanthropy And many areas of the academy. We know we're not sufficiently prepared to understand the massive change The technology is bringing and Why is that? It's a very good question and It's a perfect lead-in for me to tell a piece of the origin story which leads us here tonight about seven years ago Ford in partnership with a number of other foundations many of whom aren't actually here tonight But MacArthur night Mozilla and open society we began looking into this challenge and we assumed all right You know the way things are going it's inevitable technologists are gonna ultimately be working in every sphere of life but the question was We know this is gonna happen, but how quickly will it happen and When it happens will these be the technologists that we actually need? Well, they have the right skills to be effective in different environments than the private sector and Who's preparing those people to have those different skills? At Ford we were largely motivated by the fact that I had an internet policy portfolio and my grantees Which were largely a group of lawyers and policy experts and community advocates? These people were desperate for technologists and not just any kind of technologists They wanted technologists who were excited to bring their technical skills into the room and to partner with these other Disciplines and actually figure out together. How do we rescope how we solve these problems together? you know people like lawyers and policy experts and social scientists and organizers and So with the help of the Freedmen team and Tom's in the room, so we all should clap for Tom because Tom has Helped us along the way and he's brought some incredibly bright wonderful people who've helped us for this last seven years But he had conversations and his team had conversations with the smartest people we could find many of you were in the room tonight People who are our grantees But also people who had managed to create these kind of crazy career paths where they worked in the private sector and government or a government and Academia or academia and the private sector in public service so people like Danny Weitzner and Latanya Sweeney and Gigi Sohn and Cecilia Munoz and Susan Crawford and Alan Davidson you may have one at each of your tables if you're lucky And we asked them how did you build your careers and what the barriers were and Boy that first report. We got from you guys was just daunting. I remember when we looked at the original data We had to you know We had to send Friedman back to look at it again because there was like a hundred different ways That the types of technologists that we knew we needed might fall out of a pipeline That would actually have led them into public service So they start with the desire to be in the space, but all the systems were working to make sure they wouldn't have a place So Darren challenged us as a good president of a foundation would do to narrow the scope we couldn't work on all hundred and He talked about how he remembered there were philanthropic investments and experiments That Ford had been part of that actually led to the development of the field of public interest law And so he actively asked us to look at that And we made some bets in the academic environment We funded a bunch of incredible fellows programs that are really showing promise now We funded a peer-reviewed journal at Harvard so that people could actually begin to publish in this crazy area and Then we funded one of my favorite experiments, which was a class that taught computer scientists and lawyers simultaneously through a virtually linked classroom and Danny Whiteson and Alan Davidson taught the computer scientists at MIT and Alvaro Badoia who you'll meet tomorrow Was at Georgetown Law and he was teaching the lawyers and they would come together a couple times a year And they would wrestle with the really big problems the actual same problems the FTC and the FTC are wrestling with these days and that's how they got graded if they had good answers and One day Darren and I were in DC and we were just checking on some grantees and And we were meeting with Ann Marie and we were meeting with Danny So we just decided we'd meet at the same restaurant We just all had breakfast and we were all talking and Danny was talking about his class and it was really successful we were really proud of it and It was popular and the students loved it They were giving it great reviews and lots of people were applying for it and they were finding jobs Coming out. They're using these hybrid skills So we thought well this is a success But Danny was worried and he had heard that some computer science professors were dissuading students from taking the class and Darren said why? You know why if it's such a great class why and Danny explained well these professors believe that if you're going to MIT for computer science You really shouldn't waste even one credit on something that isn't computer science, right? I mean it's the best one of the best computer science schools in the world and Ann Marie if I remember correctly immediately identified this challenge as one she'd seen before Where academic institutions artificially isolate certain disciplines from one another and we had this great Conversation. Oh my god. It was such a great. I mean it was a great breakfast And it's a perfect lead-in for me to tell you tonight Why we're here together so We wondered if this had actually happened at other universities and enter Friedman again and with the help of Maria white from the Friedman team We went deeper into another Set of questions and the data came back and showed this was happening everywhere The computer science and engineering schools were being isolated from the humanities and With this evidence in hand and a roadmap to the roadblocks that the PIT professors Described they were facing Ann Marie Darren and Larry from Hewlett Caught went through their address books and just over a year ago pulled 25 presidents of universities together and Pulled them the room with funders and with PIT exemplars at the green tree conference center and Challenge them to change the pattern It's just over a year later We have 21 universities who've pledged to break down the silos and their institutions We've raised three point five million dollars to stand up a network and see two point six million dollars worth of projects at The network universities this year We have almost 30 universities asking to join the network and New funders in the wings looking to join the fund which we're gonna need as Darren said But maybe the thing that inspires me the most is that in less than a year's time from the green tree meeting Arizona State University is not only Volunteered to host next year's conference But they've also announced a masters of science in public interest technology Enrolling students for a fall 2020 semester where you can actually get a degree in PIT This academic field is growing before our eyes So we went slow for a while we're going really fast now And I can't think of any group of people who can better explain What's happening why it matters and what's coming around the corner than the panel we have for you tonight And with that I want to thank Ann Marie for her partnership in this incredible project and thank her team and dream Cecilia a Fuha and hand her the mic To begin tonight's panel all right, so Jenny actually gave you all the history of of where we are How we got here the origin story, so I will not repeat any of that, but I do just want to say one thing That when we were in green tree a year ago a year and a bit it Darren and I exhorted and We hope inspired the presidents and the provost There and I said from my discipline, which is international relations and law originally There's the one of the most famous books in the discipline is Dean Atchison's present at the creation And he of course is talking about the creation of the liberal international order Which is actually under attack at the moment, but that's another evening and another subject But what I what I invited people was to be present at the creation Not just of a university network, although it's extraordinary how our teams have built the network that you are all now part of But really building an entire Academic field or set of joint degrees and joint programs and a set of career tracks Right that to allow a young person to think if I study computer science or engineering or data science and public policy I Can go work for a big tech company and make a lot of money But I can also work for the federal government or for a big NGO or for my mayor and Change my community change the world and that so that is actually what we're trying to create and it's Fantastic to see all of you here So with that introduction, I will introduce my panel very quickly We are gonna talk more about the demand side because we have people from government from government and the private sector Currently at least Mike's had many roles and from the media and I'm going to be asking them about their Experiences hiring and working with public interest technologists in their jobs and also what they see as the needs So they're not the hirers But they they have seen how these people that we're trying to educate can work So on my immediate left is Cecilia Munoz who is the vice president for public interest technology and local initiatives At New America and prior to that was the head of the Domestic Policy Council at the Obama in the Obama administration I went to see her when she was still there to convince her to come to New America and I'll tell you I wanted her to Focus on National network and local initiatives and she was interested But what really lit her up was the idea of being able to do what she'd done in government with the US digital service in a non-profit So to her left is Mike Froman Who is now this I have to read to make sure I get it right that vice chairman and president of Strategic growth for Mastercard So he has just of actually a wonderful portfolio, which I happen to know was designed for him Growing strategic partnerships new business opportunities And I know I'm gonna move on the one day no But it's a it's an area where he gets to really look at both the business side and the strategic side of doing good In the world and he oversees the center the center for inclusive growth and under Shamena Singh who's and that that center is doing fabulous work with governments and Non-profits around the world to bring people into the financial system and on my far left is Kim Hart Who we are so grateful to her for stepping in at the last minute Kara's Kara Swisher had a family emergency on Friday And I was we really were a little stuck and Kim agreed to step in she is the city's Correspondent for Axios and she has written quite a bit on tech and cities and before that She was the press secretary for Tom Wheeler at the FTC So you've got people with experience from lots of different places. Let me start Cecilia with you Talk a little bit about your experience with us DS and talk about what since we are talking to Presidents provost and deans What do you wish those folks that us DS had studied or known? So I'm one of those humanities people who has learned about tech, so I'll just confess that at the outset Because I was in the domestic policy council when we created the US digital service Which really kind of came out of the the debacle of the health of healthcare gov, which we fixed by the way I Got to place help place digital teams where we were trying to solve hard problems at the federal agencies So my former colleague Lauren and I were just discussing the the college scorecard that the president Obama wanted to do Where we placed a digital team to help us with a data problem And they ended up coming back to us and saying this website you want to build that's fine The thing the president told you to do but actually if you release the data in an API And you make it available to all kinds of different kinds of people to use it You will you will reach many many more people in a variety of different ways and they they Came up with that insight by talking to 17 year olds who were our target for the college target audience for the college scorecard and That insight revolutionized that particular project and you know there were there were dozens of those Transformational experiences at the federal agencies where President Obama asked us to resettle 10,000 additional refugees compared to the 75,000 we were on track to resettle that year. We all knew there wasn't going to be any more money from Congress the people at the agencies looked at us like we had lost our minds because They of course wanted to do this But even if there were more resources you'd have to hire people you'd have to train people they were telling us this is not possible and They humored us by sitting down with a digital team Kind of rolling their eyes at the time But that team together the policy people who knew refugees and the digital people who had a different set of tools Resettled another 10,000 refugees that year without spending another nickel and though so we had lots of experiences like that Demonstrated for me how transformational this set of tools can be it didn't always work we couldn't always get the the the policy nerds like me to to Communicate well with the digital teams and Kathy Pham is nodding her head But when it did work it was utterly transformational and I come from the NGO world I was at a group called the National Council of La Raza now called Unidos US for 20 years I came out of government feeling like not only does government need that capacity But certainly the civil rights groups do and really the NGO world does that we are trying to solve The problems of the 21st century and we're still using the same tools We were using 30 years ago when I first got to Washington. So So this is really about I thought when I left government that it was about bringing technology to solving these problems and what I have learned in the process of Helping co-create all of this with so many of you is that it's not about technology itself It's about the skills of technologists and that we we need that skill set at the table when we're making policy if we're going to deliver policy effectively And I think frankly the tech companies need a little policy experience at the table not just in their government affairs shops, but on their engineering teams that we would be running into and creating fewer problems if There were more kind of bilingual bi-cultural people if you will and that's Who you will be growing and creating? So I guess if there were one thing I would I would ask for I Think this skill set is so multi-dimensional that it can't live in anyone in any one program or any one department The people who are successful at doing this are This is gonna sound like a terrible word, but it's true it really defines it They're kind of the hustlers people who are using lots of different skill sets. I mean it endearingly. I really do exactly Not sure you want to put that in your your curriculum guys They're entrepreneurial. Yes Yes, not in the Jennifer Lopez way doing absolutely right no, it's it's so President DeGioia close your ears What I mean is like my colleague Marina Martin. She's an engineer She is obsessed with the foster care system She talked her way under the hood of the foster care system in Rhode Island and has created innovations which were now taken in nine states Marina is a hustler in the best possible way and She she's bringing what she learned in engine school, but not just that she's bringing process design She's bringing common sense. She's bringing the fact that she deeply honors the civil servants who are Who are caring for foster kids every day and it's that combination of things, which is what makes her successful so Mike and in reading in Describing your MasterCard position I sort of left out that you were the US trade representative I try and forget that too at times So So if you are free to also comment on what you saw in government, but but I also want you to Talk from your perspective now Where if you look over MasterCard generally you're part of it the Center for Inclusive Growth How do you see the kinds of people will be training having stints in the private sector? Or as I said your job at MasterCard sort of straddles public purpose and private profit So first of all, thank you for having us and I want to introduce Nicole Lindsay who's here from the Center for Inclusive Growth and represents us on the advisory board Let me start by saying I'm in violent agreement with everything Cecilia said and there really is this this is incredibly important initiative We're delighted to be supportive of it and it's it's so important to be able to bridge that gap in understanding and involvement between these two seemingly separate separate worlds And let me give you three examples from that we're involved in that we're trying to explore in addition to to this partnership One is to help develop cyber talent and with Microsoft and with Workday We have a fellowship program where we put cyber talent into the government for a couple years and then get them a job in the Private sector so they have an understanding of the government's perspective on cyber security and then can help bridge that gap with the Private sector and we waive their student loans as part of this overall Activity that's one two and and we should talk about cities at Axios We've taken on the issue of Opportunity Zones Which is a pretty controversial tax credit that was in the the Trump tax bill bipartisan support Cory Booker Tim Scott were the the primary proponents of it it could end up being a windfall for real estate developers or It could end up being one of the most powerful Forces for investing in distressed communities that we've seen in in recent history. It turns out at Mastercard We have some data Quite a bit of data on on spending now. The good news is by the way We don't know your name just you know, we don't know your name. We don't know what you buy We have no personal information But we do know where money is spent when it spends and we know for example if somebody who probably lives in this neighborhood is traveling four miles away to go to a grocery store because there are no grocery stores in their neighborhood and there's a food desert and We're beginning to work with mayors all over the country to help them understand what their Opportunity Zones look like How they compare one to another and how they compare to similar zones around the country and then how to use our data Both as a baseline to be able to show what progress is or is not being made through this tax credit But also as a way of attracting investors into their Opportunity Zones and saying you know We'd really like to see a grocery store and a retail complex built in this neighborhood We have the data to show you that there is demand in the neighborhood And so that's how we're using it and and I'll I'll mention One other example and then come back to to really your question Which is that the third example is a partnership we have with the Rockefeller Foundation. Sorry Darren To do a data science for social impact collaboration and our first joint project is Grant to DataKind whose job is to work with NGOs And we're hoping to encourage them to do it with governments to help them even understand What is the data they have how to clean the data what data hygiene means how to formulate an appropriate data science question And then brings in volunteers from the private sector data scientists to help that nonprofit solve Whatever issue they have we're trying to help get them to scale make them relevant to governments precisely for the reason you said because The data divide excuse me the digital divide that we're now very concerned about is the digital divide That's emerging between the private sector and the civic and and government sector There's so much advance going on in the private sector every day every week The tools are incredible the platforms are incredible We can do all sorts of things now that we couldn't do six months ago or a year ago And yet those tools are not being made available to the nonprofit or the government sector And we need to find ways of bridging that gap and one way is in training people to see training Technologists to go into public service and even if they don't go into public service one one reason I use the data kind example is We've got 500 data scientists sitting in St. Louis We want them some of them will be willing to take six months and go volunteer for an NGO Some of them may actually go work for a government agency at some point in their life But we also want to give them the opportunity to spend four hours every Friday afternoon working on a data science for social impact project and Giving them that experience to understand what the social sector and the government sector needs. There's a huge Mars versus Venus problem going on here as you all know and I President Joe I don't I don't know my Cicero, but I do know that Pericles once said You may not have an interest in politics, but politics has an interest in you and You know for those of us who spent time out in Silicon Valley with our friends out there and Jeremy's here You know for up until two years ago.  Because those government agencies are now very much interested in them So Kim, you're the city's correspondent and I know you've been writing Specifically on how tech is changing cities and jobs in cities. So maybe you can talk to us about how More broadly perhaps with what this landscape looks like from your point of view, but specifically that sure thank you And I'm no Kara Swisher, but I'm happy to be here So I actually just took over the city's beat about three months ago I created it at Axios as I saw an opportunity to really cover something that was not being paid attention to Which was this intersection of technology and local government Especially as more and more gridlock was happening at the federal level on a lot of policy issues you know citizens and You know constituents are looking to the states and this their cities to fill the void But also cities are just being bombarded by all of this new technology all of this new data Smart cities sensors internet of things 5g and so many of them are just kind of at a loss They don't know where to start They don't have the expertise to really figure out what's real and what's hype when something will actually benefit them And there are citizens and when they're just being taken for a ride by really good salesman and so I Actually So I've been covering tech policy in DC for about 15 years and then did a sit in the private sector And as and Marie mentioned went into the government for a few years as well But I think that with and that you know Tech policy is like the beat to have in Washington now But frankly I think that it it kind of starts to feel like Groundhog Day a little bit to me as I've been in this World for too long I guess and I couldn't see the stories the trees like well Google did that five years ago That's not a story today, and then someone would write the story like damn it. It was a story and so I I Realized I need to do something different and I was a managing editor And so we were trying to I had pitched writing about cities and kind of trying to fill this void Especially because local media is struggling so much to a lot of these stories in the local level just aren't being told people aren't kind of hearing about What the dynamic is on the ground you hear so much about the big cities San Francisco, New York, Seattle the big cities where all the tech money is and where there's there's still a pretty thriving media or Landscape there, but in cities like Kansas City even Miami. I mean they're struggling with very very different problems But there and that's part of the the situation we got into with politics, right? They're getting kind of just painted with the same broad brush Even though they're completely different and have very different issues and are trying to come up with creative solutions to them so Like one example that I'll give you I was in Atlanta last week Filming for our HBO show and we were in a very small town called Peachtree Corners seven seven years old and we are and I was talking to the They're they're you know, they're trying to become a test bed like a living lab for Kind of the next generation of technology and they have an autonomous vehicle test track they're completely wired with 5g one of the few and They're being very aggressive on this and I talked to their city the deputy city manager who doubles as the chief technology officer and this is a town that has five full-time employees And it's a town of 45,000 people, but that's a lot of the towns. What do you think about it? Like the big cities are only I mean that's what we hear about as we talk about but the majority of people actually live kind of Like on the inner ring or in this like the the exerbs if you will And so I was asking him well, how did you become a chief technology officer or what's your background? Is it in technology? He's like no I was a consultant I was a consultant for this company that like consultant, you know bid me out to different local Governments to do their work because a lot of these small local governments They can't afford their own people or they don't have the expertise and so they go to consulting firms and hire people to Come in as contractors He said I just realized how important this was and so I taught myself And so he's learned it himself and he had to kind of you know Take that extra initiative and he has added benefit of that being in the private sector and also seeing the government side But you're starting to see all of these cities big and small trying to figure out How are we gonna have the expertise to get to get our cities and do what we need? We know we need to do for our citizens and to help and it's not just for the citizens But also especially for some of these cities that have really seen the bottom fall out that with deindustrialization How do we get people to come back? How do we get how do we re-energize our base get jobs to come here tech jobs to come here? So that those high-paid high skilled workers come here and then we might stand a better chance of surviving kind of the next wave of automation and AI and the You know kind of in some ways devastating impact on the workforce that we're about to see with the fourth You know industrial revolution. So I think there's so many dynamics happening at the city level that are kind of being overlooked Not, you know by by malice just because there's so much happening I mean there's every new cycle every day feels like seven new cycles and even talking about the I mean There's there's so much happening at this at this level that very few people are able to kind of think about that And even at the tech company side, they're looking at I mean they have plenty of challenges of their own on the You know the top level and so it's not an area where or they have very small teams Or it's not really their priority to be focusing on the cities and the local governments and states When it comes to an actual product Standpoint it's more kind of more of a policy and lobbying standpoint still So that's kind of what I'm seeing and I think there's a lot of There's a lot of opportunity people see the opportunity But there's also fear because they don't know how to cross that chasm between where they are Which they know is falling behind and where they want to go and they don't know how to get the skill and expertise to get Them there or to work with external partners to help them cross that gap You are also free to respond to each other But let me on that and I don't know who is here from the University of Chicago when I'm gonna describe this And if I get it wrong, okay, so so as I understand it the Harris School has a program that is funded by a look by a tech company to Put your graduates in with city government local municipal governments for nine months after graduation. Did I get that right? So I cite that because it's you know, we do need to prime the pump, right? And often people sort of know they need tech expertise, but they don't know how they don't know where this is something that you can Obviously encourage when kids are in school as internships, but it's also it strikes me as a way of getting Local tech companies to fund Something that will then help close the gap that Mike was talking about Well another part of the gap though and Jenny really made reference to this is is the if this is the supply side is the demand side is Making sure that governments know what to do with this skill set and with this talent because ultimately what we're trying to do is Create the job market for the folks that you that will be coming out of your programs and it's it exists in in some ways But it's it's different and like some cities have CTOs in some places That there's a you know a person situated somewhere at some agency who is frankly enough of a hustler that they're the person You know making it happen getting things done It doesn't always necessarily live in the CTO's desk if there even is one Sometimes it's just some entrepreneurial person somewhere in the inner workings of a city That's kind of making it all happen. So getting to a place where there is sort of systematic demand in Local governments, but also among NGOs. I've Personally focused on the NGO piece and I found that to be a really hard nut to crack in part because Everybody's up to their eyebrows in because everything's on fire right now So which is a considerable problem. They don't know what Even though they have a myriad of problems are trying to try to solve They don't necessarily know what to do with this skill set in order to solve it I always think about that as every time I buy a new computer or a new phone I think okay this time. I'm really gonna I'm gonna read the manual I'm gonna figure out all those things that it'll do for me Right, and I have exactly enough time to figure out to do it with it what I was doing on the old one Right, and it's a sort of the same you know in theory your life could be a whole lot better if you use Tech in a more sophisticated way, but you don't necessarily but sincerely I want I wanted you also to talk about the problems you saw on the policy side. So in other words Let me ask it more specifically Cases where the tech person thought we will build this tech and it will be adopted or here's a cool solution And it will be adopted not not always so easy from the engineering I mean to move to the policies. Yeah, well in fact one of the deep lessons that I think are The folks who have come and joined our public interest tech team in New America from the government experience really learned was If you if you're actually starting with the technology that usually means you're getting it wrong Usually means it's not gonna work. It usually means you're not asking the right questions We have become deep believers in Sometimes the thing which fixes the problem isn't technology at all and there's Kathy nodding her head again I know I'm getting it right so that that Coming in with the technology is a sign that you're not actually listening to what the problem is and the the deep listening to what the problem is and who is the user and what is the What is the broken thing that you're trying to fix or the benefit that you're trying to get out the door That you have to start with what's gonna actually accomplish that so to give you an example from an opportunity that our team recently had With a state which wanted to digitize all of its forms and on its face Digitizing the forms that are on paper that a state government uses seems like a perfectly good thing to do I've been telling my doctor's office for 20 years But you know I learned from Denise Ross who's sitting right there on term again as who's sitting right there If you digitize the forms, but you're not actually looking at the processes that those forms represent You're not actually fixing anything for anybody and in fact You might be creating more problems for the adjudicators sitting on the government side of receiving those forms so the the the deep dive into What are the ten? Intersections that the state has with its public What are the ten most common ones and what should those processes be because it you might not need a form at all And if that's the case then digitizing the form is a terrible idea so so the the real lesson here is is that the You need technologists who can do the deep dive and listen and and have the humility To know that sometimes the solution is not building a thing sometimes it's fixing a process Mike you were nodding. Well, yeah, I look I think this is one critical part of it which improving government processes and how a government provides services to its Its residents its inhabitants and there's a lot that can be done in that regard But I think the the other piece of this or one other piece of it is the whole Regulatory Legislative piece of it and we saw the rather embarrassing hearings on Capitol Hill of the deep penetrating Critical insights the members of Congress were bringing to the table as they were questioning technologists you know we're as There is a natural inclination to want to regulate and it's appropriate and but really getting underneath the surface of those kinds of Conversations to what are the the fundamental principles? We're trying to regulate around what are we trying to preserve? What does privacy mean what is data protection really mean and what are the multiple ways of getting at it? in a way that that makes sense and Can't leave it just to industry to self-regulate that hasn't worked terribly well but we have a lot of education to do and a lot of seeding of Congressional staffs and agency staffs with people who have the expertise To be able to make intelligent decisions there and we see that not only in this country But we see it all over the world we're dealing with governments all over the world They have absolutely legitimate concerns that they are addressing on behalf of their their citizens and their people But they need a lot more education about the areas that they're trying to regulate in order to do so intelligently and Getting people in there who have that capacity to understand not just what the technology does But what the the normative framework is in which the technology operates is awfully important We found we have tech tech Congress Travis. Where are you right there? I mean that number of the the tech Congress fellows we put them into Congress and When they take they can do extraordinary things right? I mean suddenly an entire committee staff suddenly Can pass bills and and develop a sophistication for you know for others? It's it's harder, but we imagine somebody in all 535 offices. They can find a truck with all the tubes in the back that Allow the internet to work What is your business model? So Kim I want to I want to shift gears just slightly and ask you you've also done work on technology and and Life-long learning so yeah, since I don't want to miss that given that we have a crew of academics here Sure, in fact a few about a month ago a colleague and I kind of collaborated to do it of one of Mike's deep dive newsletters on higher education and where that was going and it wasn't something I knew a ton about but it was something that I've been that kept coming up in my conversations with mayors and city leaders of how well how are you going to get this next generation of workers to your Into your towns and so much of it was we have to create the pipeline of workers And a lot of companies also because they're struggling tech companies that are struggling to fill their tech jobs Also see the need to work with universities Specifically to make sure that they're not just pumping out computer science majors or any other kind of technical field But that they're actually using a curriculum where they can walk right on to a job at Salesforce or IBM or a Cisco or what have you because and I think there's also a lot of appetite You're seeing a lot of these companies now, especially the really big one start to work with community colleges there's they see a really big opportunity there with Associates degrees but also figuring out how to implement a certificate programs so as part of the curriculum really working with the university with the schools and Sometimes in tandem with the local universities as well to create more of an ecosystem so it's not like the community colleges are kind of you know pushed aside over here is like well That's where the you know that the safety kids go or the ones who don't get into the major University it's like no, let's bring it all together and build a pipeline of kids who can have some some skills who get a baseline of skills at the community college level and Either can go on and get some certificates and go straight into work Or you can go into a four-year degree and specialize more for a specific job after that And so creating different pathways I think that the also the the first time full-time student that we've all been so accustomed to is Really no longer what the student base looks like it's much more diverse people are going back to school to prepare for second careers you know and as they're 35 or 40 and and Frankly the companies need those people to be reskilled and get back into the workforce in new jobs in Orders for them to grow and reach their own goal. So there's kind of like this symbiotic relationship that's forming That's actually pretty neat in in Arizona for in Phoenix for example, they have ten community colleges and they're working with Apple and Cisco and For four or five others that I'm blanking on right now They're all like the big household names what I'm gonna say the wrong ones if I try to guess them and they they work with creating certificate programs and actually embedding with the curriculum providing subject matter experts to go into classes and They've created kind of actual pathways of if you want this kind of job at Amazon go here and do this and here are the like the three certificates that you can get and also Amazon Web Services has actually made they started a certificate program at I think it was At the Nova at Northern Virginia Community Colleges and just made that program and curriculum available statewide to students there To try to create people create a pipeline of people who have skills to walk right into a data center and do that job Because they just can't find the right people I think there's also less of an emphasis on four-year degrees and traditional four-year degrees that we're we were all familiar with and still are And what other kind of hybrid learning situations and credentialing that you can get to fill these jobs? as you know as automation takes over and Changes in ways that we can't predict Finding people who are flexible and having those that adaptability in some of those soft skills too so making sure that the computer science majors at MIT are Forced to take a humanities class or something like that to kind of get them More acquainted with some of the things that they're gonna have to do on the job like work as a team Like coming up with a solution, which is not something that all engineers. I'm married to one so I can say that kind of come out you know Really able and adept to do great So I have promised to get everybody out of here by nine So I'm gonna ask one question and just let any let you all Respond how you want? So what we ultimately hope for if this is really going to take off We hope that all of you those of you who are provost certainly Encouraging your presidents those of you who are deans You know what I'm talking about you will be talking to your alumni and your donors about how to build these programs at your University because ultimately that's how it has to take root. It can't we can we can seed it We can support it and we can share Both lessons and and and collaborations and maybe even a little competition occasionally But it's got to take root in the same way that public interest law did such that every law school in the country Or most have a public interest law clinic and fellowships and they they they put students in in various ways They fun if both internships and fellowships and after graduation and they have the courses for students when they are there So the question is if you were making that pitch to a donor What is it that you would emphasize? I mean when you really distill this and you think all right look Here's what we need. What would you say? So I've I make some variant of this pitch all the time and what I've learned is that when you When you bring up tech people think they know what it is that you're talking about and Usually what they think it is and what you're talking about are not the same thing And so I've become a deep believer in the value of storytelling as part of this Part of this endeavor that and I do it all day long. I tell the refugee story Which I told a piece of today. I talk about the college scorecard I talk about the foster care project that we're doing jointly with foster America because the the details of those stories are What illustrate what this is and the impact of what it what those people who you did those particular projects accomplished Is extraordinary and the that's how I got my head around what this is, you know, I'm a policy nerd I you know, I now think of myself as a technologist, but I came at it through the policy angle What evangelized me was seeing this stuff happening and so I'm a deep believer in telling the stories of what's possible of What it will mean when we reach the day that somebody in middle school now says I want to get a computer science degree because I want to end homelessness like when when she makes that connection We will have succeeded, but you have to tell stories in order to Make that vivid for people what she said Look, I think I think I would I would make the point that these worlds are converging and they can either converge in a really wonderful mutually supportive harmonious way or we can have a lot of conflict and If we care about the technology sector, it's got to learn about the public policy Social political sector and if we care about the public policy sector It's got to learn about what technology Offers as an opportunity and what needs to be addressed as well in terms of issues of of concern and otherwise we're going to find ourselves in sort of a mega, you know trainwreck and The other the other point I would make and I know you and I have lots of conversations about this from back in the Woodrow Wilson school days I think it's important to make the point that Technologists can contribute to the public interest in lots of different ways Some will go and work for governments Some will go and work for NGOs and some will work for technology companies and use the tools that they're Disposal to have a broader impact on society and I will tell you now being in the private sector again It is so powerful to be able to offer a technologist the opportunity to work on a social impact Project and it's not just the Millennials who you know, I'll have ADHD and you know can't concentrate on anything It's it's the 60 year old Technologist who comes to my office and says I've got one more You know turn it at the crank here at this company before I retire I want to do something meaningful while I'm here, you know put me on a project And so I think we have to define public interest in the broadest possible way including what we can do from the private sector side To have that impact as well I guess the only thing I would add is as I'm thinking about this from kind of like the if I'm a student going into a University and I know feeling terrified about this as a student is well I have to choose a major that I know I'm gonna get a job in or I want to choose a major that I like But that I also have a job and I I'm on the advisory council for the journalism school at the University of Florida Which is my alma mater and one of the things that we were trying we were talking about Recently was how in terms of increasing enrollment and keeping them there How do we help journalism students realize that there's a lot more to do in the journalism field than just being a reporter? And I think that that is applicable in so many different Majors and fields of study, but especially Technology fields where people think that it's very one path that they only have one way and one one way to succeed because that's what they see in the news and that's what they see people who become millionaires do and a lot of people if they realize that there are different paths that are actually kind of mapped out in a way that is tangible to them and achievable and that also helps their parents feel better about paying the tuition bills and so like Helping the universities map out those plans and paths can help on a lot of levels as well I would think so we'll know we've succeeded when in 10 years that whatever the version of the Big Bang Theory or Silicon Valley is on television Those engineers will be going off to work at the Ford Foundation and the Center for Inclusive Growth and the mayor's office and the federal government So thank you all you can see why I have such a spectacular panel I want to thank President DeGioia again, and I'm gonna just echo all the things that both there and And Jenny made so that we can end the evening, but please join me in a round of applause


Now I have a clicker question where we get to calculate our allele frequency in class. Oh, yeah. So I just wanted to summarize this conversation by kind of giving you a little bit of a heads-up about where we're headed next. We're going to do, we're going to spend an entire day on the mechanisms of evolution. And so we're going to go into more detail in both micro-evolution and macro-evolution and how exactly that happens. And there are some pretty awesome mechanisms involved, one of which is natural selection, and that's actually the topic of our entire lab activity in the following week. We also are going to spend an entire day talking about the evidence for evolution and the kinds of studies, research, findings that provide support for this process and that this actually does happen and has happened over the entire course of life. Now it's really important that I actually, I feel like I already said this, but I can't remember so I'm going to say it again. The fact is that evolution is a scientific process and it is supported by evidence and we're going to talk about all that and there's mechanisms that totally make sense. Evolution is not the origin of life. There is mad research happening, people are super interested in where did it start, just super legit, I mean I'm curious about that too, but we don't even talk about that. We'll look into the leading theories in that region, like what the research is saying, what it's pointing to, like we're not, no, we're, we've got, this is enough, we've got lots of work to do talking about how it happens and the evidence that supports that yes indeed it happens. But I really hope that this, for me, evolution and faith are not mutually exclusive, like I don't know, did I just say that right? Basically you can have both, that's what I meant to say so I don't know what I just said, but they're different things so the process of science gives us this evidence that we can look at and analyze and talk about and be interested in and learn about and it's different than faith which is really important for lots of human beings and I hope that there isn't a block in there for you because the part that we're gonna talk about in this class doesn't mean that you don't get to also have faith or that we don't also have faith. The last thing I wanna do today, tonight, right now is talk about the lab that's coming up. We're gonna do a tree thinking lab and I just wanna kind of roll through my vision of what you are going to get out of doing that lab. I be right back.


So good afternoon everyone your best Italian host ever. It's again here We have had all lunch. We all have our refreshments. I See no, okay We have now Robin from Sipman is Robin Sipman from IG lead developer. It's I must say it's It's beautiful to have this is our first Consumer of Kubernetes coming on stage. It's we always see vendors Getting their talks through as not brother pitch is fantastic You guys are doing a great job But it's also nice to see consumers to see what the troubles that they go through to to enable Kubernetes within their organization so Robin Would you like to join me? Yes. Thank you very much. Give it up. Thank you So the mic is working that's good and everybody has a full stomach, I hope so that's nice I'm gonna talk about I and use container hosting journey. So Like the red is that we have our own private cloud and we're building container hosting there And one of the servers that we offer on that container hosting is what we call namespace as a service And this talk is mainly out. What does that infrastructure actually look like and how did we build that journey? So first a little bit. My name is Robin Sipman I'm the lead developer in the ing container hosting platform team That's what that acronym stands for and ing is a bank, but we write a lot of software We are very apparent in in Europe, but we are all over the world And in fact the container hosting platform team the team that I'm in is also a team that has many nationalities in there I think that's very cool And today I'm going to talk about why do we actually do namespace as a service as opposed to maybe giving full clusters out in the private cloud And then what does that stack actually look like in the private cloud? structure and then I'm going to zoom in on How we built the namespace as a service on top of OpenShift So I already spoiled it. We run OpenShift instead of Kubernetes on top of Kubernetes I'm going to talk a little bit about the dependencies that we have to to build this this journey And then I'm going to zoom in to film also the the controllers that we have some of the applications So there's some Python code in there. There's some go line code in there So I hope you're excited for that and of course a demo. I recorded it though So it's not a live one, but it's a demo nonetheless So let's dive in so we only offer namespace as a service and The goal is I'm just going to go through the full slide It's a lot of text, but in ing we have a full cluster right and We don't want to give full clusters to consumers because then we give out a lot of notes a lot of resources And they will not be utilized fully But if we built a multi-tenant cluster and we over namespace as a service Then we are in full control of the compute and we can give out the resources to namespaces of those Exactly what they need right so if one namespace of one application requires 10 CPU cores and 10 gigs of memory We give it to them and another namespace requires etc etc For we spread it out over the cluster, but we are in control of the compute and that gives us a number of advantages Also in terms of compliance the the people that request a namespace. They don't have to worry about the underlying infrastructure Right the cluster it works They they don't have to know what compute nodes they are running on If they are patched stuff like that they offer or they take the namespace that we give them as a service and they also Get the compliancy on top of that so they know that the platform. They are running on it has been pen penetration tested It has all the risk controls in place and so on It does mean that these teams we have a lot of them We have hundreds of teams that run that use the same Kubernetes cluster They are in control of the stuff that they deploy in their namespace so the compliancy aspect of the cluster Yes, that's in the container hosting teams control but anything that you Deploy on top of that. That's the the application team needs to maintain that We will zoom in on that later if you want to know more about what it's like to run on the container hosting team There will be a talk from Adnan tomorrow and he will give you the journey So what use cases do we actually support on our clusters? We have two main use cases one of them is 12 factors. I'm wondering here a show of hands Who here knows what a 12 factor application is? That's quite a lot. That's good. That's good for those who don't know 12 factors The easiest way I would describe it is that you focus on your app being stateless So that makes it very easy to scale and if one of your applications gets killed It doesn't matter you just spin up a new pod. So it's all about portability scalability and so on So it means on the Kubernetes cluster itself. There is no persistent volume claims. There is no persistency at all If you need persistency in an as an application you connect to an external database So external as in it's not in the side the Kubernetes cluster, but still within your own cloud, right? So these are this is the main type of workloads that we offer For consumers or for data services providers So these are teams that really know a lot about how to handle persistency how to handle storage For those workloads. We actually offer a storage which is built on top of per port works where they are for us They are completely separate clusters So this namespace is a service offering what I'm talking about. What does that actually mean? Like why do why do you have to build so much stuff for it? You can just do OC ADM new project and you're done, right? Well, not really because in the fact that if you do OCM new project Then how do you know which users have access to that certain project? You need to give people access to it, right? A group maybe you're also gonna need some some networking in there So maybe this namespace needs its own IP address. Maybe you need your own private network and so on We in ING we have Azure DevOps where we deploy our codes to on premise But there also needs to be a connection between that on premise cluster and Azure DevOps So how is that then created and furthermore we have multiple data centers in ING So if one data center goes down we can move to the other one But it also means that if you request an namespace in one data center, it needs to be available in the other So those are actually a lot of steps that OC ADM new project doesn't do Yeah, and we have automated that and that's mostly what this presentation is about To do all of that. We have a lot of components I have listed them here. One of them is the ICHV API that knows all about the different clusters in different data centers How to orchestrate it then we have project controller that ensures that namespaces are on the cluster and Yeah, there's more. There's off-delegator, CDAS controller, image reporter, pod researcher and quota autoscaler There's a huge list of them. We have about 36 at the moment Obviously, I can't handle all 36 of them right here But I will zoom in on three of them. So I'm going to talk about the ICHV API project controller and What is it the autoscaler there in this presentation? But before I do that, I want to show you what is our infrastructure actually look like so For ING we have a data center We have multiple data centers and then we have a team that offers bare metal as a service So they make sure that the physical compute nodes that come in they are registered And ready to be consumed by by platforms And then on one side we have Azure DevOps Well, I should say ING one pipeline And it which happens to run in Azure DevOps and then we have a lot of apis that we depend upon So we have seen the be that's for asset registration. We have a charging endpoint There is some monitoring and logging in there security monitoring And networking so these are all apis that are offered in the ING private cloud Now that we have our infra code so everything for us is infra s code And then we provision nodes via that using an ipi installation running open shift 4.10 by the way And then we get our open shift container platform So this is just this is an open shift installation without any of our own components on top of it yet Then we have all the applications that you just saw they're also In infra s code via git ops We deploy them via args cd on the cluster and then we truly have the ing container hosting platform And you can also see that These components they connect to the apis that are there right the internal infra apis And then finally we can offer the namespace as a service on top of that So we have a cloud portal inside ing very similar to to what the public clouds have Where you can click like hey, I want a new namespace and you go through it And then actually you call our apis and then the consumer has their namespace And of course the consumers once they have their namespace they want to deploy their application So they have their consumer code also in the one pipeline and that goes into the namespace Any questions about this slide? Because then otherwise have to go back No, that's excellent So let me zoom in on some of the applications that we have the first one I'm going to handle is called project controller project control is written in python And it's for namespace configuration management When we originally started building this component We were very well first in python in the team, but there was no python operator framework for kubernetes So we build our own Which we call scaffolds And I will zoom in on that And what the project controller actually does is it takes a specification of a project inside ing So for example, hey, I want a new project with this name and I have I need these resources It's bound to this group and then the project controller it creates all the resources associated with it So it creates the namespace it creates resource quotas it creates role bindings And so on But let me zoom in on the scaffolds framework first So the way the scaffolds framework works is You have your own application. So this is if you're building python code, right? You have your own application and then you import this stream watch the stream watch is offered by the scaffolds package Which listens to a kubernetes object? So this can be a custom resource that you have to find yourself or maybe it's conflict maps or whatever you please, right? and then Via a watch the stream watch listens to that and then it calls event listeners So event listeners also a class that the scaffolds framework offers And that's what you inherit from so you have a bunch of classes inherit from event listeners So this might seem a bit abstract, but if I zoom in like this You can see that you implement a bunch of event listeners So for example in the in terms of project controller an event can be like, hey I want to create a new namespace. So you have an event listener for a namespace You have an event listener for a resource quota And then when a new specification comes in so a new custom resource comes in So let's say somebody tries to create an ihp project Then it goes to the first event listener that says all right I need to create a namespace if that's successful it moves on to the next fund It needs to create a resource quota. All right cool that it moves to the next one creates And now roll bindings and so on and the cool thing is that all these event listeners. They are called in order But if one of them fails So let's say one of them fails then the ones that have already been called they will be rolled back That's what you see there with with the rollback Another component we have is called the ihp api So the component you just saw the project controller. It's very cluster specific It only knows about clusters, but the ihp api it knows about different clusters And that's what you see here. So we have a user it goes to the workflow It clicks. Hey, I want a new namespace Then you hit the ihp api and it calls all those infrastructure apis that I mentioned before And then it calls The cluster api on every cluster Which generates the ihp project spec and then it's picked up by the project controller and then we have all these resources there Well, that's a lot of orchestration a lot of things that need to happen And all of it is of course time consuming First let me take a look. So this is what the api spec actually looks like So these are the things that we offer you can get some information about your namespace You can create one update it and delete it and you can also patch it. For example, if you only want to update your resource specifications, then you can do that But in order to go through all these steps Or in order to get the proper namespace these are the steps that we need to do So a request comes in and if we first need to get some network information And that network information needs to be registered in the seem to be so that's the asset registration part And then lastly if that is all complete then we need to charge for it and we actually need to create the namespace on the cluster Now all of these stages they happen sequentially So only after stage one is completed. We move on to stage two and so on But all of the units in a stage so these blocks on the left and right there are called units. They are concurrent so Registering stuff in the seem to be it takes a second or so, right? But we do it all at the same time Same for that to speed up the process But it does mean that if we are in the first stage We don't want to wait all the way up until stage three to find out that hey, maybe there is a naming conflict, right? So at the first stage we already do a check stage we call it So we do some sanity checks if the request is likely to succeed And only when the request is likely to succeed then we continue with the run stage And then we do some actual networking and so on So, yeah If the flow is all good, but at the last time, you know, like at the last unit something fails Then we are in a little bit of trouble because we have executed a lot of actions Right, we've created we have done asset registration already and the networking is done But somehow we cannot create this namespace on the cluster In that case we need to roll back everything that we have done so far Now what does that actually look like in code? So all the steps that you just saw in in the diagram. These are it is in code So we have these the first curly brackets that is a stage block so to speak And you can see the network creation step there And then we have the other steps that seem to be create The charging and the namespace, but they are dry runs in the first stage And then we have all the other steps, right? I hope this is Clean to see and then finally we execute Everything when the in the run stage cluster actions and reply call All right And the last component I want to show you is called quota autoscaler. So We found that on our clusters Or when you request a namespace The requester of the namespace has to fill in how many resources is your application going to consume And at the moment where you're requesting it, it's very hard to estimate, right? Some people haven't even started writing their application yet and we are already asking them to give an estimate on how much they are going to use So that's very tricky And on top of that Then you start building your application, right?  That's really tricky and to help with that. We have a component called the quota autoscaler So what we do is we look at the namespace resource quota So a resource quota is basically You know, I hope everybody knows what a resource code is because we're all kubernetes. But in case you don't A resource quota is basically a limitation of the compute resources that your namespace can consume So the sum of all the pod resources in your namespace, they cannot exceed what it says in your resource quota Okay, so this is also what you pay for by the way in the in the ing private cloud everybody pays for their resource quota Now if we notice that that resource quota is getting full We can watch that using this ishp quota scalar component And when it's almost full we can automatically do do calls to our Charging endpoint so to the ishp api we say hey this quota is almost full This team needs more resources then we call our automation and which increases the resource quota This behavior is managed by a custom resource called a quota scalar object Which looks something like this Um I hope it looks really familiar because it's almost the same as a horizontal pod outer scalar So you have some behavior and you can also set some minimum Values and some maximum values And you have to read the behavior like what is the What would you like the cpu ratio to be? So If you set down the values 50 and 70 it would mean that I want my resource quota to be between 50 and 70 percent utilized If you put down 100 percent it means you want the 100 efficient usage of your resource quota That means that yeah, there is no space for pods essentially to come up. You're fully using your resource quota I want to share some metrics with you with how that affects the resource allocation on a cluster So not too long ago. We were running on oba shift 3 11 And this was what the cpu allocation looked like so we have over 9 000 cores allocated in resource quotas These are the actual Request so this is what the resource quota looked like what the the namespaces have what you pay for And then the request is the sum of all the pods like what do the pods actually request And you can see that teams highly overestimate how much resources their application actually need And you can see the usage there As well, so you can see that the memory usage in terms of request is quite good And for cpu, uh, yeah, there is a lot more burst to it And now we built another cluster an open shift 4 cluster Uh, and there we implemented the quota scalar and you can see the difference, right? So we went from 9 000 cores to Give or take what is it 2000 or 1700 core and also for memory So we users no longer have to worry about What they fill in for the namespace resource quota. It kills automatically And one other thing we did is that for dev and test workloads We saw that there were a lot of high cpu requests on a pod level And this also allocates the resource on the cluster, but for dev and test namespaces Yeah, you don't they can be a bit more Flexible right so for dev and tests we automatically scale down the cpu to 10 millicores But we still allow users to set their limits as high as you can So for those who don't know if you request resources in a pod You have resources to request requests are resources that you are guaranteed to have Whereas limits they are reached only if the resources are available on the cluster And since we have so many compute nodes on the cluster with us you're Almost bound to hit your limits anyway So this allows us to Gain in this margin here So we have the request for v1 and then for dev and test We force it lower and that gives us a nice bump there Obviously for memory we can't do that because we can't if you memory is not as compressible as cpu So here are some other metrics. So here you can see the quota scaling and there you see the the pod resource mutated I was talking about By implementing the quota scalar feature we save 7.6 thousand cpu cores In namespace resource quotas and with the mutation there for the dev and test namespaces we save 500 cpu cores in in the requests Yeah, so I already mentioned this right so the pod resource mutator for development and test namespaces only Forces the pod cpu request to be 10 millicores But cpu limits can be capped as it is All right Last but not least I have a demo It will be a very quick demo It is a video but it shows the full stack as I've just told you in in the story So we're gonna requisition a namespace Via the ishb api After the namespace has been requisitioned We're gonna scale up some some workload in it And then we're gonna see the quota outer scalar in action All right Here we go So there we go Here is a namespace specification for the ishb ios so it has a name We have some workload type we have some resource that we wanted to have Then we post this payload to our automation right we create a namespace So this creates the namespaces on multiple clusters We can also see what operations it did So we can see that there was some networking in there. There was some asset registration and finally It was created on the cluster Those are all the steps Now if we actually look at what what is on the cluster for the automation on the cluster it creates its ishb project spec Again, it has the the same name. It has the same quotas It has the same workload type And there's the name and this is then picked up by the project controller And it that creates in turn creates the namespace It creates the resource quota So you can see that whatever in the specification is reflected in the resource quota there It also creates role bindings. So the what groups have access to this namespace And you automatically get a quota scalar Now we have our namespace. Let's deploy something in it. So I have this nginx pod here It has some cpu Request it has some cpu limits So let's make it happen It's starting up And we can see we are using some of our quota right We're using 250 megabytes out of one gigabyte and our quota. We're using one out of four cpu limit So we can scale up to four replicas before we hit our resource quota So we are completely full right now if you try to create another pod it will fail because your resource quota is full So we have four pods running And now if we scale it up to six We get all these scary error messages because well, you're it's forbidden, right? You're trying to break out of your quota But this is where the quota autoscaler comes in It's actually listening to these events and it knows that you are trying to scale up outside your quota And it sees here it calculates based on the specifications of the replica set How much resource is it actually you need more? And then it calls our Our automation and it updates the resource quota And we can see it's actually updated there And all the pods are running So that's what you mean that what I meant with the 100 efficient resource quota usage Yeah there's more I have I have one more slide so I showed you a lot of stuff. I showed you some python code. I showed you some some other stuff I mean, it's really nice to boast about it was not very useful for you unless you can touch it yourself So can you have all this code and the answer is yes. Yes, you can So at at kubecon we will open source this live Yes, very cool And what will you actually get so you will get the code for the quota autoscaler the full component of the quota autoscaler you will get it for the The operator the python operator framework. We will also open source it. You will not get a project controller yet We are also planning on open sorting yet. Maybe but for now you will get the python operator framework and for the The orchestration you saw we had all these lines where we have like these stages that we run sequentially and the units all concurrent That's what we call orchestration and that will also be open source so yeah That is it. So now you can Are there any questions there are questions But you guys don't have microphones, so that's uh, thank you for your talk I want to ask do you use this in production because I've seen data from non production and I wonder if I if you already use that Yeah, we we use everything in production what you see here But the components they are all replicated across different classes. So for dta There is an instance of the ishp api quota autoscaler running and for production. We have another instance running Across multiple clusters as well. So yes, everything you see is is running in prod I wonder why the choice of giving the data for non production is a little production because the It will be more impressive maybe to see how much resources you save In production. Yeah, the the reasoning for this graph is that We notice for dev and test a lot of resources are allocated Um, and for production workloads, we are not very eager to scale down the request Like we if a team says hey, they need that many resources to run in production. We believe them We don't we're not going to touch that word for dev and test. We can be a bit more aggressive Um, and yeah, that's why those metrics are a bit more interesting, but For the production metrics, they are Perhaps a bit less Because you don't have the pod resource mutator, but for the quota autoscaling level is still Very beneficial Yeah, you're welcome And we have a similar workload And uh, I wonder well, we made some different decisions and we used actually used some Market solutions like capsule for creating namespaces automatically by The developers and themselves also the vertical pod autoscaler But the last one. Yeah, I It's not working properly for us So maybe this is Convenient to use but I'm wondering why are you? I mean why is Angie? Developing these solutions themselves because there are some solutions in the market Yeah, so for The namespace as a service part. We have a lot of Interfaces inside ing that are very ing specific like for example our our networking implementation That that yeah, it is just very ing specific. So we cannot ask a vendor to to build that for us Uh, and for the second part of the the scaling part So it actually works with the horizontal pod autoscaler as well as vertical pod autoscaling you can use both of it And this quota autoscaler kind of runs on top of that So you can see it as like a management for your namespace research quotas instead and We talked actually with multiple vendors and multiple companies And none of them have built it yet. So Therefore we did it and you can also use it because when it's open source So I hope that answers your question Okay, thank you One question Because I saw that there is a cmdb record against the namespace So can uh with this orchestration can you get multiple namespace per? cmdb record or it's one namespace per cmdb record Yeah, so what we do is Because we run many many clusters And all these clusters they can have the same namespace right depending on how many times you want it replicated And what we register in the cmdb is we combine the namespace name with an identifier of the cluster that you have So let's say your namespace is example Right, and then we will have example one two three depending on like one two three being the cluster identifiers Does that make sense? Uh, yes Hi, thanks for the talk Do you have any restrictions on the upper limit for that quota autoscaler in case of misuse or security instance? Yes, yes, we do and uh, I'm sad to say it's also necessary Yeah, there is a there is a limit. We have uh from the top of my head. I think we have like 35 cpu cores and Like a 150 gigs of ram per namespace Uh, it can be that some of the workloads that we have They actually have an extremely high load and then they use all these and perhaps they need more and usually These are yeah big consumers that we talk to them about their use case because if a team hits these limits without talking to us Uh, then usually they are over committing the resources that like they are requesting more resources than they actually need So this limit is in place And then we talk about it on a team to team basis if they need more We have a question in the back in the back of the room robin. Oh, yes Hey, you did explain about the support for 12 factor applications I was just curious that iCHP support stateful applications as well or Yeah, so we currently only provide stateful applications for data services So for most of our consumers, it's all stateless 12 factor but for With inside ing we have providers that offer data service. For example the elk stack. You might know the elk stack So these guys worked in ing for five years Okay, so the yeah, so the elk stack is a consumer of persistent storage It uses the same automation in in the end Robin, are you okay to take one last question? Yeah, of course Thank you. I have a small question since you're basically in the Like private cloud service provider How many clients do you have at this moment more or less? So inside the cluster we have a little over 2000 namespaces And I think it together with the non-prote and prod we run about I think it's roughly 5000 pods and that's per Environment, right? So we replicate everything in another data center as well Release the metrics you're looking for or I was because I was curious actually about the more technical Part of this which is the persistence of the kubernetes clusters that you're using under the hood. Basically Did you go with the custom solution regarding the Storing of the state or are you still using the standard one which is etcd? You were still using at cd. Yes, but for the So are you referring to the storage we use for our own apis or the storage we offer for our data services consumers? No, the internal one specifically for the need of the kubernetes clusters. Yeah, it's at cd. Yeah It's okay. Thank you And we found at cd. I think it's it's performant But it's keep a good eye on the metrics and you may also need to fine-tune it, right? There are quite some nice articles about it where you can say, okay I have a huge cluster what metrics do I need to fiddle with? Or what? parameters, yeah all right If you have more questions, I'll be walking around still so Thank you very much for listening. I have a great day Well, thank you very much robin I am a hundred percent sure that people will come find you Not necessarily for good things we have Now we're going to restart at 2 30 with ara and she's going to show us about The gateway api so you can stick around you can go get some refreshment. Please be on time We're going to start 230 sharp. Thank you