So if you haven't noticed yet, there is coffee in the back of the room, thanks to our friends at Databricks, VMware, and Mattermost.  For those of you getting in just at the end here, just a reminder, we do have coffee in the back of the room from our friends at VMware, Mattermost, and Databricks. Cool. So let's go ahead, we're just a minute past 10, so we'll go ahead and get started here in just a second. We've got a couple administrative items and then as folks join us, we'll kick off with our keynote today by Demetris about making our community more diverse and inclusive. So thanks again everybody for joining us for the second, for the Sunday morning keynote here at the Hilton LAX for Scale 19. As a reminder, we have another keynote closing this afternoon with Dr. Vint Cerf and if you're not familiar with him, he is one of the creators of the internet. I think all of my early reading about the internet and CS and as I was getting started, I'll talk about him. So this is a very exciting moment for me, also with Demetris as well as we talk about how to grow our own communities. If you haven't had a chance yet to head by the Expo floor, there's plenty of time after this session as well. Please do go by and thank our sponsors for helping make this conference possible as well as all the community groups that join us to present, to share about their open source projects, their lugs, their meetups, their initiatives, they all put in a ton of effort and so wanted to take a minute to also share some of the winners of the Scale exhibitor prizes. We surveyed you all about what you enjoyed at the conference, what you enjoyed on the Expo floor and the ballots and results came in. And so without further ado, we'll drop these off at your booths afterwards, don't feel like you have to rush the stage. The people have spoken and OpenSUSA has won best presentation at their booth. Our friends at the Linux chicks have won the prize for best spirit. We are very thankful to Spot and the rest of the AWS team for all of their support this year and they have won most memorable booth. If you haven't been down there, AWS is doing some amazing things with open source, they've also got some fancy sodas for you and other cool swag. So go on by and talk to them about how they're helping build a more open cloud. GitLab has won most interactive booth and so it sounds like there's some fun interactive things to do at that booth. I encourage you to go by before the Expo closes at 2 and try those activities out. And then finally, people really love the swag that came from the CNCF, the Cloud Native Computing Foundation. They were working with a bunch of Lego sets of all their favorite Cloud Native characters and I saw some people building some exciting things with them both at game night and on the Expo floor. So again, go check them out as well. So we will bring those certificates by your booths later on, no need to rush the stage just now. And so on that note, again, we're for next year, we are planning to have scale back in Pasadena. I know everybody enjoyed this blast from the past here at LAX, I hope you got your in and out in and anything else that you enjoyed from this particular area. But next year we will be back in Pasadena with all of the fun activities that that brings with it. So again today, we'll be here until 4 at 3 o'clock. We will have Dr. Vint Cerf's presentation about how open source helped build the internet as well as some fun raffles and giveaways. We've got a new laptop from Purism that you have to be in the room to win, so I encourage you to come join. And a closing sort of happy hour with drinks and snacks and other things. So again, we'll see you all here at 3 to 4, give or take. But let's transition to what you're all here for this morning. I know we all braved game night last night and stayed up till the wee hours playing video games and winning at casino games and escape rooms and such. But we're here this morning early on a Sunday to learn all about diversity and inclusion. And it's one of the most, I think it's an important topic for us as a community as we try to figure out how do we continue to grow it. We can't just continue to build a community of just ourselves. There's lots of friends that have come from different backgrounds, different perspectives, different areas of interest. So we thought it was an important opportunity for us to learn with Demetrius and later in the conversation with Karsten how we can all continue to grow and contribute to that diversity and inclusiveness. So without further ado, I'd like to introduce Demetrius, who flew in from a very special family occasion just to join us today. And she'll be sharing with us a little bit about the work she's doing at GitHub and her past and elsewhere. And so thank you very much. I appreciate you joining us. Thank you. Thank you so much, everyone. I'm delighted to be here. Oh my gosh. I don't think I've ever got a yell before at the beginning, so I appreciate that. So wanted to let you all know just so you'll know you'll have time to ask us some Q&A. I'm going to speak for about 20 minutes here. And then my friend Karsten is going to join me for some Q&A to hear. And I have been more of a fireside chat type style. And then we're going to save about 20 minutes for you all to ask any questions that you may have. So anything that comes up for you during this presentation, please just jot it down and I'm happy to answer that afterwards. So OK. Let's get started. Let's open source diversity and inclusion and figure out what the heck did I mean by that. So again, I am Dimitris Cheetham. I'm the Senior Director for Diversity and Inclusion Strategy at GitHub been there for about two years now. And I have to tell you about my time at GitHub, specifically when I was interviewing. So I was interviewing with Erica Brusha, who sends her love and regards for scale, by the way, when I told her I was coming here, she was like, Oh my gosh, I remember when it got started and all these things and they are my friends.  But Erica Brusha was the COO of GitHub at the time where I was interviewing. And so she was interviewing me and she asked that all important question. You know the question that's going to come up in every single interview you do. She said, Dimitris, why do you want to join GitHub? You see, she had just finished talking to me about how GitHub was actually approaching diversity and inclusion in a much more holistic way. Most of the time when you hear about diversity and inclusion, they have it right in HR, which is what I used to do at Red Hat and it's so important to be there. But GitHub was now approaching it from looking at their platform. When I started, it was 50 million developers a little under two years ago. And now they're at 83 million developers. They were looking at it from their philanthropy or their social sector, their giving. They were also looking at it from people, which is their HR perspective. And they were also looking at it through their policy efforts. And so when she asked me why did I want to join, I gave her this little tongue in cheek answer. I said, Erica, I want to join GitHub because I want to open source diversity and inclusion. And I smiled at her and she smiled at me and I guess it was a good answer because I'm here. So after I started, you know, they let me go through my own boarding, you know, figure out where the virtual bathrooms were on Slack and all those things. And then all of a sudden she said, okay, Demetrius, now go open source diversity and inclusion. I was like, oh, you remember that? What does that mean? I didn't know. So I just started doing what I love to do anyway. I just started talking to as many people in open source that I can. I got on Twitter, anybody's name that I saw talking about DEI and open source. I was DMing them, asking them could they just have a quick coffee chat with me. And over the course of hundreds of conversations, there were some themes that started to come out. The first was inclusion happens at the community level. You see, when someone new comes into open source, their first one or two interactions in their first community really sets the tone on whether or not they're going to stay in open source. If they have a really positive and inclusive experience, they say, you know what, those open source folks, they're my people and they're here to stay. But if they have a negative interaction, someone ignores their contribution, someone doesn't respond to them, they say, you know what, I don't feel welcome here. And they're gone and researchers saying they're often gone for good. And so then when we started looking at the community level, I said, well, who's responsible for driving inclusion in the community? And overwhelmingly, people said maintainers or community leaders were the ones that's responsible. You have to think about maintainers or community leaders like your managers, if you're working for a corporation or organization, right? You can build all of these amazing corporate programs, these trainings that everyone will have to take, these mandatory, you know, tutorials and education materials. But all of a sudden, if you take these mandatory trainings and you go back into your team, your day-to-day experiences will undo anything that you've learned on these trainings. So it's the manager. You need me to stay right here? I'm not moving. Okay. And I'm a walker when I talk, so I'm just going to keep going back and forth. That's good. All right. So it was really those managers that set the tone for inclusion. So when I started talking to those community leaders and those managers, I said, well, what do you need? What can we give you? And they were like, please don't come with more trainings where they were like, we have education materials, there's podcasts, there's articles, every conference we go to. People are talking about it constantly and constantly. We are overwhelmed with material. And so I said, well, if you have all the resources you need, what's the issue? And that's where the conversation started to just break up a little bit. So when I talked to managers of smaller communities, they said, yes, we have the resources. We also have the influence with our community because we're speaking with our users and our contributors almost on a weekly basis, but we don't have the time and we don't have the bandwidth. We are all about trying to get hands on the keyboards, contributors from wherever we can get them from. We don't have time to focus on inclusion. So I said, okay, well, if this is what the community leaders of the smaller communities are saying, what about of the larger communities? And what they shared was, yes, we have the resources. We have the time. We have the bandwidth, but now our community has gotten so large, we can't easily influence the culture.  So you have this tension point. The time in which you needed to focus on advancing inclusion in your community was the time when you didn't have the time to do it, right? So what do we do? But one thing, no matter what community leader I spoke to, large or small, they said, here's what we know. Regardless of what we do, there are still some really, really great barriers to entry from people even being able to get to open source. So they said we can build the most inclusive, fun community there is, but if we don't address those barriers, if we don't address that wall that's there, no one's even going to be able to get to our communities. So this was a classic case of if we build it, they will come. Not if there's that big wall that's there in the middle. So that's what we wanted to address. That's where the equity comes into play. And so that's what we've set out to do with this open source community called All Land. All Land is going to answer that question and do that thing that I said in my interview. We're going to open source diversity, equity, and inclusion. And we're going to do so through access, community, equity, and data. So we kicked off All Land this community about a year ago as a matter of fact. And so during that first year, we said we wanted to do three things in which we have checked off all three of those and moving forward. So I'm here to share about those three things. The first thing was we needed to kick off an open source diversity, equity, and inclusion survey. There were a lot of surveys out there. There were many for individual communities, but there wasn't really a broad survey that kind of cut across all of the different open source communities. We had done a little bit of survey work on our platform at GitHub, but I made sure to let them know GitHub is important. We are a major player, but we're not the only player. There are so many communities on other platforms that are in other forms, and we need to understand what their sentiments are about diversity, equity, and inclusion, if we're going to advance inclusion and open source. The second thing was we kicked off a maintainers listening tour. Again, if maintainers are the ones that's instrumental in advancing diversity, inclusion, and open source, we really need to hear from them in a more structured and formal way. And then the third thing, which I'm so excited to tell you about because we are almost finished with this one, we kicked off an all in 12 month pilot, and this first year was focused on students, and we're getting ready to kick the one focused off the one that's focused on maintainers. So let's talk about this survey. So again, we didn't want to just do it right on GitHub's platform. So we partnered with the Linux Foundation, and we introduced this survey over 60 some odd questions, close to 70 questions.  If you say that you don't have data for open source, diversity, inclusion, and open source, you have not looked at this report, please go there. And so when we did the survey, it was some surprising things that came out for me because I've heard so much about how open source is not a welcome inclusion. There are some really big issues of things that's making it so that it's not a great place anymore. But when we surveyed, and it was a representative sample, we found that 82% of survey respondents said, I feel welcome and open source. They agree with this statement. I can tell you this, if you go to any company and they get an 82% approval rating from their employees, they're going to be popping bottles and there's going to be confetti raining from the ceiling. That is a darn good number. But guess what? That's not the work of open, that's not the work of diversity and inclusion. We needed to give voice and hear the stories of the 18% that did not agree with that statement. What were their stories? What were their experiences? What were they facing? Because I can tell you, wasn't that 18%? That's probably what's keeping millions of others from even attempting to come into open source. Those are the stories that they're oftentimes hearing. So we needed to hear from the 38% of non-binary or third gender contributors who said that they didn't feel welcome and open source. What are the 29% of people of color, specifically in North America? What are their experiences? 26% of women said that they do not feel welcome and open source. 25% of persons with disabilities. And we can't forget about that 15% over there of men that said that they did not feel welcome and open source. And amazingly enough, that number was composed mainly of white men in North America. And so that was something that was just mind blowing to me. That's where we had to dig in. That's where we needed to see what was going on. And so we got over 10,000 written comments that our data analysts went through. And here's what we found.  I started, sorry. Just need to like one of those little stick things up here. OK, so here's what we found. Most people did not feel welcome if they didn't have the technical skills and knowledge that they felt was necessary to contribute to a community. But the interesting part about it was when I talked to maintainers and community leaders, they were sharing with me all of these very important and critical responsibilities of a community that did not require technical skills. Things that people could do while they were building up their technical skills. So why was it that when someone was coming to the communities without the technical skills, they were made to feel inferior? That was something that we knew we needed to look at. Lack of response and rejections to contributions. The interesting part about it is, well, contributors were saying, I will send an email or a response or a message to the community leader and would never hear back. Or they would wait three or four weeks before I could hear back. And so what the maintainers, especially in those smaller communities, they were like, we're drowning here, we're busy. Like, we don't have time to respond all the time. It's not because we're disrespecting them or don't value them, we just don't have the time. Your new contributors don't know that. If you don't have your documentation that's setting up expectations for when you can respond, how to engage, they're thinking that you just don't want them into your community and nothing could be further from the truth. Along those same lines, especially amongst women, they just felt like their voice isn't heard. Their contributions aren't valued. There was one comment that really, really stuck out to me. And this person that responded to the survey said, if I am not white, if I am not male, if I am not rich, if I am not highly educated, no one wants to hear from me. And we know that an open source, that is not the case. But that's how a lot of the people that were responding to the survey were feeling. And finally, microaggressions and stereotypes and written communications. This was a really, really big one, especially when you started looking at international respondents. So you have really high numbers of feeling welcome in India. You have high numbers in the continent of Africa. But when you started looking into Spanish-speaking countries, especially South America, Portugal, Brazil, in which the numbers that are people contributing to open source are going up higher and higher every day. They said they didn't feel welcome at all. And most of the time, there were language barriers, cultural barriers there. So those are some of the things that we need to be thinking about as well. We have to get to the point that an open source inclusion is not the exception. It is the norm. Like it's not looking where we're seeing, you know, giving awards for people being exemplary. Like everyone should be doing that. And I think that it's possible for us to get there. So the second thing that we did was the maintainers listening tour. And so we kicked off a series of focus groups, individual interviews, as well as an online form for people to submit feedback. We asked them questions. What do you need? What are some of the challenges? What's working? What's not working? What are the gaps? We got so much rich information in that. And so I released a talk and you can find it on our website on some of the top six things that we've learned, like those six themes. But I can tell you, we're doing the final report of it every day. I keep looking. I think we were supposed to release it back in May. And I was like, it's too much information. And they keep telling me to cut it and cut it. I was like, but I can't. So we're trying to figure out the best way to release that. Your maintainers and your community leaders, they actually have the answers. They've just never been given the opportunity to communicate it. And so I have a meeting next week with some leading researchers at some universities because I said, I need you all at the table. They're giving us the sentiment. You all need to give us the data and the research. Then I'm going to the corporations and the foundations to give us the money to build the solutions and tools that they're asking us for. Like one of the things that they said was, hey, everybody's telling us how to solve for diversity and inclusion. And everything they tell us is this long, manual process that'll take us hours to do. They were like, can somebody build a tool or a script for that? And I was like, oh, we are developers, aren't we? And we just need to give someone money to develop the tools. So at GitHub, we definitely will be kicking off a major grant program for developers to build some of the tools that these maintainers said that they need. And we're going to make sure that they're freely available to everyone that needs them. So I want to talk about the next thing. All in a 12-month pilot. We were going to do one for the students and the maintainers within the first year. But we got so involved with these amazing students that I'm going to tell you about. I told the maintainers they have to wait. And we're going to kick that off in a couple of months now. So with these students, we went to universities that are usually overlooked and no one is showing up on their campuses. Those are schools like historically black colleges and universities, and not those top five or six, but those ones that no one's going to. And we also went to a school that was founded for the education of Native Americans in the United States. And so we went to the schools and said, we don't want to put up an application process. Because even filling out an application or an essay, that's a privilege and a barrier in and of itself as well. We want those students that really deserve a chance. And we asked the schools to select them for us. That's all we did. We said, send us their top five students. And we're going to work with them. We're going to give them, during their fall semester, they got open source education, which was open source one-on-one, learned about the language. What is a community? What is a repo? What's a maintainer? How do you make your first pull request? What do you do? All of those things there. And we offered them professional development because you can give someone the technical skills. But there are some things that all of us wish we had known when we first started in our first job, especially in a corporation, right? Like what do you do when you're the only one on a team? Like how do you deal with microaggressions? How do you leverage your employee resource groups? How do you deal with equity when you get it? Most of these students were from very, very underserved communities and under-resourced. When they come out of school, they will be making more money than pretty much everyone in their family. So we wanted to make sure that we even gave them financial education. So that's what we focused on during the fall semester. During the spring semester, we then partnered them with Major League Hacking so that they got experience for 12 weeks actually participating in an open source community. We just didn't give them the theory and the academics and just put them through training. They got experience actually sometimes of learning how to code for the first time, even though they were sophomores and juniors. And they got experience with actually presenting their code, learning their communication skills. And all of this was to get them ready for a summer internship experience with one of our corporate partners. And so we are in week eight of 10, so the students are actually almost finished with their internships. When we started, over 90% of the students in our pilot had never heard of open source before. Over 90% had never had a summer internship before. And I'm proud to say that we place all of them, 100% of them, many of them in their first open source internship experience. And I have to tell you about these amazing students. You know, one of the things about it is if you go to any school, if you get those that's in the top 10% GPA, they'll find their way to an experience, right? Someone will eventually hire them. We knew that GPA is not the only aptitude for success. The students in our program, some of them are commuting two hours each way to school every day because they can't afford to stay on campus. Some of them are full-time caregivers, especially during the pandemic, where they have to live at home, or they have students, or they have kids. We have full-time parents that are in our program. You have students that are working part-time, full-time, or they are active duty military because that's the way they're paying for college. And then you also have some that were on academic scholarship because my question to them was, why don't you have an internship? And they were like, we have to train during the summer because otherwise, we can't afford to go to school. We're here on scholarship. And so I'm like, so if you are a scholar athlete, that takes you out of the ability to even be able to participate in an internship. That baffled me. Guess who can solve for that? A very asynchronous community like Open Source, right? So we have some of our students that were on teams that might be in another time zone so that they can work flexible hours. Some of the students had to pick up college courses and we managed to work around their schedule. That's what it meant to be all in. That's what it meant to give students who are off the overlook who just need that shot. We just wanted to crack the door open for them so that they could just have an opportunity. And we did that through the amazing partnership of schools, organizations, and corporate partners. So whenever I show this slide, I say this is what it looks like to open source diversity, equity, and inclusion. And we have plenty of room up here for other logos and there's some that's gonna get up at it already. But then when I look at this inaugural class, this is also what it looks like to open source diversity, equity, and inclusion. This is our first cohort of graduates of all in. And guess what? We started with 30 in the pilot. We are growing this year by 10X. There will be 300 students that will be going through all in. And I have to admit I settled on 300 because Microsoft and GitHub was like, when are we getting to 1,000 and 10,000 please? And I was like, whoa, whoa, whoa, wait. Because one thing about it, I just told you about the experiences of these students, giving them technical education in a training course, that actually was still some gaps that we didn't see. That's why I wanted to keep the pilot small. It never occurred to me, it does now, that if you have students that never had an internship and you're gonna throw them into a corporate internship for the first time in a virtual experiment, what does that mean for them? They're like, oh, you wanted me to show up at work? Like every day at nine o'clock? I thought this was like a class schedule or whatever. Just things like that, their elevator pitch, communication, there were some students that they needed more of an intensive program. So next year, what we're gonna do is for some of those students that need more of an intensive experience, we're not just gonna send them their first year to a corporation. We're gonna build, I'm gonna make this name up right now, like the Open Source Academy, where for 10 weeks, they will be with us getting more and more in-depth experience, more and more training, giving them what they need. That's what it means to be all in. They are not a number to just push through, push through a program and say you graduated, we're gonna get them job placement. If they need training, if they need training for public speaking and presentation skills, we're gonna get them that. We have some students who identified as neurodiverse. We got them with training with a neurodiversity coach, someone that was neurodiverse themselves, because we didn't want them hearing from anyone that they needed to train and assimilate in order to fit in. We did strengths-based coaching. Let us understand who you are and what you need, and then we're gonna help find the right opportunity and the manager for you. And when I say that I figured out how to get them that coaching, it was through my friend Karsten and Sam Kanuta at Red Hat. I called them. That's what Open Source and Diversity and Inclusion mean. I'm not sitting here in a silo trying to solve a solution for something. I'm calling on friends, everyone in this room, to help us. And so I wanna make sure that what you're hearing today is understanding that this is not a program, it's not a set of programs. All in is a community. It is a community of people. It's all of us coming together with your best, your thoughts, your ideas, your energy, your resources, the money from your companies and organizations, jobs, whatever it is. That's what it means. So we need all of you to participate. Here's how you can participate. You can join the community. Go to AllInOpenSource.com. If you are a company or an organization, you can agree to hire interns next summer. Or if you wanna contribute financially, we have ways that we can take your money too. And we need it. Because all of the students, not only did we give them those things that I talked about, each one of them received professional head shots. We actually had a photographer go to their schools because a lot of them didn't even have head shots to go onto their LinkedIn profiles. And we wanted to make sure that access was not an issue as well. Every student received a free MacBook as part of the program as well. Because what we learned were some students didn't take internships over the last couple of years because they didn't have a laptop at home. So we have to make sure that we're addressing those things as well. So lots of opportunity for you all to join. And so please know that I extend this invitation. Less open source diversity and inclusion. Thank you. So I'm gonna be joined up here for Q&A with my friend, Karsten. I moved all these on because I thought I was gonna be walking all over and then they told me to stay still. So do we need to go over there? We just need to sit down though, Karsten. Well, you can swing your legs if you want to. And so we have about 30 minutes but I wanna make sure. So Karsten is gonna ask me a few questions but if you all have questions just raise your hand and we'll interrupt here as well. Yeah, thanks. We'll see if any of these questions are ones that people got in their mind and then we'll do the cleanup. Okay, well good morning. Good morning. Thanks for joining us at scale. This is one of my favorite shows that I've told you before. You've told me. Like five or six. Exactly. Yeah. So Jumita, I know that you've had a really successful career in serving in the chief of staff and in senior director level roles and what prompted you to launch the all-in open source community at this point in your career? Yeah, so when I, maybe about five or six years ago, I had the opportunity to really think about what I wanted to do next in my career. Typically I haven't had that space but one of the things I wanted to do is I wanted to go back to my first love which I was a computer science major in undergrad and through a series of twists and turns which we can talk about cause I think that's an issue with inclusion as well. I kind of got away from tech but then I said I wanted to go back and it was funny enough I moved back to North Carolina I signed up for my first program in class cause listen y'all I was a really good programmer C++ which totally dates me but I was like I'm gonna go and be a programmer again and I think I was like an hour into the class and I was like so I'm gonna be on the business side of tech again. And so then I just thought about what would really make a difference and you hear all sorts of statistics about people of color, women just being underrepresented in tech and specifically with open source I know how important it was for programs for me coming from a really poor background you know it was so funny when I graduated high school my mother with three kids in college was making $13,000 a year. And so I just think about all those program project breakthrough which is a program for HBCU students that IBDM had I was one of their interns project uplift there was so many of these impact for programs and it's so funny when I hear about those programs most people are like are they really effective? And I'm like ooh ooh ooh they're very effective. Like a lot of people like if you can't scale it if you can't help millions and millions of students it's not work the ROI. And I was like without those programs you wouldn't have a me and you wouldn't have a lot of my fellow classmates. And so I said I wanna join an organization which is gonna let me figure out what type of program I wanna create and that's why I created all of them. Yeah that makes sense. So then it's not really hyperbole to say that the advancing diversity, equity, and inclusion in technology in particular has been a part of your life's work, right? It's a part of my life's work. Yeah I live and breathe it every day and my daughter comes like every day she's like what are we doing with all of them? Mom she's 11 and so she'll go to school and start talking about it as well. And so this is what we talk about at the dinner table at night it's my life's work. Oh I love it. And so in your professional roles and your open source involvement were we affected those things so but what are some of the trends that you've noticed that you've got more as you've got more involved in this effort of trying to make open source more inclusive? Yeah I mentioned it briefly but one of the things that I was just I don't know I was astounded by it. There's a lot of people that's working on diversity and inclusion in tech and even in open source. There's a lot of articles a lot of there's a lot of money being spent. And so when I look at how much effort and finances are going to diversity and inclusion and then I look at the numbers as my 11 year old said the math ain't math then. It's like something is happening here and what I started seeing was that everybody's focused in the same areas like that top of the funnel. They want to give students internships or they said we're gonna do this or once you graduate then we're gonna have all these amazing opportunities for you. There are so many hurdles that students and people have to get to to even get to those internships. I love when a company says oh we're gonna go to this school and we're gonna give them all of these certifications for free that they can take and I was like but they don't have laptops and they don't have wifi on their campus. How are they, they don't even have professors that's teaching their coding classes. How are we gonna get them to your amazing certifications? But no one wants to do that work. That's the long term that's not the easy ROI. That's not the work you see in the headlines. That's where you might only impact five students at a time. No one wants to do that but that's where the equity lies. So that's when you start hearing a lot of people take away equity. They'll say let's just do diversity and inclusion or let's just do belonging or let's just do justice but they don't like that equity word because that's where the hard work is. Yeah and that's what struck me first when I thought I'm like wow how are we gonna work on equity? That was the, yeah that was a tough one and also when we first met you were in the middle of your maintainers listening tour and I know you've picked up a lot of things in a lot of ways. What are some of the interesting things that you would highlight or would call for all of us from that? Yeah I think for the maintainers I mentioned earlier like they kind of know what they need and they had never been given the platform for us to really listen to them. Some of the amazing things that I saw come out just in a listening tour especially in person there was one, I don't know if you were at this one it might have been in another conference or I did the listening tour where one of the maintainers she raised her hands small community and she was like one of the big things that I have is that we'll have someone that's new to open source they're coming to our community we give them opportunities, they get up skilled they start learning, they start doing really impact for work for us and then they're hired away by our company and we celebrate that but then all of a sudden we have this big hole or gap in our community that no one can feel and oftentimes they're doing a lot of the work that we need for them to do the advanced diversity and inclusion within our community so this one maintainer she said they had someone that was working on removing the dead names from their code and they were doing that and they got about 80% done 80% complete and then they were hired away and so that project has just been languishing for over a year now and all of a sudden someone from a very large community was like we have like 300 people working on that for us and now they're looking for their next project we're gonna send them over to you next week and they'll get that done for you and it was like a light bulb moment why aren't we doing more of that that literally is open sourcing like diversity that's open source and so one of the things that I found was just this partnership between it's not a competition people want to work together people that are saying Demetrius just tell us what to do and I promise you we'll do it they just say it was so many resources in there that were overwhelmed so those are some of the things that I saw that really really just struck with me on that piece of it and I remember another part of what you found and that really struck me as well was that you found the data that showed that people who were in marginalized communities or minority status were creating communities that were amongst themselves and some folks might see that as a positive outcome but what's the other side what's the concerning side about that yeah so just to give everyone a little bit of context when we were looking at the data from the open source survey we started seeing very high numbers of people of color and women that said that they felt welcome in open source if they were a maintainer and I mean it was like a really huge spike and I was like huh that's something to be celebrate like people who you know make it into leadership roles they feel welcome and so then we started talking to some of the maintainers from the maintainers listening tour and right at the end it struck me I started asking the question why did you start your community and what they said was we tried to join larger communities or communities where we were in the minority and we didn't feel welcome so therefore we went and created our own community over here and then I said are we creating separate but equal like not intentionally but that's what is coming out as and so now we have these larger communities that are like we need more people in leadership and those that tried to be in leadership are like nope we're not joining your community we're going over here and creating our own so what seemed like something that should be a great thing to celebrate it actually has like some you know some undertones to it that I think that we need to address so when I talk about joining with universities that's what I want them to research I like you know no offense to anybody in the research space and they are my friends are working on some amazing work with them but oh like we do not need another research project on how to create a more inclusive communities like there's like a million research papers on that let's start digging into that work that separate but equal work to see what's going on there that's when we can start you know finding solutions yeah I see can we save a question for the end thank you so well let's shift a little bit because just because I was thinking also about what you were doing when we first met was you were right in the middle of that first cohort of students and I mean you brought tears in my eyes in the pictures and I mean I love the professional headshots like that those polishing pieces are such a huge difference or overcoming biases at the first run but so and there are a lot of camps and educational programs and things out there that are for students so especially in technology and even in open source and what do you see is really the different for students for all in for students compared to a lot of the others yeah you mentioned there's a lot of boot camps out there are a lot of programs corporate organizations that underwise but I think not to sound cliche but we're all in with these students like we just don't stop it was I'll give you a prime example of what this meant it was right after the holiday break the students came back and they were supposed to go with major league hacking and you know do their I think it was 10 hours a week and we got an email from a student and the email was very short and curt it was like I can't do 10 hours a week so basically y'all need to figure something out and most people would have been you know organizations or boot camps would have been like oh you can't make the requirement you're out like it would have been no questions asked but I actually picked up the phone and called her professors I called the professors and said what's going on and they sent me the article where she was in a really bad car accident like concussion protocol she wasn't even supposed to be in school this semester but she said she has to graduate on time because she can't afford to extend for another year so here I would have if I did it just quick if I was just looking at her as a number she would have been gone from the program replaced with someone else who wanted the MacBook and you know the free stipends and the academic credit but I said I told these students I was like we're not letting any of you go I was like this might sound a little cultish but I will be all I will call your we've been on the phone with parents we've been on the phone with professors the deans of their schools we've been involved with them even during the internship process we are really really making sure that we go all in and really address the needs of their students we are not letting them fail but I can tell you that my boss reminds me of every day it's not sustainable like this like you're 30 students almost wore me out so imagine 300 students imagine 10,000 students so that's where we're looking now we're gonna be issuing some RFPs we're gonna have to have partners people that do this and do it well that bring what they do to the table we can't solve it all ourselves one of my pet peeves is when like companies try to like create a university like tech company I'm like you're not a university why are you doing that like go and give money to a university and let them do it well and so we can't be all the things right and so that's what we're doing with all in I think that's what makes it different we're not looking at it at a competitive advantage someone asked me when I first started this program he was a member of a C-suite of a company he was like Demetrius is an amazing program but you're open sourcing it what if somebody comes and steal your idea and just take it and they do it and I'm like yes that's what I want them to do because that means it's gonna be 30 times 30 times 30 times 30 and he looked at me like I was crazy he was like why don't you start a B-court nope I'm not doing it so everything that's in my head I am pushing it out and hoping that people not compete with us but join with us to do it and I think that's the difference with all in yeah and I really feel like you've caught that spirit of like what makes open source work when people are inclusive and being able for all of us to work together through this model I mean yeah that's what sold me on and for well you know I just was oh you're in Carson you're in I grabbed you I love it I love it so speaking of the similar thing of you know we've looked at the little bit of the wall of funders aspect and all those and the participation and where does the open source software funding fit in to the picture of all in we gotta pay people like you know I was speaking at a conference it was virtual at the height of COVID and someone put in the chat and they were like contributing to open source as a privilege and I was thinking that I don't know what I thought they were talking about but I actually picked up the phone you know and call the person and it was on Zoom and scheduled a meeting and she was like even during the pandemic if we didn't have time it's if it's not your full time job like some people still view open source as a hobbyist activity extracurricular activity so if you don't have all of the support that you need at home or if you're not working multiple jobs or you know all those things you don't have time to contribute to open source and so when I started talking to a lot of maintainers they were like there's some amazing people that can help them in their communities but they can't afford to pay them and they have other jobs and they have other commitments so they kind of come and go as they can and I was like I think with all the money that we make in tech I think that's like one of the easiest things to solve for but then you know a lot of times people think you need like buckets and buckets of cash one of the things that struck me when we were doing the maintainers listening to I was talking to some folks that were internationally it was a contributor in Africa as a matter of fact Nigeria specifically and I was like yeah we heard about lack of infrastructure you know and you all don't have access to data and blah blah and she said that's not our problem we got access to data all over the place and I was like oh well what is it she was like it's so expensive to actually connect to the data and that's why we can't participate and I was like oh well how much is that gonna cost cause I'm thinking we need to go build some power lines or something I don't know and she was like oh probably if you gave every person like that wanted to contribute to your community like $200 a year and that would get them unlimited data and I'm like y'all aren't contributing and aren't participating you're shut out for 200 bucks I was like what, what does that even mean and so that's where I think we have to really think about and look at and really break down what is funding, diversity and inclusion look like because a lot of people are thinking it needs to be millions and millions and billions of dollars but sometimes it can be as simple as $200 for data for a year or a MacBook or a laptop for them to do an internship and so we're solving for that even through our corporate sponsors program we get up. Well so shifting to just some questions about the advancing of diversity, equity and inclusion and open source and you know from the broad perspective as a lot of people know that contributing open source software for the first time can be it could be daunting for new people even if it's no matter how eager they are to be doing it from research what are some specific steps that projects can do like the Drupal community that to take the foster that more inclusive and welcoming atmosphere. For any community documentation is gonna be critical people are looking at your documentation as soon as they come into your community trying to figure out how they can contribute how can they be a part of you know whatever it is that you're doing have robust documentation and I think that's the thing that's most often overlooked and then the second thing I would say that we've heard a lot of is just community hospitality when someone joins your community you can have through GitHub actions by the way the shameless plug you can have it where as soon as someone joins your community it sends them an instant message that says hey welcome to our community this is how you can contribute we're glad you're here your voice is you know really really valued here just those small little steps like that that was the number one thing that we were hearing out of the survey to say it can advance diversity and inclusion I think a lot of people were thinking it was gonna say hey close captions for every meeting which is also very very important or you know all these other things that you might not have time but they were like no just let us know that you see that we're here and that could be something that's automated so documentation, community hospitality if you get those two things I think that you are going you know you're well on your way to advance and diversity inclusion in your community and then we can start doing some of the equity things as a docs writer that's maybe another reason why you won my heart over from the beginning with the proof in the pudding so from another perspective you mentioned how crucial it is for open source communities to be sharing and working with each other about their experiences with diversity, equity, and inclusion what are some of the mistakes that you've seen open source maintainers make with diversity, equity, and inclusion? I think the number one mistake that I've seen and it's not a mistake it's just an opportunity I would say you know if I had to change the word a little bit what is it? It was, it was funny and so I you know during the listening tour I was having individual conversations and it was always a maintainer of a large community or a well-resourced community and they were saying Demetrius we're doing all of these efforts we're you know having these spaces for people to come and learn about our community we're having these diversity and inclusion talks so that we can start identifying future leaders all these things that we've set up we're pouring thousands and thousands of year into this but no one's showing up we can't find women or people of color anywhere they won't come and so I'll ask them a question and I love asking this question it's almost like and I already know the answer but it cracks me up to see their face but I'll say well tell me about your marketing efforts for those events that you're having and they were like oh yeah we're sending out on Twitter and LinkedIn and I was like who's Twitter and LinkedIn?  You have to expand your personal networks and I think that that's something that is often overlooked for maintenance people join communities they're almost like friends that join together I've heard so many communities start from people meeting at conferences like this or they're hanging out at a bar somewhere it's friends and family members and they start this community and then they bring in the people that they know and the people that they know and even myself included most of our personal networks are people that are similarly situated to us so if your community if your personal network looks like this and that's all you're advertising to you're never gonna reach a broad and diverse audience to bring them into your community to introduce yourself so I would say make sure that you're going to conferences not just with your friends and the ones you know that you've been going to year over year keep coming to scale because we need to scale 20, 20, 21 or whatever but there's also other conferences that you can be a part of whether it's AfroTech whether it's some for the Hispanic community go there as well and start expanding your personal network and I think that's the number one mistake that I see people trying to advance diversity and inclusion with people that look just like them are from the same backgrounds are the same abilities. Excellent, thanks. Well, I'm mindful we have got a few minutes left I wanna make sure we have there's at least one question in the queue I did do I once gonna ask you one more thing about what you thought about the future of diversity and inclusion and open source so we can do we want to talk about that one or do you wanna go ahead and jump to I've got a question I've got Gwinnivere's I think over here it was one all the way over here first the mat, yep I think she was first do we have an amazing skirt by the way I think I need that Yes Yeah, so I really love the effort of going out to colleges that have traditionally been underrepresented how do you empower those students to advocate for themselves when they get to their first job offer they will they will be paid less than their more privileged counterparts that's just the numbers Yeah, how can we change that? Yeah, I think the number one thing is just arming them with information and that's why I said as part of our curriculum we went into what does finances look like what are typical salary ranges how can you start reaching out to people to talk about what different salary ranges than those things are but what I've learned through the summer is that the students they still aren't there yet we're talking about how can they advocate for themselves and we still have a lot of these students that are dealing with a lot of imposter syndrome just to ask them hey, what have you been working on this week you would think that's one of the hardest questions that they've ever heard before and so just getting the more and more comfortable hearing their voices in front of those we noticed that when we first started all in the fall semester they were working with a lot of folks on my team and folks that look like them from underrepresented background when we put them into and they were very engaged when we put them into major league hacking and those are they're my friends but mostly white people you saw the students just clam up inside of themselves and it was amazing to watch that but I was like we have to even get them more exposed to people to realize that they're gonna have to work with people from all parts of the world and they need to be comfortable in themselves and I think that's just practice and practice and it's having people that do it and do it well it's not us doing it that's why I said we're partnering with an organization and we're trying to find one that they know how to do this and do it well to get these students prepared so it's definitely a step by step process Thank you Yes Hi I was wondering if you could talk a little bit about resource allocation so I work on a lot of open source hardware and so one of the problems we have with our projects is in order to be a contributor you need to have access to the ability to put together a circuit board you need to have access to Fusion 360 to then maybe if you have a 3D printer or something along those lines so there's like individually each one of those things to your point about speaking about privilege and the ability to contribute now it's not just a time thing you also have to have a resource thing and so I'm wondering if you could speak about how one goes about expanding an open source hardware project to be more inclusive while at the same time recognizing that we are also strapped for cash and can't be oh here's a $2,000 care package to just give you whatever you need to start Yeah I think the number one way is making people know what the work is that you're trying to it's so funny that when you said that my ears perked up I have a student that they said that after the summer they realized they don't wanna do software they wanna do hardware and I was like I don't know where to tell you to go who's working on hardware and so like so us having this conversation and knowing what everyone's doing in these pockets of open source we can then connect because I can solve for the funding like I have this one student if you tell me he needs a whatever it's not like a million dollars or something right like if there's enough in his reason we can get there there's foundations that have been reaching out to us the Ford Foundation has generously funded us we're talking to the Sloan Foundation the Linux Foundation they have been all over the place and they're like if it's money tell us what you need but we've just been trying to figure out where best to get the money and so I think it's just us having these conversations and that's where open source and diversity inclusion you know that's where it comes into play so if your community says we wanna take on five students we would need $10,000 to take on five students and we can match then that's what it looks like when we're all coming to the table and bringing what we can bring to the table I'm sure Cisco Intel all of those companies that deal with hardware they would love to you know if you all are in corporate sponsor get help sponsors to be able to send money or a way to do that especially for those from underserved communities yes entrepreneurism and the networking say of churches and existing can you maybe talk about some of the strategies or ways that you think that we can approach building those type of resources yeah so I would say it's two things one one of the things that we've learned through this pilot we talked about corporate program you know the corporate partners getting the students for internships to these corporate partners not everybody wants to go work for a major corporation they wanna work for smaller you know organizations like startups they wanna contribute directly to communities because they have entrepreneurial dreams they wanna be a founder one day and so that's why for this next iteration of all in we're looking for not just corporate partners I probably need to change the name of that we're looking for any organization that's willing to host students because people want different experiences we're looking at schools in Washington DC and they're saying hey we actually wanna stay in DC are there government agencies that do open source and so we need government agencies there so the quant that you're making about you know churches and non-profit organizations we started the pilot with students because they were readily available demographic but we are gonna expand all in for everyone that wants this type of opportunity because we realize there are retirees there are community leaders there are people that are more advanced in their community and they want to give back so we're like let's teach them open source so that they can go to the communities that have these young folks that they can actually start bringing into the communities in places that we can't reach so we are definitely expanding all in so they can be for like I said retirees for career changers people that are currently and formally incarcerated I always say open source is the lowest barrier to entry in tech and so we need to be reaching out to more and more so we definitely have that in our roadmap yes I have a question there and a question here too okay we have time for maybe two more so we'll start here I'm Sam Coleman I had a kind of a complex question what it is is that when I did my doctoral research it was an open source technology in public education about six up seven eight years ago and a lot of people in this room had a pretty significant impact on my development in open source and all these people don't look like me and I love them and they may or may not know who they are I'm just gonna say that it's important that you know we need to recognize the level of impact that we have on people I wouldn't have become Dr. Coleman unless I had a lot of people in this room help me my question to you is this through your survey have you been able to identify the entry points into open source for people of color because for me I kind of stumbled into scale in 2004 I had no idea what I was getting myself into I just I was looking around like you know who are these guys kind of cool and so I was wondering has your survey gathered that data so that we can kind of analyze what the entry points are to maybe help hasten that process yeah we didn't ask that question specifically but we have enough data and some other questions that I think we can make some assumptions about some things because there was still a high number of people that they got exposure to open source through their undergraduate curriculum but then when we can start looking and there's been some research that if you look at historically black colleges and universities and minority serving institutions they don't have open source curriculum so there's a gap there so we started seeing an entry point that more people of color came into open source later in their careers it was after they had already been you know working for sometimes over a decade before they even got exposure to it and so that's what we were looking at and all in how can we infuse it more and more into curriculum and we're looking at Microsoft Tills which teach they train people to teach computer science education in high schools and so I told them I want open source curriculum as part of your curriculum now so that we can train people if you have a local high school in your community you can go get trained through Microsoft Tills where you can go once a week or once a month to actually introduce open source curriculum so we have some data and the data is open anything that's not anonymous and so we have the data set so I would love to partner with you just to look at your research to see whether or not we can start building some type of you know research and data around that and something we can solution oh awesome well I send me the clip now everybody sends me their dissertation I love you I love you love it okay last question oh my hi my name is Solonette I was wondering so you talked about a lot of the universities and things of that nature have you looked more at like technical colleges and you know community colleges that have the associate degrees because they have a lot of programs there to help bring people up and they kind of know what they're doing in regards to that so like bringing in this to them I think could be extremely useful I used to be a director at ITT oh please so and I totally changed the curriculum to make everything up on the internet so that they have all had websites and portfolios and they graduated into jobs and so it's like open source has that same thing where you can just graduate into a job if you got the code already there correct so to answer your question yes so the pilot we did at those six schools but for this full program this year we're expanding it to any student that goes to a minority serving institution so that's for Asian and Pacific Islanders women tribal colleges Hispanic serving institutions as well as historically black colleges and institutions they can go to a community college or they can go to a predominantly white college as well if they are from and underrepresented background and tech specifically focused on women and racial and ethnic minorities in the US and we are already talking to Brazil about what does this look like to start introducing this into South America as well so community colleges are welcome to participate this year thank you yeah I think that's it oh yeah you said last question oh and I'm over time thank you all so much so thank you both for joining us and for this fantastic conversation very insightful there's looking forward to hearing more about the program as it expands absolutely so with that thank you all again for joining us for the Sunday morning keynote and thank you Demetrius for joining us as well as part of the skill family supply it is in the mail it will arrive at your home and in Raleigh in a few weeks like I said supply chains are fun but in any case thanks again for joining us we've got the expo floor open to for another couple hours still two o'clock if you haven't yet done the expo floor bingo scavenger hunt passport game thing there's still a chance to win some fun prizes we'll be here again in this room at three o'clock with Dr. Vint Cerf and yeah looking forward to seeing you all here as well as in Pasadena March 9th through the 12th 2023 perfect testing test test hello hello hello everyone in 10 minutes we have the cloud native track starting we'll have Joe Smith talking about cloud native home labs so if you want to hang out and watch sweet if not start making your way out thank you all right uh... thank you everyone for coming this is the cloud native track uh... next up we have Joe Smith he is going to be talking to us about cloud native home labs uh... I am particularly excited because I too run a cloud native home lab so I'd like to you know trade notes and see how each of us does this so take it away thank you rock and roll thank you alright thanks so much everybody for uh... for coming to the talk uh... thanks first of all thank you so much to all the volunteers and the organizers of scale uh... you know this is a volunteer conference so I just want to give uh... all those committees a plug if you had a great time uh... if you're interested in contributing and pitching in uh... I'm pretty sure every committee possible could use it hand so whether that's the tech team the dot org committee uh... pretty much if you are interested in pitching back in uh... reach out to the uh... committee chairs they'd love to talk to you about that uh... and also thanks to the sponsors that we can actually do this uh... we can be wired with coffee which is great uh... so we're gonna be talking today about uh... running a home lab and what that looks like uh... in particular we're uh... we're gonna talk about sort of the journey that I went on and kind of how I like proceeded further and further away from hardware and how I got way further up in the clouds uh... you know being disconnected from what it actually means to like run systems and thinking about the actual hardware that's actually running the workloads you're using so you know it's great obviously we have all these different abstractions and so we don't need to be thinking about nicks and hardware and all the different problems we have with that you know we can think about services we can think about the different regions that we're running uh... workloads in and how we make sure that we're distributing that so that way our customers are able to connect that very quickly so when we're building these distributed systems we're able to leverage a lot of the tools and techniques that we've built out over the years what I found at least is that I was sort of losing that passion and you know there's something different you get when you're actually having to like fiddle around with some hardware you're really thinking about how am I going to architect this at a very low level and if you're thinking in terms of thousands of servers or tens of thousands of servers you kind of lose that like you know the feeling like the pet like oh here's my little like you know my home server this is the like hard drive that has all my like you know home videos or whatever that I'm saving and so I think there's a lot of benefits here that we'll get into uh... the way we're going to sort of build this up is through different layers so the first one we're going to really think about how are we approaching the infrastructure like how are we thinking about the hardware and sort of what's that base layer you know this is primarily the stuff that AWS, GCP is going to abstract for you that you're not really going to be able to leverage anymore once you're running in the cloud so I'm advocating that if you don't have a home lab I think it's worth your time pardon anything it's going to be you know you're going to learn a lot and you're actually hopefully going to be more connected to what you're running once that's in place the talk is actually going to pick up speed a lot I think this is where we're able to use a lot of the cool cloud native technologies that the community is built out and so we're going to be able to take advantage of that and then you know kind of what's the point of this like what do you want to build on top you know what are some of the different cool self-hosted projects that you can use once you have this infrastructure that's you know in your apartment or in your house I'm sure a lot of folks are going to ask this was the first question I asked when I saw slides like this there's an open source project written in Python called look at me which you use to generate the slides so if you're at all interested in that I'd recommend you look at this and then I used graph is to ask a generator for some of the diagrams that we'll be going into so I graduated from Chapman University down in Orange County in 2010 with big Rob who runs the network in the Wi-Fi here we we had a pretty good time and you know we're building a bunch of the systems that you know are still in use today which is pretty exciting he was telling me I think the spacewalk server we built 15 years ago is still down there and so you know going from really like tinkering on all of the hardware really like using mini-com and like serial ports to like flash APs think about you know hey the DHCP server and like Smith Hall is down I've got to go like truck out and like drop a replacement in there you know really getting that like hands-on experience I think really was beneficial especially as a student and then sort of over time you know going from Google and sort of like moving up the stack into the cloud you know again we're thinking in terms of large scale but you know we're moving away from that that opportunity and so you know to great pride I think I still have you know at school I was definitely that kid who's like so obsessed with Linux well can you Linux and open source you know whenever the yearly IT team would come by and like reprovision each of the computer labs I would like follow behind them lab by lab with like 20 live CDs and I would drop in like an 8.04 like hardy hair and disk and I would like dual boot everything which was great because a lot of people were using Linux for the first time even if it was kind of inadvertently which was sort of good so so I think you know trying to find that joy again is really the point of this so from a high level what we're thinking about building you know for me one of the goals that I wanted to do like the application that was my killer app that sort of justified all of this was something called task server and it's the synchronization system there we go synchronization system for like a to-do app that's primarily run in the terminal and it's very powerful very feature rich there's a ton of complexity in there and like there's several talks on YouTube which I encourage you to check out if you're interested but basically you can create different projects and have all this running here and it's great by itself if you just want to run on one machine like one laptop like this but if you have several laptops or server if you want to run it on your mobile device you need some way to synchronize that data and that's where the task server comes into play so there's a way you can configure each of your clients and make sure that they can communicate with the server upload that data and then bring it down so you're always able to know you know what is the thing that I'm running a little behind on and I need to keep tabs on so in order to run this application the thing that I decided to use for this was Kubernetes distribution I finally decided to kick the tires on that I decided to use a distribution called K3s which is from the Rancher folks who were inquired by SUSE super super impressed with this very easy for me this wasn't necessarily like my opportunity to learn how to like do Kubernetes the hard way I just wanted to get this up and running and I was really focused on like more of the lower level again I wanted to think about you know I just want to run CUBE and you know for me the exciting part was like thinking about the lower levels how am I going to pixie boot servers stuff like that which is a great transition because the technology we're going to use for that is called MAS or metal as a service and that's by canonical in the Ubuntu community and we're going to spend a lot of time digging in there and that's really going to be the meat of this talk we're also going to talk about what types of physical machines you want to use what are some of the servers how much you think about tying up the network and how do you want to architect things in order to make sure that they're going to be able to communicate with each other and you can actually connect all these services properly and to start we're going to kind of go into the home lab and so like I was saying the you know when I first started I was really going in you know I was really excited about technology right there was just so much potential with the open source community so many people that had their own itch to scratch they were coming together collaborating and building something pretty awesome and so I think over the years you know I really was focusing on you know solving business use cases you know how do we think about the customers how do we make sure that we're really accelerating our delivery how do we sure we're doing things in a reliable and repeatable way and I think you know focusing on that obviously super critical but I think I lost that opportunity to really like jump in to the technology and kind of have fun with it as well and so about a year ago I joined a new company called Neuralink and we're building medical devices for patients with spinal cord injuries and so we're building a brain computer interface it's an implant that's connected to the motor cortex of the brain for patients with paralysis so this is somebody with like quadriplegia who is paralyzed and can actually move their limbs anymore and so by using this implant they're able to just think about moving a cursor and they're able to connect to the computer over Bluetooth and actually control a computer using their thoughts alone which is pretty wild it's like the craziest thing I've ever heard in my life and I'm still super so to work on this so we're at this really really cool place where I think we're going from like more of a research lab like just trying to prove out this technology and I think we're actually you know if you've seen the monkey mind pong YouTube video if you've seen the pig demo like we're pretty convinced this works and it's more about how do we scale out this technology how do we make sure we're bringing the infrastructure and the resources we need in order to make this happen and so from an infrastructure perspective you know being an infrastructure engineer this is really exciting because we're really the early days of this entire industry to be honest and definitely in terms of the company you know there isn't that much infrastructure there and what is there is you know very solid and I'm super stoked for Logan and all the folks that started before me but I think the ways that we need to grow the company and the challenges we're going to be facing you know really require us to make some very hard decisions and I think right now in order to make the right calls we need to have a lot of experience and we need to be able to decide you know what are the things we want to build and how to make sure we roll that out in a safe way that's going to scale with the organization so we don't need to like do a ton of migrations again and again and again and so I think you know when I think about you know when I joined the company and I started to understand you know what are the requirements we have what are the different problem domains we wanted you know we really do need to iterate we're so small we need to take advantage of the best technology that there is available and we need to contribute back to that we need to build it but I didn't want to go off and like leave like 50 half-baked systems in like a trail of destruction behind me I didn't want to like leave a bunch of things that are out there have a couple people try some stuff out you know those become critical but you know I'm really excited about something new and I'm trying to build a new system and then meanwhile I'm also trying to like keep this other system or these other 20 systems you know dragging along and so I needed the safe place to try out new technology I wanted somewhere where you know it still mattered but you know the sample size was one you know it was just myself and you know my wife and you know we were the only people who are going to be using this you know this is just providing functionality for the personal network and not something that was necessarily going to be involved in the corporate setting you know we still want to iterate very quickly when it comes to work but I wanted to have a you know sort of a prototype for the prototype before I really jumped into that thing and so in order to do that you know again I sort of thought about you know what is it that we're doing at the company and how do I start to apply that and sort of create a parallel universe at home and one of the first things was that thinking about you know how even though at work we're still going to be able to leverage the cloud we want to take advantage of as many managed services as possible we want to be able to you know hand over a lot of that operational workload to you know a bunch of the software engineers who built these battle tested systems but at the same time you know we have these physical implants you know and we're trying to we need to measure them we need to understand how they're behaving we need to track their temperature and a ton of other metadata about them so that means you know we're always going to have you know Bluetooth antennas that need to be plugged into some amount of servers we have you know an implant assembly line that you know for each of the different stations along the line we have a bunch of sensors that need to be used to validate that we're building the components to within the specifications we have and there's a ton of software that needs to run to verify that those components are behaving as expected and that needs to run on something so each station needs to have physical hardware as well so you know again as much as I'm you know able to leverage the cloud experience that we have we need to become excellent at running physical hardware and so that meant that I need to remember how the heck to do that and so to start that means building our own infrastructure as a service so what is this base layer how do we think about the infrastructure the hardware and the servers that we're going to use how do we make sure we're able to bring them up in a repeatable and reliable way and then how are we going to connect all of these servers together so they're going to be able to talk to each other and I'm actually going to be able to access the services that we're building and again I kind of touched on this but I think we're really trying to make sure that you know we have this easy opportunity to try new things you know if I'm really kind of like oh you know this was a great idea it seemed like a good idea at the time it definitely is not you know let's scrap this throw it away it's something where I only need to change some configs on my own you know I just need to reprovision something it's not a big deal there we go so when you're looking at the the opportunities for how can you provision these servers how can you create these operating systems and make them run obviously you're thinking about servers and we're going to talk on you know some of the different things you can use for that you also want to think about how can you isolate out the network and so we're going to touch on some of the switches you might want to use if you don't want to use that we'll talk about crossover cables which I'd forgotten existed so I wanted to call them out specifically and then you can also thank you yes exactly I've been in the cloud for a long time so and then also if you don't want to deal with this you know you can just get started with some VMs which is great so as I discovered yesterday when I spilled coffee all over my laptop it is not a good idea to use your daily driver for a bunch of this experimentation so I would highly recommend not having to blow apart you know whatever you're using for your day to day since you again want to have that opportunity to experiment you want to be able to try different things you want to distro hop here and there you want to try you know fedora silver blue you want to try these different os's and different approaches and see if that's something that's actually beneficial and maybe you can apply that in different areas so one of the cool things is the next unit of computing boxes that Intel launched our PAL system 76 who have a booth in the expo floor they have this cool little box they call the meerkat which is awesome so I have two of them running on my desk they're great because they're just super easy to to run a bunch of different things they don't take up that much footprint they're pretty low power so that's been something for me that's been great is just having these this hardware has really empowered and enabled a ton of different systems and a lot of experimentation which is awesome you know if you're looking to upgrade your laptop you can always keep your old one around that's great because you don't need to plug in a monitor and keyboard it's super easy you know again this is something I wish I just like thought of years ago I feel like I really would have benefited a lot from having this environment set up and so I'm just encouraging y'all to think about like you know can you set this up if you don't have this already there once you have the physical hardware's physical hardware set up you need a way for it to connect to things and so most of us probably have what's called an unmanaged switch and so this is just like a little tiny box you can just plug a cable in you can connect to it and make things you know it'll talk to whatever network it's plugged into and those machines will be able to communicate but if you're using something called a managed switch this is something that's a little bit more advanced maybe a little more expensive has more advanced software on there and it's going to the firmware will let you customize each port and you can set different things like vlands you can set which network these things are on and you can actually isolate out different services from each other so if you have like your normal desktop your normal laptop or your wi-fi router those can be connected to each other and they can all talk and do their thing and then you can create a separate isolated network which you're just using for some of this experimentation and so if you just want to play out for a player play around with something like you just want to mess around with mass especially as we get into talking about DHCP if you want to create that isolation if you're using a managed switch you'll be able to specifically siphon that traffic off and make sure it can only talk to ports which you've specifically configured and enabled and these are all different depending on the settings you're using but for me this was a great opportunity to get more involved with Fortinet we're using some more of their switches at the office so that meant I had the chance to like sign up for their support center you know figure out how to upgrade the firmware do all this and again this is something where it's like very low effort you know I'm just doing it on a weekend for fun and it's not you know when it comes time to like upgrading the firmware at you know for the core router at the office I'm like oh yeah I know how this goes you know I know where to download things I know how to follow the steps so I'm feeling a lot more confident which is pretty great if you don't want to deal with the the switch again the crossover cable something I again very embarrassed I forgot about but what's really cool about this is this is an ethernet cable where they've actually twisted the wires and so this cable is taking the input from one laptop and it's going to the output of the other machine and it's the same on the other way back normally like a straight through ethernet cable you're expected to plug it into a switch or a router and the way that that those network cars are set up is it's expected to communicate directly in that like a normal pattern where the network cards are set up with that and machine like laptops and servers you need something like that that switches it so that way you can communicate that way directly so this is great because you can just manually configure the network in your operating system and then you don't even need a switch you can just again super easy to set this up you just need a little cable it's super red so yeah and these are pretty cheap online which is great or you can make your own if you just have it and you can just switch that up and again if you don't want to deal with any of this stuff if you just want to get started any machine that you've got that's running on modern OS you can start virtualizing this you know there's a ton and ton of ton of documentation a ton of tech talks on things like this you know Ubuntu and canonical have multi pass which is awesome there's a great tutorial on Maz so I highly encourage you to check that out if you're not not familiar with that okay so we've got our setup we generally have some network things are plugged in which is great now we need to start talking about like how are you going to you know we need to power on these servers we need to bring this up something I didn't realize but like most of our machines do not have bios anymore this is blew my mind I you know this is very old information to like everybody in this room I'm sure but but everything's been replaced by a new standard called UEFI the unified extensible firmware interface and so for me you know I would say like oh I'm just going to like you know boot the bios and I'm going to connect in and I'm going to set it to network boot or something totally not a thing like none of the machines that I have actually have a bios anymore this is all like very ancient technology and there is no standard like this this is just like a de facto standard it just works the way it does because that's how like the IBM PC was created and this is just like people just sort of like YOLO tried to figure out how to like set this up the same way and one of the things that it uses is something called a master boot record which we're going to get into so your machine the actual laptop or the you know server that you have has some firmware on there and it's very minimal it basically like powers on the box and it's going to hand over control this is the the process for booting bios essentially and so this is still helpful to know like you know kind of the basics like where we're coming from so this is going to hand over control to the bios and what it's going to look for is it's going to look for like a special like set of bytes on the very beginning of your hard drive and it's just like kind of like this magic area that nobody really knows like technically how long it's supposed to be like how much space you have like how things are formatted it's all just kind of like reverse engineered essentially and everybody's sort of like winks and goes along with it and so this here the master boot record it's a way to structure how the rest of your file system is partitioned so where can you look for essentially the bootloader so in Linux we're using a system called Grub and Grub is a little bit more advanced you're able to build up a little bit more functionality here and really the once you're at this stage you're able to think about which Linux kernel version do you want to boot which operating system do you want to boot but again this is like very limited functionality still at this point and really it's mission in life is just get the kernel up and running which again as most of us know is kind of that intersection between the hardware and actually the rest of the operating system that you're booting so BIOS obviously works super well I mean there's a reason it was around for decades it's you know it made things work out pretty well but again one of the several issues including it can only access one megabyte of memory and so if you think about like you know to make fun of myself you know if you think about Slack like there's no way Slack is only taking one megabyte of memory so you know that is really not that much space and so there's a ton of other downsides to this in addition one of the big problems is BIOS is really only good at booting and operating system from disks and so if you want to do something a little bit more intelligent like for us we wanted to boot off the network that's just something we're not going to be able to do unfortunately and so huge props to Intel they obviously recognize this problem and they created the extensible firmware interface and then they realize that you know hey it's great that we've got like this thing that we're building on our own but the PC industry is this huge you know there's a ton of different companies there's firmware vendors there's motherboard manufacturers there's PC vendors there's operating system vendors all of these different components all need to interact with each other they all need to know how this system works and they need to be able to communicate with each other so we need some standard we need to have some way to agree on how the heck this machine is supposed to work like how are you supposed to get this hardware all the way up to something that you know someone like you or I could take advantage of and so that's where the UEFI standard was created and there's like an umbrella organization called the UEFI forum and this group they publish all of the standards online I think it's relatively dry reading personally but like it's all there if you want to read it tons of details tons of documentation on how to implement this how all of this stuff works we're going to dig into a little bit of this and talk about how this this happens so one of the big things obviously now we can take advantage of modern processors so now this is again why I'm saying that you know this is what most of our systems are using is UEFI standards you know you're really having a lot more capabilities here because you have because you have this opportunity to start to define things you have a process to make modifications and one of the big things that it's doing is actually defining the standard for how are you formatting the file systems on disk how are you creating that bootable partition and you're telling the firmware how are you actually going to start to hand over control to the operating system and so this standard means that you can start to build tools that take advantage of that this GPT format is a structure that everybody can start to build against and make sure that they know how to use it there's not too much in there so we won't go into too much detail about it but really the important part here is now at least we have one way for any system at whatever layer of the stack you're in to be able to to query that information and figure out what operating systems are available what are the different partitions and how can we take advantage of that so for example if you're just running you know Linux right now you can run EFI boot manager and you can actually print out and you can query that information like you can look at that partition and you can see the details about it and so here you can see for instance that you know using your operating system you can actually set the boot order for the devices that you have and so here you see the first thing is 2001 and then 000 so here first you know this is this laptop here it's you know if I plug in a USB drive that's the first thing that's going to boot and then you know there's no USB drive then it's going to boot this EFI hard drive which is great and so this is something before you'd only be able to change in the firmware and again each implementation might be different there might be different ways of doing this but again this is something that now way up in user space up in you know once you've booted the os this is something you can configure the firmware on for your device which is pretty sweet all right so we've got our firmware we have our uefi set up we sort of know that you know we have a way to configure this stuff and so using uefi that is the way that we can actually set up our network booting but in order to get there we kind of need to like make sure we run through some of these different systems that you know if you've been in the cloud for a long time you may not remember how they work you may not have experience with that so the first one we're just going to touch on very briefly is just the concept of a subnet and so if we think about a network it's a connection of it's a collection of machines or a collection of servers the intranet obviously is a you know intranetwork group of networks so you have multiple networks that all connect and communicate with each other and then a subnet is a subdivision so it's kind of for you know our use case we think of it this is our home lab so this would be the you know all the machines connected to our wi-fi that's on our home network here and so you know most folks standard standard network it's going to be 192, 168, 1.1 or 1.0 and that's going to be something called a slash 24 long story we're not going to get into cyber math or anything in this there's many folks who will give much better talks about that than i will but long story short is you know typically you're going to have about space for about 256 hosts in your typical network which is green and so you know when we're thinking about like hey maybe we want to create a separate isolated network for joes like wild experiments that's going to be something we're going to create a new subnet for and we're going to to configure that network switch to say hey anything that gets plugged in here we want it to be on this 192, 168, 200 network and so we maybe we don't set up routing rules so there's no way for machines on this special experimental network to connect back and corrupt whatever is going on so if i'm setting up something you know again we're going to talk about DHCP in a moment but maybe i'm messing around with some of the network settings you know that could totally destroy that network and if i were to do it on the wi-fi and i had friends come over they'd be like hey like why is your wi-fi broken i can't connect so again creating that isolation again gives you that opportunity experiment which is really important the next thing we're going to talk about is the distinction between unicast addressing and broadcast addressing and this is super important because we need to think about the distinction between you know once i have an ip address once i'm connected to the network if i want to you know connect to google you know i just looked up dns here this is you know google has its own very complicated thing which again is probably like months of talks to figure out how their network works but but at this point you know if i wanted to connect to google this would be the ip address i'm doing and so when i say hey i want to send a search request my computer knows that it needs to talk to this ip address and it's going to send a message which is only intended for google and so that's the only server that really cares that i'm talking to it and it knows that i'm sending this message over to that server and broadcast is very helpful when you need to send a message to everybody on your network and this is typically restricted to just the subnet that you're on but this is a way for you to sort of blast out a message just sort of like i am right now i'm sending out this whole broadcast message to all of you and you're hearing it whether or not you want to and you're you're basically know that you're potentially intended recipient for this for this information and you know knowing broadcast addressing is important because there's something called dynamic host configuration protocol and so whenever you turn on your phone or you open up your laptop and you connect to the wi-fi or you you know join your friends wi-fi the first thing your computer's going to do is it's going to say hey I don't have an IP address which means I cannot communicate with other servers I can't talk to other machines and so I need an IP address so that way I can send messages and people know how to reach me and get back to me and send me that information and so you know thank goodness all the early engineers I'm sure some of you folks who are involved in this thank you for all the hard work this is really important because this is how we establish sort of the rules of the road and so I think in the knock like two or three rooms over over that way great excellent it's over there so every time you come up on the network your client is sending request and we'll dig into what that looks like here so it follows a process called Dora so that's the first step is going to be the discovery and so that's when your laptop opens up you're going to broadcast a message and you're going to say hey you know I would I'm a new machine I have a specific like hardware address and I need an IP address if the network's been configured appropriately then so there's going to be a server there and it's going to say hey you know I'm going to set aside this IP address for you would you still like this and 99% of the time your laptop's going to say like heck yes I definitely want to I know on a stream YouTube or whatever so I will definitely take that IP address if it's still available and then hopefully the server is going to agree and say yep that sounds great I you know that's yours I'm going to mark that down for you know certain amount of amount of time and that's going to be your IP address until further notice so that's great so at least we boot it up we've connected to the network and we're going to be able to you know get an IP address and send traffic the next thing we need to think about again when we're thinking about you know how do we want to get get an operating system to actually boot is you know we need to get that data onto your hard drive so again if I'm you know taking a server from from scratch and I need to get an operating system probably I think there's a thing called Linux if I want to get that installed I need to get that data transferred over here but this process is happening you know there's no operating system right and so we need a protocol that's very very simple or trivial and so this is going to be something that hopefully is very easy to implement that's very standardized and is very clear how to do this so essentially once you know where TFTP server is you're basically able to make a request to that on port 69 and you say hey I would like to receive this file and the server is going to respond back if it's just a read request which for us is all it's going to be it's just going to respond back with a data packet and each time it sends a data packet over the UDP protocol you have to respond back and acknowledge that and say yep cool I got it give me more data I'm just going to respond back with more data you're going to write that down stay give me more and you keep going until you receive the data the way you know it's done is when you do not when the data message you receive is not the full size of the data packet when it's truncated that means you know you've reached the end which is great and if the last one is like perfectly sized so it's the full length the server will actually send you back a packet that's data packet of side zero so once you get that you know you're done which is great awesome so we've built up a way to get an IP address we built up a way to get an OS image which is important because now we're able to talk about something called pixie and so this pre-boot execution environment is a standard actually it's been around for a long time but it's finally added to UEFI and this is the way that most machines are actually going to request that network image and so you know at work or in a lot of server environments like this is the way this is the process that servers go through in order to get that network image which is great so we're going to build on top of these two protocols that we've already defined earlier the big one that we're going to do is during that Dora process during that initial discover request we're actually going to add some special configuration and we're going to say hey I actually don't just want an IP address I'm also ready to rock and roll and I want a new version of Linux so I want to get you know whatever operating system you've got for me I want to take that and I want to boot that which is pretty great so if you have a special DHCP server which is configured appropriately which you know Soiler Alert Mas is configured appropriately it's going to tell you hey you know here's your network information here's the subnet you're on here's your DNS servers but also here's some information about where you want to go to get a TFTP server to get that data you're looking for that's where you're going to go to get that network that operating system image one caveat about TFTP is there is absolutely zero security built into the image so if you want to do some like extra validation or extra check something that's 100% on you that is not part of the protocol so there's you know no security on any of this stuff so you know if you've heard of the term of like a rogue DHCP server handing out leases like you know make sure that if you were sending out a pixie request you know hopefully you were in control of that network and you know that your TFTP server is you know secured and somebody hasn't put like sketchy images on that box which is you know the whole lab is very unlikely which is great all right so we know how to boot things we know how to set you know we know the fundamentals here we know sort of the first principles about how to actually use these protocols and how to leverage this and so now we can talk about a specific implementation about how to take advantage of these things so metals of service again comes out of canonical and the boon-too community and for us you know this is how we're going to provision our servers very quickly and so we're thinking about again reducing the cost of errors you know making it really easy to you know kind of have that similar cloud like experience like hey you know I just trashed this operating system I was messing around with system D you know I don't know how I like blew up my network manager config this is really bad I'm just going to re-image this box like I've had it like I don't have time for this like this was fun now it's not let's start over so this is pretty great because it provides a lot of that abstractions that make it much easier and so you know we're going to dig into a lot of the like implementation details there's again the maths team actually is like totally revamped their docs in the last like three weeks tons and tons of details in there which is great so I highly recommend going through that or checking out some of the tutorials we're going to spend most of the time on the implementation but going through the web UI it's pretty self-explanatory if you know these fundamentals and you can apply this to the process so there's three different components we need to think about here so the first one is going to be called the region D the region controller or region D and this one you're typically going to use one per office or like one per location for us obviously in our lab we're still we're going to have one of everything obviously but this is something which I think of is like the brains of the operation this is where you're really going to be thinking about you know doing a lot of that processing this is where you're going to locate that web UI that API server this is the thing that you know as an engineer like this is what I'm going to be interacting with as a human day to day it's going to persist all that data in a post-gresql database and so that obviously you know a few you know for us you know it's a home lab it's not really that much work so you know I'm not going to worry about backups or anything like that but if you're deploying this a production environment that you'd want to care about Region D itself has nothing to do with DHCP it doesn't respond to the network requests the rack controller is what's responsible for talking to machines so the rack D server is what machines communicate with so Region D I think about is for humans and then rack D for machines so this is what's going to serve DHCP and it's also going to handle the pixie process which means it's also going to serve the tftp server as well now the important part though is that rack D is what I'm very thankful it's stateless which means that most of the time it's actually proxying the request back to the Region D server when so when our request comes in which we'll see in a moment it's actually talking to Region D which is what what knows what to respond with so rack D doesn't need too much configuration we're doing all of that in Region D which is great so here's the the typical mass flow let me set this a little bit better I think so when we do want to add a machine one of the first things we need to do is actually enlist this box and so enlistment is when you're first connecting this into mass and you're sort of first telling the mass server what the heck is going on here and mass actually has some pretty comprehensive scripts in there whose sole purpose in life is to figure out like what in the world is on this hardware like how many hard drives are there what type of network card is there how much memory is in this box and it's trying to just sort of like query anything and everything I can think about figure out all the little details like what hardware vendor if you are like dealing with enterprise hardware there's something called IPMI like lights out management which is super cool and we're not going to get into but all of that stuff is what mass is trying to figure out so it knows how to communicate and deal with this hardware so the first thing you're doing is you're actually going to enlist this box and you're going to bring it into this state where it's going to run a lot of these built in commissioning scripts and so these are sort of the default and kind of give mass this like first pass pass of just like what the heck is this hardware and how do I deal with it once you've done that it actually creates an entry in the mass database the region D server persists that and then it's going to create a cool page in the web UI and it's going to show you a lot of those details so it's going to tell you like oh you have an Intel network card you have like a Samsung hard drive you have 64 gigs of RAM whatever you have once it's in new state that's where you need to commission the machine and this is a little bit more advanced because you can actually run some custom commissioning scripts and so if you have extra hardware that you've added that maybe aren't being queried for in the default set of commissioning scripts this is what's actually going to query that and make that available in mass if you need to it also can do things like actually update NTP so update the system time and gather some extra information which is good and then at this point it's going to mark the machine is ready and all of this so far has basically been to get us to this point this box is finally like able to it's finally allocated and it's finally ready to take like be pixie booted and actually like install an operating system here so we need to all of this like you know 1975 knowledge or whatever just like install the next for the most part so the last step there is when you actually deploy the box and this is where you decide like hey I want to like install boon 20.04 or 20.20.04 I want to install this cloud in it configuration on there which we'll get into but basically here you've like got a ton of the details you know what sort of hardware there is you know what architecture it is you know what images appropriate and you're going to be able to deploy this onto the box if you needed to add these custom templates they're on the region controller and they're just located in this directory and you can actually take a look at some of the scripts and you can see you know what is it that they're doing how are they querying this information and there's a lot of cool things if you want to dig into some of the hardware and see how this is presented and how it's available so there's again a ton of information that you can find here which is pretty cool and highly encourage you to go into the maz docs because they're actually very comprehensive here if you have more questions so now we're going to really dig into commissioning and really understand you know what is this process look like and how does this work so when you first plug in your laptop or the server that you're going to be dealing with the first thing that it's going to do again is you know make that DHCP request and if you've set up in UAFI if you've set that to network boot it's going to add in those special configuration and it's going to say hey you know not only do I want that IP address but I also want to connect to a pixie server and I want to actually get that TFTP image so I can run those commissioning scripts so rackd is going to you know which is configured as your DHCP server it's going to respond to that and before it sends that response back it's actually going to query the region controller and it's going to ask it hey I need some of this boot metadata like I need to get some of the details about where's the location of the right kernel where's the TFTP let me make sure that I'm going to be the TFTP server you want me to use and it's going to query region D to get that information and all that stuff is you know the defaults for us are just going to work just fine so once rackd gets that information it's going to pass that back to the server that you're provisioning and it's going to send that pixie configuration back to the box and the box is going to do that thing where it uses TFTP to connect to that server it's going to write out that ephemeral image it's going to boot from that temporary image and then it's going to run all those commissioning scripts and get that information there it's also going to fetch something called cloud init from the region D server once the box spins up which we're going to get into a lot more detail and then it's also going to query the rack D server and tell hey like here's all this metadata that I found I want you to to talk to the region D server and I want you to persist this way down in Postgres so this is how the commissioning process works and how we get all that information we need in mass so then as we go through and say like oh like you know which server do I want to deal with you have all of these details here and you're able to figure out you know oh is this the right box that I want to re deploy or re image you know you have all that information there available once it's done with that process it's like publish that metadata the box is actually going to shut itself down so there's this is again just an ephemeral image it's just like a temporary thing that just gets you this information you're not really installing the operating system when you're commissioning that's the job of the deployment phase and so once you deploy you're using the same you know again assuming you're using Ubuntu you're using the same installer you would on a live CD and that's a tool called curtain and curtain is short for the curtain installer it's like whole mission in life is just install stuff really quickly and on first boot it's going to execute a program called cloud in it which we're going to talk about in just a bit but to really dig into curtain the whole mission here is it's supposed to be able to handle a few different configurations and make sure no matter what you're going to what setup you have you're going to be able to write a new operating system image onto a hard drive here and so it's basically it's just like a really like robust like DD command essentially so the first thing that's going to do is again you've kicked off that boot and so that's whether you've like plugged in a USB thumb drive you've got a CD ROM or you've done the network boot and you're basically bringing up this like live image which is going to run the curtain installer and so the first the next stage is going to do is run what are called early commands and this is basically loading in some modules for some configuration you're going to set up some of the environment to make sure that the hardware is primed and then if you want to you can actually update your apt cache so that way make sure you have all the latest booted packages when you're installing so you're not installing some like super out of date version once you've done that the next step is actually partition the hard drive so again we actually want to persist this we want to write an OS image and so we really need you know we want to create the partitioning scheme correctly hopefully we're going to use like you know we're going to make the EFI boot partition hopefully we're going to use the GPT format again because we're using UEFI because BIOS is deprecated ha ha and we're also going to create the template for the file system tab or the FS tab and so this is where we're not actually writing F-stab we're actually just going to write out a template which will be written out a little bit later the next step is then going to be doing a little bit of network setup so this is actually going to again write out a template to configure your network and the reason we're writing the template is because when we write the disk image we want to add that configuration after that's written out so here typically you're going to get a tar ball with the Ubuntu OS this is what's going to get extracted onto the disk that you just partitioned and you're going to write all of that information out onto the hard drive the last thing I actually for this talk I discovered I didn't even realize this was a thing but you can actually add a little like webhook at the end of the installation process so if you have like a Slack team or like Discord or whatever you can actually like configure it to like post in a channel and say like hey like you know I've been finished installing so this box is ready to go so I thought this was pretty cool it's an extra little hook that you can configure and add that in there which is awesome sorry that was too early so this is extra configuration which you can set up maybe install the webhook script and then here's where you can actually execute that script when you put that in there all right so again we're like building up these layers right so we have an operating system installed pretty much we're mostly like a stock Ubuntu image at this point and so we really need to like you know we want to install like the stuff that we want right like we don't want to just like have you know it's great that we can install an ISO but like you know we could just plug in a thumb drive and it's not really saving us that much time what I think really gets interesting is using a tool called cloud in it and if you're running on any of the cloud providers they all support this tool Amazon you might be familiar with user data it's it's really use it like it's using cloud in it to run this stuff and it's the same format which is great so this program is what runs on the first boot so after you've created the image it's the first time that you're actually going to bring up the system it knows that it needs to execute this code and execute the configuration that you've given it and so an example would be this is you know very basic but you're basically what this is the cloud native part of the talk because it's YAML but it's basically the first thing you're going to do is the package update running app to update getting the latest packages in the repository and then you can upgrade them and then you're also going to create a user ID there's a ton of like additional features that you can use this is awesome because you can actually connect to github and then pull down your ssh keys it also has launched pad support for anybody who still uses that so this is great because like you know again even if we just did this this means that we can just provision a new image we can provision a new server and you know we always know that we're we're able to start from scratch and that's awesome but unfortunately you know sometimes you're just like I want to try a couple different instances or I want to run this application you know maybe I don't want to deal with you know all the common problems we think about when it comes to isolating different application versions or different libraries so we need probably another platform and as one of the quotes I use all the time is obviously the solution is that another layer of indirection or abstraction which means I finally need to learn kube which I try to avoid through most of my time at Slack much of the chagrand and my colleagues so here's where we really get to move very quickly and you know there's a ton of documentation for running k3s so we don't really need too much you know like if you wanted to just get a Kubernetes instance running on your server you can just add this to your cloud in it and it's going to be there it's wild so huge props to the Rancher folks huge props to Patrick at the Sousa booth yesterday for giving me the tools to try and save my laptop yesterday so props to all these folks but yeah k3s is super solid again it wasn't something that I wanted to like go too deep into so you know there's a ton of opportunity here to like really play around with you know what are the best kube distros you want to use how you want to run this and how are you going to configure this appropriately but if you just want something super easy this is going to be great so we pretty much like made it most of the way through we've got you know a platform and we've got kube right and so now we have a way to get servers on our desk running kube if you wanted to create a second kube second cloud in it file if you wanted to have multiple servers that are all connected to the same kube cluster you can actually set some environment variables here and you can pass it the name of your initial kubernetes server that you provisioned so the first one you spin up probably looks like this the second server you spin up you're going to pass in a few details like the the ip address for the kubernetes server as well as like a little pass code basically like a little certificate that you're going to pass over so it can authenticate and join the cluster so again doing this and having this means you can start to like provision you know a ton of like raspberry pies or a bunch of other servers you have lying around and you can really add them together pretty easily and if again any of them have problems you want to add a new one or replace one again using cloud in it you can do this super super quickly which is really cool so again I was using macOS for a long time which you know was fine but there's a great to do app called omni focus but again I was thinking about you know for me the killer app was you know running this command line to do app which is very appropriate for scale I think but there's also a bunch of other applications like home assistant which is a really awesome Python project has a ton of community support behind it you can think of it kind of like an open source home kit or like an open source Alexa and it's really great because it actually works across a lot of these ecosystems and so for folks who use different components from like you know the Google Google home ecosystem or Alexa you can use something like home assistance to bridge that gap and I believe the plugin is called home bridge actually to tie home assistant together with home kit which is awesome and again you're running all of this locally so you don't need to worry about you know extra network latency you don't need to worry about you know who might giving access to all this data this is all code that's actually you're running on your desk at this point which is super cool you can also run like the Pleximedia server if you have again a bunch of home videos or something that you want to show off and then even there's like cool like self-hosted apps like I think it's tenduri recipes like if you want to have like a recipes server that you're running but again this is really giving you a lot of control you know you can run different apps you control the data you don't have to worry about where's it being stored what happens if like you know Google Reader gets shut down things like that so you know I think the usual benefits we all know about open source start to to come into play here which is pretty great so I think again we as we're thinking about apps though we get kind of the benefit of kind of diverging from sort of what I would consider like SRE best practices you know I am very hesitant to kind of diverge from sort of what the like the best practice of the upstream project is and so the task server is expected to be installed like directly on the OS and it's not necessarily like there isn't like a ton of folks that are necessarily running this in a containerized environment but since it's just me you know if this thing goes down or if I need to troubleshoot something like I only have myself to blame and it's something where I can fix on my own schedule and that's fine so here it's totally something that's that's great but fortunately there's Yehengi I don't even know how to pronounce this person's name but this person on the internet decided to dockerize the task server and actually create some kube config which was awesome I made a couple tweaks we'll dig into but huge props to this person for like blazing the trail and getting much of the setup so if you want to look at the docker file it's up here on github I kind of broke down the docker file here but again this is all the stuff that we're used to doing right but it's just that we're able to run this on our desk which is pretty great so we're just going to check out the repository of the code we're using CMake to configure the build and then we're just going to build the actual application we're going to expose the ports so we're going to tell docker that hey I'm going to be running something that's going to be listening on the network on these ports so make sure this is available so I can actually connect this application and then we're setting the entry point which is what the actual application that's going to run there that we've built one interesting thing about the task server in particular and the reason that I'm actually like comfortable running this is that it does actually use TLS for off and so you do need to generate certificates there's some decent documentation in the upstream project I wrote a little little like helper which is basically the more rise version basically but basically here if you generate some client certificates and the server side certificate so long as you're transferring the client certs to your devices securely you'll be the only person that's able to access this data which obviously is important so so you'll need these and we're going to use this as we deploy this to the kube cluster so again so I have a little fork we're going to just dig into the actual script that's running but basically here this is just you know again if we're used to kube this is how you run all these instances and how you deploy this so first we're creating the namespace to make sure that we're isolating the pods and making sure that all these resources are within their own area within the kubernetes cluster we're creating a persistent volume claim which again is you know k3's by default gives us as like local path provisioner which is just basically just like writing files directly to disc which for us is perfect right like we don't need some like web scale solution that like persists to like 17 regions around the globe like we just you know I have a cute little mirror cat on my desk right and that's what's going to take the data so that's fine so we're setting that up we're setting configuration and then just for my own sanity because I kept forgetting as I was like iterating on this a few times I just added this little helper to remind me to set up the dang tls certs and then we're actually going to upload all of those certificates once we've generated them which is great so we've done it you know all of this work you know going back to like 1975 just that way Joe can actually run his like to do to do app in his terminal but you know we've created this thing it was you know may not be what you want to use but it's what I wanted to use so that's the beauty of the home lab so that way we can actually take advantage of you know using this and creating this fun environment you know I finally got to use Kubernetes I have to be cranky at it for you know five or six years you know we got to have a lot of experience with mass we're able to provision these servers and make sure we understand you know from first principles like how are we going to build this out what are the protocols that are necessary and again now that I'm going back to work you know come Monday I'll be able to do a great job of building this infrastructure and I have a lot more context and I have a lot more opinions about what I like and what I don't like because I've gone through this process you know we I finally had to deal with physical machines thank god it did because yesterday I had to do surgery on my laptop and again we've we've kind of created our home lab and now we're hopefully you're excited to build your own as well so thanks so much for the talk we're hiring at Neuralink comes check us out in the expo floor and thanks again everybody for the time does anybody have any questions we got like ten minutes for questions going once oh sweet I may have missed it but where did you install mas on that is a great question I actually just you snap to install mas on the box itself just so you do need to have like that is a I did not add that that is so wild so you do need to have a separate server that I provisioned so that's why I was saying like you have you know multiple machines for that so for that I just dedicated something to run mas and so that's the thing that's serving DHCP and so that's where I didn't want to get too wild there is I think the airship I think does have a mas running in Kubernetes but I think that's just running on 2.8 and now I think we're on 3.2 just got released so exactly correct Linux box running mas is the TLDR good question about the hardware setup I'm trying to set up my home map as well awesome but I'm thinking about like server ITX motherboard and use ITMI for accessing to it what's your recommendation for hardware setup hardware setup for using for using IPMI or just regular setup like the whole home lab like motherboard CPU like oh interesting I mean I think lik