Okay, yeah, we can always switch to that so did anybody come farther than Washington for the conference this year? yesterday we had some people from Spokane and Seattle Any anybody everybody's local past California? where you from? Rancho Cucamong, well, that's a bit of a haul orange Torrance Torrance, okay. Yeah, that's a little bit of a haul too. Oh Thank you for coming Okay What kind of um, are you are you work? I mean tech a tech company in that area? Yeah, I'm always surprised at how much when I drive out that way going either to Joshua tree or whatever I'm always just surprised just how much more there is of development Yeah shopping. Yeah Yep, you have the butterflies right now to everybody noticing the butterflies The painted lady butterflies because of all the rain They've had a real It's a massive amount of butterflies and you'll see them kind of fluttering through like a little train Okay, thank you for joining the mentoring track. We're gonna go ahead and get started with our speaker Hello, I Understand now why they were saying that this was interesting Okay Folks can hear me right So welcome folks, thanks for coming out to the mentoring track, sorry couple minutes late I Believe I'm a technology guy, but I can never get the hardware stuff to work. So I'm really thankful for The folks here at scale. My name is Karthik. I'm from Austin and today I'm gonna talk about some of the stuff that we do in Austin with respect to the tech community over there Draw some parallels and see we know what are some of the best practices that we've learned over a length of time So I actually work at Oracle when I say that I'm actually not a database person Sorry, I do know how to like do one normal form and two normal form But I find that the questions I get asked are very complicated. So if you do have DB questions I could find somebody inside Oracle to help you answer that if that's relevant I do more cognitive stuff. We actually my team got acquired into Oracle. So I know a lot of stuff about containers Containers and all of that the other stuff and I used to be a developer on the managed Kubernetes team Before Oracle, I kind of like did a whole bunch of stuff Most of these companies are based in Austin or you know, they were like remote kind of gigs But in general like I like building You know stuff with friends of mine the Austin community is pretty big So we can make connections pretty easily I'm a maintainer on Gauntlet, which is like an open-source security scanner And I also help run DevOps days often Container days and Cloud Austin, which is a little user group that we have over there Do a bunch of other stuff, but you know kind of all the same Quick shout-out to my corporate overlords. I like to say So Oracle is doing a big thing with like cloud and whatnot and there's a lot of confusion in the enterprise space about cloud native and what that is So we have like a little site cloud native to oracle.com has the solutions blog posts and whatnot Feel free to check it out. I made a quick link bit.ly.com. I heard cloud I heart cloud So check that out if you want to So we're here to talk about community or I hope it's not just me talking about community I actually have I tend to like do a lot of slides, but only have like 40 or something slides I'm gonna try and finish early and so community in my opinion is like a very collaborative thing and so it's not like One person doesn't make a community. It's kind of like, you know a group effort and People who are like like-minded kind of come together and like talk about stuff And that's how you built, you know, that's how you kind of like build it build it then kind of grow it So from like a high level The Austin Tech community today. It's like very very vibrant we have like You know a bunch of conferences that come that are located in Austin, but also come to Austin serverless days Longhorn PHP conference That happens in May DevOps days, which also happens in May Day-to-day that happens like early in January last con which is security conference happens in October B-side just happens in a couple weeks Keep Austin Agile which also happens somehow like May is a really popular month for Austin conferences Because I don't think we believe in the rest of the year, but for whatever reason will I do everything in May? So yeah, keep Austin Agile happens in May as well.  There's also a bunch of meet-up groups I was trying to get like a number of meet-up groups But like it's almost impossible because there's like six or seven meet-ups that happen every day From Monday through Friday, and so literally any day of the week you'll have like some kind of meet-up going on Austin Tech events is like a pretty cool thing that it's one of the like side animations that I like to talk about I didn't do it.  I'm not sure If LA is a part of those but you can ask them if you'll want to do one for LA too It's just like a bunch of four or five people So I actually run cloud Austin, which is a meet-up group It's kind of an older meet-up group now This looks really cool. In fact, our logo is brand new like this is something that I made up Think like last month or a month before It's just using Stock photos shouldn't touch this because it's volatile stock photos and some fancy paint work But the meet-up group itself It's been rebranded once before it used to be Austin cloudy as a group. We call it cloud Austin today What was it? It's like the Facebook movie, right? It's called the Facebook before and they're like, no Just called it Facebook. So we're like made a four letter thing into just a simple two letter thing It's over 10 years old at this point in time But and it has like 2,600 people that not everybody comes to it every month But we have like a steady stream of like at least like a hundred people that come out to each each user group We also kind of do like two talks in a specific meet-up So like if the first talk is kind of boring you can be like well the second talk was interesting so like you know it's We have like multiple things that kind of go on and we kind of mess around with the format talk about that a little bit later on but Today, you know often Has kind of grown a lot over 10 years Personally, I've stayed in Austin. I think it's like 13 years right now We call it like Silicon Hills Which is like cool for us and then I was reading this article the other day And so there's like a hundred and thirty eight thousand tech jobs in town And I think Austin only has like eight or nine hundred thousand people. So That's like, you know one ninth for actually more than that With number of jobs for for technology folks 31 billion dollars go into Austin's or the area is over economy overall economy and 35% of that kind of comes from technology gigs, which is really cool Apple is also we lost the fight to have Amazon HQ to in Austin kind of like one of the towns that they were looking at But Apple decided to expand in there and also the Army Futures Command is built HQ in Austin, which is all kind of cool But like things were when I started my career and often was like very very different 15 years ago You know these were like primary employers in Austin so like with people that kind of Lived in Austin and like, you know, you'd meet like friends like friends of them to whatever you'd be like Hey, what do you work and be like it was either a Delaware IBM and if it wasn't Delaware IBM It would be you know, Nationalism in Samsung EMD There were some folks that work, you know for So the local government kind of runs out of Austin.  So some folks work there and Bleeding edge folks back in the day used to work either at bizarre voice like Rackspace or indeed And there are a couple of other companies But like a lot of times you'd be like, you know, when somebody used to leave an eye We used to joke about it where we say hey your transition path is like you start as a programmer analyst You go one two three and then like you end up at bizarre voice or indeed or something like that So it was kind of like a little joke that we had in the community and From like a user group setting like There were a few there were few and far between in terms of user groups There were like some stuff for the language developers So if you were a Java developer, there was a Java user group and I tried to trace this back I did some like history stuff The other day so I tried to trace it back and it actually started in like 2001. I see I saw references to that dot-man and php also kind of been around for a long time in terms of use groups in Austin and then oh, which is the I forget what the oh was Full form is but it's a security group That started in 2000. Well, it's been around but the Austin chapter started in 2006 and you know the ecosystem in Austin is really interesting because like the different groups so like developers kind of talk to Operations that talk to security. So it's kind of like not three different siloed kind of groups They're all kind of like blend together in some ways, which is really nice And you know we we can kind of like work together in a lot of ways to solve problems Also user groups used to be a thing inside of companies So this is something that you might see in large companies where it's like hey, you know every every month You might have like a speaker from the company kind of present, you know technical things, you know inside of the company So like we had Java technical exchanges. We like to call it used to happen every month, but It would be you know for You know engineers or program analysts inside the National Instruments And we talk about like different like Java things that we kind of playing with or whatever Much like you we do today at a meet-up by just like an internal kind of setting so oh Yeah, and apologies for all of the so all of my slides have cats on them and I did that because I'm like an enterprise guy now So like I can't have like cool slides So I was like, you know what I'm gonna put cats on every slide, but they're mostly just like background kind of things But you know every That I think that thing that really changed was about ten years ago Everyone started doing a whole bunch of stuff with the with AWS and they were trying to figure out well actually not ten years Yeah, like a little over ten years ago They're trying to figure out what the heck this was Whether this is actually gonna be a thing or whether you know it was gonna go anywhere or whatever and so like we found ourselves in a same kind of team where it's like Our task was to build products that we could host on Amazon both services So we're like, okay. Well, you know, let's go and figure out what You know EC2 is let's go figure out what S3 is blah blah blah working with those APIs, etc and Back in the day like we met some folks at a different company called pervasive so We had a team of national instruments And we met these other guys at a conference from pervasive which is also like another company in town and we're like Oh, you're doing AWS stuff. Hey, we're doing AWS stuff, too And then back in the day it was like AWS is really simple because it was like S3 You know EC2 and a couple of other things like before RDS before all like all of that stuff So it was really simple and so we had like questions on S3 API So we're like chatting with them and whatnot and so or in training emails So we were like, okay. Well, you know, we were like kind of helping each other out. So we like formalized it and instead of like doing stuff over email, we were like, well, let's meet for lunch and So this is actually like a picture of the cafe that we kind of met at And you know, we were like, let's go to lunch and it was still kind of awkward because like not everybody on Their side and you everybody on our side and we're all we're all engineers, right? so it's like Sometimes it's kind of like weird to meet new people and like talk to them and you don't you don't want to feel judged Like you don't want to feel like hey, I don't have any questions or I don't have any answers I just have questions and sometimes it's like a little bit awkward But so it was like it is kind of awkward because we were sitting there around round table I think there's a couple more folks that I don't like that were at that first like at first lunch I just don't remember who they were so I like I didn't add pictures But there are two more people But we were like talking about stuff and like all of us just had a bunch of questions We didn't actually have answers and we're like, well, you know, they do you know how this API works blah blah blah And we're like, I don't know and so we're like, you know talking about stuff a little bit more And then we're like, well, I wonder if there's actually somebody with answers To a bunch of these things. So we're like, maybe we should like start like a meet-up group And you know, we because like you're from a different company. We're from a different company Maybe there's other people kind of doing the same thing. So we're like, okay Well, let's start like a meet-up group.  So like this guy Bill He was I think the director of marketing at pervasive and pervasive actually had pretty big Pretty big space that we could use so they were like, hey, you know, we can just do like a meet-up here It's okay if it's you know, it's kind of like the room like this But they were like, yeah, it's okay if there's like 10 people in there. We're just talking, you know technology So it doesn't matter So bill is actually instrumental in like Getting us through the workflow of like how to get room and whatnot Tristan He still lives in Austin Works, I don't know where he works anymore, but he used to work at buzzer voice And he had a lot of like technical knowledge. So back in the day people would like asked Tristan a bunch of questions A lot of folks already know Kote and Michael Jordan Kote works at pervasive art. I'm sorry. He works at Pivotal right now But back in the day, he used to work at Dell and then James Wicked He does a bunch of security stuff in the day But he was kind of like a big champion of on the security side in Austin He actually helped start the last con conference and I was also like did a whole bunch of stuff for OAS Boston So it's kind of interesting to have, you know, his perspective to help build a meet-up group And so we're like sitting in this in this room kind of talking about, you know technology And we're like, hey, this the first meet-up actually went really well. I Think Michael Jordan says Yeah, he says there were like 60 people that showed up which is kind of big because we were expecting maybe 15 And so when 60 folks showed up we're like, whoa, so a lot of people are doing cloud stuff I guess maybe this is gonna take off So it's kind of like the consensus back in the day And so our journey kind of started there and then you know from a tech standpoint We're using like Google groups because Google groups is like the coolest thing back in the day We use that we used Eventbrite to kind of like help people sign up and find out more info Nobody use meet-up Because meet-up was more for like use meet-up to find like tennis buddies or use meet-up to do Sports kind of stuff like you never used it for like technology and it's like completely different today But we have this like one transition from like, you know, this technology stuff Where we did Google groups Eventbrite and then we're like, oh, we should do meet-up as well And then we're like, oh, maybe we don't need Eventbrite. We'll just use meet-up And then finally we're just like everything goes in meet-up So initially it used to be like random whenever we had topics and then we kind of formalized it into monthly meet-ups And we also one of the cool things we do is like a festive thing at the end of the year where we do like a 12 Clouds of Christmas so they're like five minute lightning talks that people come and present And every year it's kind of grown. So at the at the start I think that 12 talks and now we do like 20 talks or something But it's really cool because like the whole room is packed and people just kind of show demos that a five minutes long So like the barrier to entry is not really that large But you know, like I'm talking about all the positive stuff, right? And you know, there are things were like not always like super happy and You know, things kind of fell apart every now and then one thing that happened a lot is like people will be like I will sponsor food for your meet-up and food sponsors are Super important because when you're trying to get something off of off of the ground Sometimes people don't care about what you're talking about. They're like more in it for the food or more in it for the swag So when like a food sponsor like yes, I will help you bring pizza And then they're like, ah, you know, yeah, we can't do it And they tell you like a day before or like the day off it pisses off people quite a lot They're like, man, I came here you promised me pizza and I'm here and like I don't see anything And now I'm like now I'm cranking and I'm hungry and I don't care about what we're gonna do. So I'm gonna leave So that that happened quite a bit early on But you know, it's gotten better over time. But you know, that was kind of one of the issues And one thing one incident that we learned a lot from was this Jeff bar road trip So everyone knows most folks know who Jeff bar is he works at AWS You know, he kind of like does all the product announcements and stuff like that Back when it was just, you know, it was just kind of him doing the product announcements In 2013, he was like, he was like, I'm gonna do this crazy trip where I'm gonna start like way in the east and go all the way West all the way west to Seattle I'm gonna do them in a car and stop on like different cities along the way So he was like, I'm gonna drive through and so he was like, let me know if you want you want me to come by So like I sent him an email and I was like, hey Jeff, you want to come speak at this user group in Austin? And we'd love to have you so he was like, sure.  And so we're like, okay, we'll do this thing. It'll be awesome. It'll be really cool content great And Jeff is a really cool guy but the thing we we realized was you know There was a lot of demand from people wanting to come to it so we're like, okay well our we can only accommodate like a hundred people in the room and You know, we'll set the RCP still hundred so like pretty early on we were like, man, we're gonna blow this number And so we talked to the facilities that pervasive and we're like, okay Well, like what can we do and they're like well the fire marshal code is 120 so we can add 20 more seats So we'll add 20 more, you know seats to the room, but it's gonna be like really packed And it's gonna be, you know pretty pretty tough to have like everybody in we're like well Still he's like a really big names to be there. So let's increase that number. And so we did that And sure enough like within the day of those like 20 seats are gone Because 20 more people are repeat and we're like, okay Well, let's start a wait list and we had a wait list of like 40 40 something people and Then we're like well our hands are tied from an organizer perspective because like we can't break fire marshal rules But at the same time we're like well, we don't want this is like one of the big things that's happening in community So we want people to be able to attend But we were like well, you know, we tried our best So we were like, okay, you're not gonna come, you know on RCP so somebody can take your spot blah blah blah But in the end what happened is like a lot of people were really upset They were like, well really want to go to this thing We have something cool going on in town and you're saying you can't we can't come to it And what happened what was even worse was That day or that evening we had like a weird weather thing. So like it started to rain I don't know what it's like in LA, but in Austin if people if it rains like nobody nobody goes anywhere So it rained and sure enough like I think totally we had like 75 people show up So we have like, you know, it's like I'm it's still pretty packed But there's like open seats everywhere So then people got even more pissed off because they were like I RSVP to this I would have come to this thing, but you said I couldn't come and you know, it's not not even a full room so it's kind of like some of the flat that we got but In we kind of like as an or group we kind of talked about it for a while We're like well, you know one one thing maybe we could have done better is made it more like Made the process of like the stuff that we were doing behind the scenes a little bit more transparent So we ended up sending like a letter to or not a letter but like in our meet-up thing We sent up sent out an announcement to everyone kind of like talking through apologizing for You know what happened, but at the same time kind of talking through some of the stuff that we were trying to do from a silly facility standpoint and also like how to kind of smooth stuff over a little bit more But it was kind of a train wreck, but that was like one of the things of all the things that's happened in like the 10 years This actually taught me a lot from a visibility standpoint There's like people are more forgiving if you kind of like talk them through the problems that you are running through versus just kind of keeping Things very, you know saying saying no, but not really giving reasoning behind Why you're doing what you're doing so I was kind of eye-opening and that's something that I kind of take today Even with like Delos is Austin and like other conferences that I help organize So today, you know, like we've grown quite a bit. We have like 2600 folks where we always kind of do With third Tuesday of the month we always do our stuff at Rackspace and You know, we'll talk about some of the reasons about why we do what we do and why it's like Oyster Tuesday always that assistive venue and we we try and get food and drink sponsors whenever we can From an org perspective we've kind of grown so it used to be I think like 304 folks, you know, a couple folks in pervasive a couple folks that help kind of like, you know, get speakers and Do like all the meetup management and whatnot, but we've grown to an org of Like eight or nine people where we have some people doing venue stuff some people doing You know trying to get speakers some people doing meetup management some people talking to You know a lot of sponsors end up sending us an email via meetup or whatever So, you know managing the communication from that front so it really helps when you have, you know A lot of folks kind of involved But one of the issues when you have a lot of people is you also end up You know trying to do the same thing over two people are trying to do the same thing Which is normal like large teams, you know, there's replication or duplication of effort ends up being a little bit confusing so We we used to do everything via email, but now we've kind of transitioned to do a lot more over Slack And it's a lot more efficient because we can say hey we you know Here's what we're doing like here's our ideas and everyone can kind of chime in From an organized organizer perspective well to do a little bit of like the Zapier integration with Slack And we have Instead of you know people emailing us because we're this is kind of like not our we're not paid to be organizers It's just kind of like a side thing that we do for the community So we're like well, you know It takes a while to like respond respond to emails and whatnot So we have like a Google form for if you want to speak at cloud Austin There's a form that you can go for a lot and you know, let us know Sponsoring is kind of the same way now and we do this integration with Zapier into Slack because we're all about Slack So this is kind of like an example of that words like the bot tells us Hey, somebody wants to do and talk on whatever and they want to do it for this month or somebody wants to do a sponsor for this month, etc. I Like high or I got rid of some of the sensitive information there And yeah, so we kind of do that So wanted to talk about like some of the things that I've learned over time because I think that's the most important part From a user perspective right there a lot of times the reason people come To a talk or come to a user group is because they are really interested in what somebody has to say from You know presentation standpoint demo standpoint, whatever so like if you do a presentation on Kubernetes today There's like a lot of people that were like hey, I really want to understand like the nuts and bolts to benedicts. I'll go to this thing so If you're an organizer, you know, you have to realize that content is like the number one most important thing for why people want to come to a specific event and This is like it's true not just for tech, but you know really anything out there If you have like a You know really good speaker really good really good event people will be like I really want to go to that thing because I'm interested in it On the flip side You know we in the dev specifically like devop security space There's a lot of vendors out there and vendors really want to talk about the product And I think that's a good thing But it shouldn't be like the meat of your the talk because like People don't want to sit through an hour presentation of somebody giving like a pitch deck or like here's how my product works or whatever They want to hear more like generic information. So one thing we tell One thing we kind of help our speakers with is like hey We don't want a sponsor pitch or like we don't want a vendor pitch So we don't want to see like how your product works But instead talk about you know some of the technology in your product instead Talk about engineering problems that you have talk about, you know, how you might have issues like You might have had issues like taking your product from like one cloud to another or whatever So like always kind of send out engineers to speak about What you're talking about because our audience is specifically more like engineering folks So they're like developers devop security, etc so we always say no to that but you know from One thing that sponsors kind of like is we help them we let them sponsor stuff. So audience attendees love free food So like it's you know, especially when you're kind of starting off And let's say you're building like a meet-up group today It's great to have you know pizza or something like that because people people will come for the food Especially in communities that you know what that don't have a lot going on and there's not many choices I was like well, I'll like go for the food and might listen to something interesting to and then later on there Be like I will come for the content. I don't really care about the food that thing switches But one of the things that from you know transition from the last slide is You can kind of say hey, you can do a vendor pitch But if you sponsor you can do like a five-minute demo or something like that And so that kind of helps with you know a getting getting sponsors in Satisfying their need to like demo something but at the same time also, you know providing food for the audience, which is great The other thing I've learned is like just transparency so just be transparent of your process of like the stuff that you're doing behind the scenes because You know if you're not then people kind of like question what what you're trying to do or things get a little bit confusing and then you have to You know kind of step back and do things differently, etc But it's always more convenient if you're just like we're doing this thing And meet-up is really good about you know having the comment section So one thing we have a lot of issues with recently is and it's kind of the same issue that I talked about in the Jeff bar thing But you know we always end up hitting our attendee cap So we're always like hey if you're not gonna come to the meet-up You know on RSVP, so I end up like actually sending that message Almost every meet-up it feels like now because we always end up hitting that cap But on the flip side of that is a lot of people we also have like a meet-up If a hundred people say that they're gonna RSVP to come You know we'll only have like 75 75 of those we have like a 25% drop-off There are other meet-ups where you know you have like 50% drop-off or whatever It just depends on on the group and how how core you're like How core your members are basically? This is really important it's something that we sort of forget from time to time but have a coat of conduct and I say this because You know a it helps From somebody new wanting to come in wanting to be a part of your community They kind of look for that because they want to know if like that's the space that you are providing as an organizer is a safe space You know whether they can come and like You know actually be able to Participate and not feel threatened on the flip side of this is you know, I've seen like weird things where You know you might have like happy hour or something like that and somebody like drinks way too much and get Gets really belligerent and then you're like well if at that point in time you're thinking what should we do? And what's our code of conduct? It's already kind of late. So you you rather You know things through some of those things up front have a code of conduct for or like a playbook of things that you might want to Be doing, you know in those times of emergency Versus, you know thinking about it on the spot. So Something to just kind of think through also How many folks are like introverts in the audience? So a lot right and and I am too like even though I do a lot of public speaking Actually kind of like after a while and like man I I would like to just like sit in my house and Talk to my wife has three cats. We have two dogs.  I'm like, I don't want to talk to people but one of the hardest things that for me As an individual like going to meet up there's like I hate going alone And I was trying to think through some of the stuff like so, you know, it's not just like tech stuff Right, so if you go like let's say like you two is coming to town, right? They're like, man, I really want to go to this concert, but I don't want to go alone I'm like I would rather go with like a friend of mine like family member or something like that So most of the time you're like, you know, you're trying to go with somebody instead of like going alone so we have this thing like we we call it the buddy system because Tech, you know naming is hard But you know, I'm always like, hey, I found this really interesting thing that I want to go to And I have like a close circle of friends.   That was horrible But I'm glad I was there to like share it with you The other thing I really like is just gratefulness for organizers also volunteers Because a lot of things that we do in Austin specifically, they're all like community-based. So it's not like we Nobody gets paid to do all the stuff or just like we're doing it because we believe in like the general community in town So just like gratefulness for like for folks organizing stuff and whatnot Because organizing is painful because you have to get a bunch of stuff in place. What not make sure logistics are taken care of and You know, a lot of times people You know, I take time out of work to do it's like some people have families But they're like well, I'll go to this meetup thing because I organize it and you know Takes time away from whatever else you might be doing at that time So if you're like I practice this a lot But if you're an attendee and you go to a conference and there's a lot of volunteers or whatever, you know always kind of thank them for The effort because they could be they could be somewhere else doing whatever else but they're there to like, you know Help you out in some ways. So You know, that's just that's just the thing that I do a lot and then For any of the folks here, like are you all organized this for meetups already? Or like quite a few, okay? So some tips that you know, I've kind of learned over time is Fix venue fix date and time because what happens what happens a lot is If you have you're trying to build a quorum of people Want to come out for something on a regular basis if you're trying to build, you know a set of like a core team basically It's easier for somebody to be like, oh, yeah, you know, this meetup always happens at this location And on on this on like this specific day, but so they're not like Where do I go? Like is it in the morning or is it or is it in the afternoon or is it in the evening? And then which location is it at so like there's no like Flow chart that their mind has to go through to figure out where it is It's just always the same thing.  Oh, yeah, you know six o'clock It's that that's that rack space has a thing for cloud Austin. So it really helps with you know getting a gathering momentum from from a tindi perspective and then your attendees actually you really help out too because they're like Oh, yeah, you should go to this cloud Austin thing and they remember that it's like third Tuesday Third Tuesday at rack space or you know, wherever whatever that venue is Also understanding what folks are interested in it's super important Because your attendees is your audience, right? So you kind of need to know what they're what they're wanting to learn Or what they're trying to get out of a meetup. So one question that we always ask is We also ask it in our in our form when new people sign up for the meetup Like what what topics you want to learn but also at the start of the meetup We always kind of do a show of hands of like how many people are new And you know kind of say hey, well what what interesting do you to come out to the meetup so we can kind of age generate conversation because a lot of me that meetups are about conversation rather than know somebody presenting whatever and I Lost my train of thought Okay, well Yeah, and then experimenting with formats really good too At the start we always used to do like one talk an hour long and you know like We have like food upfront so like people kind of eat for a little bit And socialize then we have the one talk and then we end And then we were like well, you know, sometimes talks were an hour long Sometimes talks were an hour and a half long So we're like the we started to notice that the longer the talks the quicker that people kind of zone out And end up leaving beforehand. So it's easier to We did an experiment where we're like, okay What if we did to like to 45 minute talks which are like 35 minutes ten minutes for questions And then transition to another talk. So we sometimes do that. We sometimes do Like lightning talks, so I talked about this file clouds thing that we do every year at the end where we just do, you know Two hours of lightning talks like couple of times a year where they're like five minute talks It's kind of like whatever topic you want to talk about And so the format's a little bit different from that perspective other times We'll just do open spaces where it's like people will come just to kind of talk about a specific technology things that they want to talk about and whoever is in the room kind of like will you know You know join the specific group to you know Speak about whatever that they want to that they want to kind of participate in so it's kind of an interesting format DevOps days conferences kind of You know the group of people that you have is the group of people that you have So that's kind of like an experimentation thing that you do over time But don't experiment too much because then they'll be like yeah every month It's a different thing so you know be consistent for like a few months and then change it around for a month And then go back to a plant that kind of works just to kind of keep it fresh in some ways be inclusive because You know we're all we all come from like different backgrounds and we are all kind of interested in different things so It's important not to like shoot down people's ideas Or you know things that they might be saying so inclusivity is really important and at the same time, you know have a team because You know organizing especially if you're doing it from a volunteer volunteer Perspective is it takes a lot of time and one thing that if you're an organizer like the I've seen a lot where People will do like a meet-up group and it'll be really good for like, you know six months Then the person running the group would be like man. This thing really blows.  So you know you kind of want to watch for Burnout I Notice this like for myself like a couple years ago where I was just like I stopped kind of caring about like tech stuff for a little While and I was like I don't want to organize this called awesome thing or I'll organize it But I don't want to go to it I'd rather just stay home and then I was like what's actually going on with me So I started, you know like talking to other folks about like burnout and You know, you want to If you if you catch it early on you can make you can make changes quicker versus, you know Realizing you're like totally burned out and you're just like man, I don't I don't want to do any of this stuff one of the tricks behind it is to get a team so you can kind of You know do it together so it's not just you so it's not just like a Road that you're walking walking alone on it's you walking With other people as well. So it's a lot more fun easier to like distribute the effort so you can kind of do things you're really interested in like the The thing I showed with Zapier early on the integration. I really like automation kind of stuff So I was like, oh, yeah, you know What if I did like an automation thing and put this thing into slack and built the bot or whatever I was like able to kind of do that and like it was really useful for the group for the organized group But at the same time, I think like a few years ago I would never done that because I'd be like like oh my god We need a speaker and that's the finding speaker is like the most important thing so you can kind of you know Do things you really enjoy And it's also important an organizer. You find yourself in a position of power And so, you know, it's important to be able to give people opportunities So whether it's like speaking opportunities or kind of like You you end up like Learning a lot from you know sponsors you will end up learning a lot from peers Speakers attendees, etc. So you can kind of make connections a lot And that's something really powerful because you know, sometimes somebody might approach you like hey, you know, I just got laid off Do you know anyone who's hiring and chances are you might have met like the sponsors a month before and they're actually like hiring You should be like, you know, you should talk to this this other person or whatever seeking help People help make people bridges basically um And yeah, so like, you know over time like Austin has grown like super friendly a lot of folks just kind of Enjoy kind of talking to each other And so it's not it's not it's it's a lot more Inclusive and a lot more open Then how things Used to be when I first moved to Austin And then you were kind of like nervous talking to somebody else because you didn't want to be judged But the sense of like judgment who has really kind of gone away It's still there, but like it's definitely diminished quite a bit over time I said tech folks kind of know each other ish because I think at the end of the day we're still introverts. So sometimes it's hard to like Go and like meet someone brand new But you know in in general, uh, if you know, if I have questions on like Jenkins or whatever, I'd be like, you know I know eric. I know eric long he could probably answer this question for me Or if like if I have questions on kubernetes or whatever like I'd be like, you know, I think paul zarkowski might know this or In fact, uh, the at the ibm booth today, uh, jj is working there and jj is from Austin So like before this I was like, hey man, so I went and I talked to him for a while Because we all kind of come from the the same place and also You know make it like a culture where it's okay for people to ask questions And because we I think this gets worse as we get older in our careers where we're like It's bad to ask like simple basic one-on-one kind of questions because that's how we're kind of molded And you you you don't want to feel like you're asking like really dumb questions But it's okay. Like, you know tech stuff changes like so uh, so quickly, you know when Nobody knew about cloud like 15 years ago Nobody knew about containers like 10 years ago like nobody knew about kubernetes five years ago So like tech evolves and changes over time. So, you know, it's okay to ask like Really simple one-on-one questions and you know not feel not feel judged for it. So as an organizer, it's like You should kind of breed That kind of culture in your groups So with that, um, I don't have anything more I just wanted to say it's like communities that really open things you as an I as an organizer or you as an organizer you help like foster that community But in the end it's like kind of hard to control how those things kind of grow Sometimes they're really fast. Sometimes they're a little bit slower But in the end it's like, you know, those are the people that you end up driving with so That's what I have. Uh, folks have any questions or anything? Um Happy to take them Thanks What's up? Thank you. So We host a meetup monthly and It's pretty awesome. And I think it's a great thing to keep going The one thing we always have a really hard time with is sourcing speakers. Obviously you mentioned content is Critical and I totally agree. You get it. We've had some really great speakers who've had a much better draw than others which We want to keep that going. How do you suggest? Continuing to find great speakers in your area and you know for the qualification I'm in the bay area. So of course we've got like tons of awesome talent around us for so Do you have any tips on how to do that better? Um, actually like what what was your meetup that you organized just to shout it out? Sure So we host at twitter on the third wednesday of the month. It's the san francisco reliability engineering meetup We're on meetup. Uh, you can also Come find me. I'm my twitter handle is mr. Mocha spell out mr I'll be talking in the next session next door too if you want more detail Nice, and uh, yeah or come find me. I'll give you my business card. We'll follow each other or whatever Won't be creepy at all. I promise So, uh, but yeah, if you're ever up in the bay area, like I said, we have a great meetup and we're always looking for speakers So we had a great little community coming together so far We're running it. I want to save for like three or four years now We just kind of take a short hiatus every holiday season around december because nobody's around that month But the rest of the time we have a pretty good following usually So come join up Basically like what what I'd recommend is and one one thing that we do is we If there's so I go to a lot of conferences And so I'll be like if I go to a conference and if I find out that the speaker's mosque or whatever I'd be like, hey, uh, you should come present this the same content that cloud user group or at cloud often So that's that's a strategy. We kind of use Sometimes we'll hear somebody doing really interesting work Or, you know, we have an org of eight eight people. So it's kind of like federated a little bit more So somebody would be like, you know, like have you seen what Tristan's kind of working on these days? It's like totally blows my mind. So we should get him like out to like talk about that stuff You know Okay Right. Yeah, and and that's the other thing like it's it's if you kind of Spread it out. So it's funny. Um Ernest and I so Ernest is one of the other organizers for cloud Austin Like he kind of is like the main guide that runs everything But he and I used to work on the same team at national instruments like 10 years ago But like now we're kind of like doing our things and you know, I'm more interested in containers. He's more interested in security Um, so sometimes, uh, you know, you'll meet people that come to the meetup And they're always there like regardless of what the content is is they're like your core followers And sometimes it's good to be like, hey, you know, you want to be an organizer with us Like, you know, come join come join the team kind of thing Um, so that's kind of how we ended up growing out because there there would be a lot of folks that would have really good ideas Or tell us like I went to, you know Jenkins rolled Last month and somebody spoke about this thing and I think they're local You know, let's let's have them out if you can and so Sometimes we'll just like follow them on twitter and be like, hey, would you like to come present this thing? You know at the local meetup or you know, I'll send them an email or something like that And as an org perspective, you're just kind of like, you know, trying to make the connection and have folks come out Um, so that's that's an idea the one. Um, I don't know how it works in the Bay Area But you mentioned how a meetups kind of take hiatus In the wintertime actually Austin is the same way the 12 clouds thing that we do It's not just our meetup. We have like a whole bunch of meetup groups Kind of join us for that one. So it's like six groups or something like that that come and like present on different topics so like The postcards meetup, for example, we'll we'll collab with us and we'll have like postcards content. We'll have You know cloud content, uh, google gcp kind of content or whatever So it's like a collection of stuff because it's all in the end like people just want to hear cool things So they're like, okay. Well, we'll like talk about that stuff So long wind and answer, uh, and then if I think of ideas, I'll I'll can treat it at you as well What are the questions? All right Five minutes. Okay. How do you deal with nervousness going to new places? Is that one reason why you should always hold it at the same place as well? Uh, nervousness you mean from a speaker perspective not just from a speaker perspective from even an attendee perspective Um from an attendee perspective. So one thing we do Which I think is like a good practice is um, since I go to all the meetups You know, I know who's kind of been there before who's new And so for the new new folks, I'll be like, hey, you know, I kind of like go a chat of a conversation like what's brought brought What brought you here? You know here's kind of format or meetup, etc.   So we're like, uh, you know, it's it's fine if you screw up or whatever So it's not the people for me. It's the new places that really get me nervous I don't know if that's that common. It seems not to be quite rare. Well, did you have something to add? Well, let's get you a mic real quick. So it's because we're recording all this stuff So I have the same problem. Also, I hate going to new places and I'm really bad at following my dps And uh, so I'm always getting lost. So when I do my meetups, I go ahead and I post Kind of like gps Images like I'll look it up on google maps and show you like, hey, this is actually what the building looks like From the outside and I'll go through and I do give people like detailed information Because the gps will tell you like how to get there But sometimes there's one weird way to get into the building that will stress people out or something like that So I try to be as detailed as possible When giving that information because I know that's something that prevents me from going to new places Yeah, that's oh, yeah, that's a good point and parking because parking sucks always yeah We do that a lot for dev ops days as as the conference and we've been doing this meetup at the same place for like Six seven years now But for a conference stuff we definitely do that where you have a lot of signage and yeah I used to be that guy who'd like drive the day before Or like a couple of days before make sure that you know Like you park in us in in the spots parking thing was open kind of thing Make sure we have sign it so because It used to be kind of a walk So it's like make sure we have signed just specific spots so people don't get confused And you're not like man.  It's like it's really it can be a really nerve-wracking From a user perspective. So um Those are I think all of those are really really good tips Um anything else did you have a question to you? Yeah So I I know you said that you have the event at the same location every single time. Um, do you Do you ever run into our Or look into doing events at different spaces to get more people come to your meet-up because in la like it's very large So I have to when I'm running my event Try to plan different venues so that we can reach all of our members Um, so I just wanted to get your viewpoint on that. Um, we we sometimes have done that where we'll be like, okay, uh, you know Those events a while ago and this was like a few years ago where We had like a doing cock croft come to speak in austin and so we were like, well, we need to make sure we have like Uh, a bigger venues. We ended up doing a call for the venue Way beforehand, but we're trying to plan it like way in advance Um, because even even with all that planning people still ended up going to the to the older venue or to the original one Be like, hey, I'm interact space. Where is everybody kind of thing? Uh, that maybe that's an austin specific problem And then I think I think I'm out of time, but I'd love to chat with you and give you some more thoughts on that too But thank you all for for being here. I really appreciate your time Thank you guys for joining the mentoring track. Um, our next talk in this room is at 130 Also, note that the expo closes at two today. So if you wanted to get to a booth make sure you do it before two Thank you. See you at 130 Can you actually hear a difference? Yeah, and the slides are actually on there No, I'm fine. All right. Thank you. Uh, yeah, it's all done Yeah, well Now we can start All right, uh, thank you for coming. Um So I wanted to cover a few key aspects of the fairly long history of the exam project. It's you know, 15 and a bit years old Um and Basically, I'm going to pick out a few sort of sets of stories and then at the end of it we're uh, going to look at why the lessons which Could be of interest for other open source projects and also for us in general And that story really covers, you know, very different aspects of the history of the project And some of them are actually not very well very well known so Before you ask the presentation is already at that, um, uh, link And I'll you can download it from there So My name is last curts. I'm the community manager of the exam project I'm also chairman of the exam project advisory board. That's a number of vendors. Um Supporting the project fund financially and I happen to work for citrix I do like flowers. So, you know, I always put up pictures with me and flowers on it. So this is actually a queen of the night, which is quite lovely and Never mind, and then I wanted to thank our country the contributors to their talk Andrew Cooper, Christopher Clark Um, me hide onto and george dunnab So I've been with the project since 2011 And so I had to pull in some of the longer term committers to get And concrete us to the project to get some of the bit some of the earlier history Of it all so What is xen? And xen project. Well, it's a versatile virtualization platform It was always designed to be a component In a software stack. So, you know making an easy to use end user experience was never really a design goal Which probably was a mistake? Well, you know If you look at kvm, for example, it's like a nice and easy to use out of the box and In xen's case that isn't really the case but That has enabled other things which will cover during your presentation So the way how I kind of often look at it is The xen hypervisor is a bit like an engine engine taking a car analogy and system integrators basically build a car with xen as an engine in it and You know kubes os is an example of that xen server and so on and so forth And then the project We have this sub project structure where we develop Several technologies which are related to xen. So there's the hypervisor itself. There's a project to do pv drivers We had a number of projects for specific goals Which were temporary like one to bring arm support to the platform or get xen support upstreamed into linux And there's a number of unicolonal related projects as well but To really cover the talk and and some of the lessons we have to go back to the beginning To 2003 when it all started and I wanted to look at some of the original use cases for which xen originally was designed And you know the primary use course was really server consolidation Um Then you know specifically also in the original paper They listed co-located hosting for facilities and distributed web services, which eventually became cloud computing And then you know a whole blurb around secure computing platforms and mobility and stuff like that But actually in reality, that's very closely tied to server consolidation with high availability for torrents and those kind of things What's really interesting also is that some of the terms, you know, like cloud computing and xen kind of historically go very well together Um, and if you're interested you can still find the original paper The original design goals are also really interesting. So There were like really three design goals. One was around modular architecture and One of the ways to implement that was desegregation Um We also have a very flexible architecture by user system services And a high degree of being able to customize the platform because The initial goal really was to target servers, but maybe also desktops as well And many of those are still relevant today And have enabled a very Interesting and complex ecosystem and i'm going to try and sort of explain that through a little bit of a idea and product Genealogy So we started off with server virtualization and on the side, you always see the kind of products Which are relevant in this space then In 2006 amazon and it was also slice host at the time which was bought by rackspace started to use xen in cloud computing and That really kicked off very widely What was interesting and i'm going to cover this a little bit because it's not very well known the Department of defense kind of took xen and forked it and built a whole series of platforms around that and there's some really interesting stories there um Then there was an attempt around 2012 2013 to bring xen to desktops There were a number of companies trying to including citrix to do that, but ultimately that flopped And but it was still an interesting milestone for xen because it enabled other things We Had a number of unikernels. So the first of those unikernels is Was galore. What was halvm? Which is a unikernel targeting haskell Which has been developed by galore um And then what's really interesting out of this whole desktop world We got specialized We i call it defense applications so that like the cubes are was very very and for dod for the dod primarily Targeting desktops laptops as well as tablets um Then that led to a number of vendors which are active in a space looking at expanding their use cases. So um, so we've actually seen platforms such as virtuosity created by company like dener works which use xen For mixed criticality applications in a different defense context and more recently and this is This happened entirely Surprisingly to me We have seen a lot of momentum in the embedded and automotive space so If you look at this right So how did we And there's also a set of technologies called virtual machine introspection Which can't work with the others. Um, which is a security technology So if you look at this How did we actually get? To such a diverse ecosystem. How does this happen because you know initially We were targeting server virtualization, right and what were What were the factors in a community which kind of enabled this and then the other question is of course, you know Is this a good or a bad thing, you know, and what would you have to do if you wanted to replicate this as a project? um so the first reason Why what we kind of went into this direction is That xen came out of a university project, right a lot of the Committers and maintainers and developers all Had a research background You know when when it was commercialized all the guys working on this stuff working at the university just You know became employees of xen source at the time So there was already a very strong Affinity Generally to research and cool ideas and you know just letting things which may not necessarily be You know good for the vendor behind the project happen and to support that covertly or you know directly um But also more importantly it also comes back to a strong security mindset Which you know which was already um Encoded if you want so in the in the original design goals So that kind of laid the substrate to make some of this happen So if you look at it, you know like so we looked at some of the design goals um So disaggregation for example was part of the original xen conception And that whole idea was influenced by microkernel thinking Um So let me just explain you know what I mean So if you look at a typical xen architecture, you have your hypervisor down there You have a virtual machine and then you have your dom zero and then we have a couple of boxes here So we have system services. So we were talking about design goals earlier so system services are basically Applications which run in your operating system and some of them can be replaced. So we have for example Two xen store variants. You can you notice one or you can run xen with with q mu as a device emulator or without one Today and there's like three different implementations of two stacks, right? Then on similarly You can replace the kernel. So you you basically run either linux or bsd But just for example work going on today to run xen without the dom zero at all Or to run it with an autos All kind of signs of that level of flexibility But coming back to desegregation desegregation is that idea To basically be be able to run some of the things you're running in your in your dom zero in some You know other other virtual machine and you decompose your system that way And that creates additional security boundaries and there were quite strong design constraints around that to make this happen. So all the PV protocols which are basically the interfaces between the various xen components were specifically designed for that use case and we had to sacrifice Performance and make other trade-offs to kind of keep this, you know, keep this going and this has been a constant throughout the life of the project um And then of course, you know, since the beginning security technologies were always in the mind of our maintainers and quite often, um, you know features which were not necessarily core To the project were just upstream without resistance Um, and sometimes with a lot of support from from maintainers so If you actually look at this um Disaggregation is a really interesting example because if you look at put it back onto the project genealogy You will find that in server virtualization and cloud computing That stuff isn't actually used, right? But it's used in all these other um products and markets and um If you then start thinking about community dynamics, right? That creates a degree of tension And that's not always that's not a very happy story to some degree You know tension is always like the precursor to conflict. So if you look at this um Until very recently all our key members of the project were employed by companies who developed products for the server virtualization market segment, right? and you know That's kind of the core use case and Everything else is kind of non core and in other words really that means that You know, everyone else who built products on the non core stuff really relied on the goodwill Of the maintainers to basically, you know sustain these other ecosystems and use cases And you know, if you look at that that's kind of You know, you kind of can see that sometimes they might might create some frictions and difficulty What's also interesting is that you know, we're now seeing a shift where um, where the community is becoming more balanced and that's Just started very recently. Um, and it's really accelerating um And that actually required a lot of actual work to make that happen So let's look at this tension again, right? So and and bring us back to desegregation because that's kind of a good example So First of all, we also have to remember that 10 12 years ago Well, yeah when when cloud computing kind of started the The attitude to security in general Um in our industry wasn't actually really very good compared to today, right? So I guess if you look at it in terms of analogies um You know, then the red security where we're now with privacy to some degree, right? um So what what happened so we had two things we had, you know, core maintainers, you know Not really having any, you know Not using a desegregation feature in there in the products which the employers, you know, funding them to work on um But at the same time you're getting features in And you have to kind of make sure that You know, this desegregation and the basic principles don't break but At the end of the day, you know, if you're working on a product, you're going to cut corners, right? So, you know, whenever whenever the core team chose Uh, you know, I had to implement a new feature, you know, they would Use the quick route or the more performance route for the main use case And that meant that, you know, sometimes those new features went into dom zero first Um Then of course also, you know, if you're a maintainer and you're familiar with your use cases and someone comes in with a very different use case um to really Understand that entire use case very well and then, you know, try and come up with the best possible approach That takes a lot of mental cycles, you know, which they're again often not prepared to do And so again, you know That sort of rigorous review which would happen for the core use case that hasn't always happened in those cases and You know, and that led to Consequences, so for example, you know Like a fairly good support for desegregation in a platform But, you know, nobody ever thought about adding build support, you know to it and the result then was that pretty much everyone on that um Graph is all their products who uses desegregation develop their own their own solution for building those things And you know that sort of duplication and increases the barrier to adoption For newcomers at the end of the day and it also creates unnecessary complexity So to summarize this section um So we basically the strong security mindset really enabled contributions which Were related to security Which at the time were seen as sort of non-core because you know, everybody talked about security, but really You know at that time It wasn't as important as as as features And there's plenty of this is really Quite unusual, you know, there's loads of examples of open source projects, which if they have a certain goal for their project If something which isn't quite fitting into this comes in There would wouldn't necessarily necessarily allow that feature to be upstreamed and We've been suffering from this ourselves Occasionally so like an example for example is is qmu And you know the primary use case is as a machine emulator or a virtualizer But xen itself actually doesn't need the virtualization support. It just needs the Device the back ends and emulated devices so xen kind of uses qmu like a A library for back ends and emulation devices And all the other stuff we actually don't need and we've tried multiple times over a period of 10 years to work with that community to have an officially supported configuration and we never succeeded And you know similarly You know another similar example is I think intel Try to do something similar for a slightly different use case And that then ultimately led to a fork called ne mu, which is no qmu, right? and so So that's just an example that you know, we were kind of somewhat unusual in that in that sense So let's get to the next story and That um leads us from you know From the classical server with cloud space to military clouds and desktop and tablet use cases And there's some really interesting nuggets in there, which I only found out about last year So that story started in 2007 at xen summit in new york And we're specifically looking at that fork from server virtualization to defense application in terms of product So at that event Do the employees presented new research around how would you be able to Um apply five common criteria security clearance for xen and a lot of this was about just Code review, you know, I'm really sifting through the code and And really checking that everything works as it should do And so there's obviously a correlation between the amount of code And the amount of time and effort it takes to do that And so you would actually want to take, you know, build a more minimal version to make this more Um financially feasible and I also developed the xen security modules That research then led to the xenon separation virtual machine monitor family, which today is widely used In all across the dod. I only found it out last year Um, when um when I was invited to participate in a security event, you know, which Those guys organize called the platform security summit if you google that you can find Some of the talks which represented that they're quite interested Uh, interesting subsequently, uh, the xen security modules were upstreamed But all that other work, you know, was just basically used by those military types and I never really tried Very actively to engage with the project or even upstream it So let's look at this in a little bit more detail because I I mentioned this already so the uh, so that research at the um, uh, at that xen summit in 2007 led to a prototype and uh, um And there were really two interesting things to highlight here. So, you know, they fundamentally Had a team of three engineers over a period of a few years kind of tracked the code base and you know, take stuff out and uh, and do the Do the actual certification to see how much effort it would be to keep on top of that and uh, and it really showed that actually the The model of design to some, you know, was validated through that activity and it also showed that you could create and maintain A cut down variant of xen was about three man years You know over Over two years. So it's one and a half, you know, that's quite a that's that's not too bad from a cost perspective if you really care about this stuff so That was kind of the one ingredient the other ingredient was an attempt by three vendors to bring xen To desktops And it was basically, you know, basically the idea would be you'd have your hypervisor on it Um, you know, you would have be able to run multiple, you know windows linux, you know, whatever you know all on your desktop and You know all those vendors thought this would be the next really cool thing But you know, it never really happened. It was a little bit like the year of the linux desktop, right or or other things like Yeah dual symphones and all these other sort of more complex things which never really picked up However this really Was an interesting stepping stone to cubes os it's a very active project and And then it also led to something we call open xt and secure view Which is basically the The dod equivalent of cubes os Which they basically use in various agencies So So that's kind of what happened And I wanted to really think about Why this happened and what enabled it So What the engagement of Some of those security types, you know with the project really did is it created a virtual cycle which Which made send a very attractive platform for security research and it's still the case today And basically, you know the enabler there again was we accepted those contributions and of course the The modular architecture and which we talked about earlier um Then the attempts to commercialize xenon desktops um Really late the foundation for a new class of project of products But you know again, you know, none of these capabilities were used in in in in the core use cases There's a similar related story um, which Played out very similar which was around gpu pass through and graphics virtualization And actually what's not very well known is all this stuff was actually developed For those desktop use cases Um, but then, you know, it actually turned out to this is quite cool stuff Which you could use in the cloud and uh, and so that's kind of an example of um, one of those no no Non-causing is actually been being beneficial for Um In the original use case cases So the next set of story is around virtual machine introspection so Let me just briefly explain what virtual machine introspection is. So if you If you're trying to find, um, you know, if you're trying to to to monitor your operating system for Intrusions like through viruses or the malware normally what you do, you know You install a virus checker or something or some software which you know monitors what's happening within the operating system You know for specific Signatures and patterns, you know, and if you detect it you quarantine it and so on and so forth um Virtual machine introspection because you actually have the hypervisor It's basically so so the the early approach is essentially equivalent to having Having a camera in a prison cell, you know where the prison cell is your vm um Where virtual machine introspection allows you to do this from the outside Which is basically the equivalent of a panopticon So it's basically, you know, like you have all your prison cells open and then you have your you know, like a A big tower which just monitors everything and That also started off in around 2007 at a at georgia tech Where A number of researchers, you know came up with this idea and started by research around it and in brian pain Created a project called center xs which later, you know became lib vmi, which is still around today Then in 2009 That functionality was basically integrated into into the hypervisor And again, you know This was like a non core use case at you know at the time. This was an interesting research project Um, um, you know, why would you want to take that into a platform? Which already had so many products built on top of it But you know to maintain a shepherd this through at the time And actually provided quite a bit of active help um Then many years later um Inter launched it has well and broad world cpu generation which had some features in it Which carried the promise, you know to To really speed up this whole concept of virtual machine You know if you if you if you look at how this works, you know, you're you're monitoring memory accesses, you know from the outside in a vm and you know The performance overhead of that is potentially quite quite high um And so those new hardware capabilities um Sparked interest by a number of security vendors to really look at this technology when I actually looked at it realized that In fact, the overhead really wasn't as big as they originally thought and I could already productize it um But you know as a result of that initial engagement, you know, they re-architected the subsystem, you know and put a lot of effort into it and In parallel, you know developed a number of companion technologies for it And then you know just two years later. It was productized And you know today a number of similar products available from a company called ais and a startup called scientific And there's ongoing contributions You know to those features, um, which indicates that some of those technologies um Are going to be included into further or existing new products So looking at this again um From a from a reflective viewpoint, you know, if you really take a step back actually Some of those security made features made it into the original Server and cloud use case so some of them were specifically developed for it like life patching That kind of happened as a need as a response to the uh cloud reboots. We had a few years ago But you know Most of these other ones just happened sort of happened by chance because somebody had a cool idea and then it enabled an entirely new ecosystem around it so I wanted to look at this in a slightly different way so so why What were the reasons for some of those technologies not actually making it into Into the core product for the core use cases in the in the platform. Well, You know one example is the the hardware capabilities to actually do this in an effective way when went there Or it was too early So there's a whole set of technologies Incent around Trusted boot and stuff like that which are used in cubes and a few other projects But they never really made it Into cloud computing until you know until now. So there's a lot of activity around this today primarily because of microsoft and Manating that for for newer versions, you know of windows going forward Also in some cases, you know, I talked about the extent security modules beforehand There use is complex and very use case dependent. So you couldn't really make You know make this easily usable in a general you know in a general sense and You know, it's also quite a lot of specialist Expertise was was required So that's kind of the one so the last part of our story is It's really going from From this desktop Use case which really didn't you know, which ultimately wasn't successful except for in niche in use to mobile use cases and embedded automotive use cases and that's where it becomes really really interesting and that's also where In the last year or two I've seen most innovation and disruption in the project is coming from vendors in that in that space And that story Started in 2008 in Seoul in Korea Where samsung basically released What I presented a you know a port of xen to arm um That was using Paravirtualization because at that time armcpues didn't have Didn't have virtualization support and both Me and Ian Pratt Well, actually was Ian Pratt at the time and later me were trying to convince samsung to really upstream this stuff But they never really we never we didn't we didn't succeed, you know samsung didn't want upstream the work Primarily, you know, I think at that time, you know, particularly the far east, you know, china Korea and all those countries and vendors in those countries really didn't know how to work with open source projects and At that time, you know, that was a big issue and You know in the last 10 years or so this has obviously changed a lot Um So we then settled, you know, we then basically said well, we can't get them to upstream this but we don't want You know, we want to draw them somewhat into the community So we allowed them to host, you know, their own separate for folk fork within the community and engage with them on an ongoing basis um Another meant that samsung, you know regularly showed what they did at at developer summits um The last thing they did was Uh A dual mobile operating system running, you know, showing that on a samsung On the very samsung devices and later nexus 10s Um, really showing, you know, like I had this cool demo where in one vm, you know, you had Angry bird running and on the other one, you know racing game and they've dealt with all the graphics virtualization stuff And it was really really cool At the time And really to do all this they had to do they had to implement real-time capabilities and they also Had to take some of the gpu work Which was done for the desktop case and bring that, you know, to the um Bring that to the arm environment So what this really did is You know, even so that was never upstreamed It showed people in the community what was possible And so it inspired others to really go and look, you know At this along Similar lines, right and so so that work from samsung fundamentally laid the groundwork for For the developments we're seeing now in embedded as well as automotive Where all these graphics and rich io capabilities As much needed as, you know, there would have been on a mobile device For mobiles that never really picked off, you know, you can imagine The user interface implications of having different personalities on your phone and so on that's all quite awkward I don't know whether you ever tried to use one of those dual sim phones Um, you know, it's a similar similar kind of problem not not enough market appeal for that But from my perspective, you know, I'm not actually seeing this as a failure And I was really a useful lesson and as a project we could really build on this So what then happened in 2012? um At that time arm brought out, you know added virtualization extensions to their cpus and That led to a complete reboot of the xenon arm port. So a number of people got together And started upstreaming xenon arm support, you know, properly using the using the Port that samsung had originally developed as a as a baseline and you know really looking at it and see oh They've done it this way. Maybe we can improve that right as a as a nicer and better iteration and learning from what had been there before and Interestingly enough, you know, so arm added virtualization extensions to To set to their servers to their cpus to target servers So, you know at that time everybody, you know assumed well in three or four years, you know arm servers would be really Quite successful and you know create a really tough time for intel, but you know, that hasn't really happened And of course, you know, we really learned from that early arm port. We ended up with a clean and simple architecture Um, there was a really great match between the arm instruction set and xen's architecture And we ended up with a really tiny hypervisor So the initial port for one platform was around 30 000 lines of code the equivalent of x80 of the x86, you know code would be around 200 000 lines of code, right? and The fact that we had that upstream support and the simplicity and smallness of that, you know of that arm port really suddenly sparked interest by By embedded vendors and for embedded use cases But there's a missing evolutionary link here because you know, we were talking about um vendors who were active in the sort of military space Having an interest in xen and that happened before we did the arm port, right? so there's a company in in called dirner works And they launched a product called arlx which initially was an x86 product they added real-time support to its scheduler support and and then later safety certified this thing that this product um for Avionics standards, right? and so that that sort of showed that you can actually take Xen take stuff out and then get it safety certified and uh, you know Have it run in drones or you know offer some software on planes and similar things and That then again, you know created that bridge, um to uh To the next evolutionary wave, which is really around embedded in automotive So this really started picking up in the last few years We had our first automotive based platform in 2015 Um, we had a first embedded arm only xen distro developed by xilinx in 2015 and in the last few years there's been an incredible amount of rnd and a lot of You know, I'm really excited about the next one or two years because this this entire space has You know create so much innovation and new ideas and it's really it's really challenging What's happening within the community? there's another little thing I wanted to talk about And this is so I I thought I called it sedimentation because also what we've seen through the lifetime of xen is that Stuff that used to be implemented in software Eventually gets you know moved down to the hardware And I thought that'd be a really a few really interesting stories around this um You know an example of that is we used to do interrupts and timers fully in software And you know today on newer hardware, we just used the hardware support and we kind of constantly had to adapt to that and there's some It's also really interesting those silicon vendors like intel and amd, you know, they have always really worked very closely with xen and so The way how these instructions sets and and this hardware support has evolved is really heavily influenced You know by that butted interaction and I thought There's some really interesting stories. We could tell but actually as it turns out I can't really tell these stories because the the These cpu vendors hadn't actually engaged with the community But they had engaged with vendors in the community and so all of this happened under under nda's So to close so What have we actually learned within the community and you know, what's there to learn you know for for you guys and The first big thing and that was a really big scenes as things through throughout that presentation Is that culture really enables innovation, right? So So because we had this culture to just accept stuff Into the code base, which wasn't core This really you know, this really enabled things interesting directions, which nobody thought could happen This wasn't really planned in our case We just inherited this due to the you know due to the project being a coming out of a university But you know, we didn't really do this well at the beginning, you know, we Sacrificed a lot of due diligence, you know initially and we could have done that, you know better And we have done it better in the last few years We also now But also didn't do this at the beginning really help companies and organizations to You know, if they come to us as a you know and say well, we want to do that We just won't shut the door. We will actively work with them and try and find the best approach And also from a community perspective, you know, we always have worked with new ideas And with forks of the project as well as at distributions so examples there are At our developer events, you know, we always gave airtime to new ideas So, you know, we're you know in the early days of unicunnels, you know I always had a platform within the community. The same is true for many of the other things Forks aren't always bad You know, you think oh my god a fork is happening, you know, we just It's a bad thing, but it's actually not Not necessarily always a bad thing and if you If you have downstreams With special needs, you know, you really should work closely together with them And that's also something we haven't done very well at the beginning But we are doing this, you know a lot better today Another thing really is there's a big I didn't talk about this in a lot of detail, but if you look at this You're the way how you do your process Can either enable or hamper, you know innovation so We had to sing incense straight from the beginning where we have we have like a feature life cycle You start off with something experimental and then something, you know And then we come tech preview and then eventually we become supported and You know at the beginning we kind of use this life cycle as a back door to get You know good to get stuff into the platform, but actually also market As maybe you know, maybe you need to polish this a little bit, you know before you actually build on it Whenever there were larger contributions of features which weren't You know which weren't part of the non-core use case who would always insist that the contributor would step up as Because we didn't want to have stuff in a platform which would spit rot and then, you know, obviously um Maintainership itself isn't you know, isn't enough they could go away. So we always have to have From the beginning, you know the capability to deprecate and remove features, you know If it actually looked as if they you know as if they were becoming stale and we have Exit, you know done this quite a few times the other thing is Is that the tension within the community you will have this tension if you have You know if you have people and companies which come from very different backgrounds and directions there's gonna be tension and these can lead to conflicts and We had quite a lot of arguments between those groups and you know, they never really go away and they can go get quite ugly and that really means You have to do, you know active community management. You have to you know, you can't step into those Conversations and try and shut them down right from the beginning because sometimes Creative tension is needed to lead to something interesting. So that requires quite a little bit of skill and it you know, it's also a little bit risky sometimes Another thing which was really surprising to me is that if you introduce a new process that can sometimes lead to a significant increase of tension so around 2013 2014 and you know again about two years ago We looked at our security vulnerability process and we kind of formalized it and tightened it and so on and that had unintended consequences because you know, like All our core developers were focused on server and cloud computing And you know, and we have the security team, you know, which is security response team, which had all security vulnerabilities Primarily, you know for those two use cases and But what happens if you find a security vulnerability in code, you know For something like cubesos or something, which you know, isn't in this cloud development segment and you know, when we started to formalize this Suddenly it raised the question. Well, you know, who actually has to fix this, you know Who has to jump drop everything, you know and And fix an issue in that in that piece of code Which some other company, you know, which i'm not being paid for Is actually benefiting from or some other project And that led to a lot of Tension and then in the end, you know, we had to resolve it In this specific case, you know, we just adapted, you know, our process and basically said well You know, we can have supported features Without security support or You know, we basically say well, you know, if you have a major feature and you have a product built on top of it And you wanted security support it Nominate some contact point, you know, and then if an issue a person is responsible for that within your project Or your organization and we, you know, if that issue arises We're going to come to you and you know, you fix it and work with us And We also started to heavily use kconfig to be able to build different Configurations and larger features which, you know, which are non-core are basically build or runtime disabled And we use sub projects as incubators for for new ideas And these are really seeing things, you know We have an automotive project and in this in that area We're currently looking at things like how do we deal with safety certification better and so on and so forth And that's basically it. So if you have questions, please go ahead No questions. That's all right Don't worry about it. Thank you very much Thank you for coming Okay Working yes, maybe okay good Now I'll turn it back off Seems like as good a time as I need to get started since it's three o'clock It's sunday the end of the day almost the end of the day. There's one more talk slot after So not the very last So but I do appreciate everyone sticking around today. It's a scale to Not not for the faint of heart to make it all the way through. So awesome Awesome and for like probably the best weather has been today, right? So, you know, there you go So thank you for coming to my talk. This is called fight flight or freeze releasing organizational trauma A little background because I don't have an interest slide here Because quarry quinn told me that resume slides are bad and I should feel bad if I have one So I took it out of this one, but my name is maddie stratton I am a devops advocate at pager duty And that is probably one of the last times you're going to hear me talk about pager duty throughout this entire talk If you want to know more about pager duty, we can certainly talk later As far as things like that go And I I really really love this talk. I'm really excited to be able to give it here I do have to you know, since mary is in the audience I'm going to give a little shout out So the first time I gave this talk was at a conference Over the summer called redeploy, which is a conference about resiliency engineering and A little bit of the background on this is I had was talking to Mary's co-organizer jpaul read about Some ideas as I was learning about post-traumatic stress and I started to say like oh, this seems really similar To some organizational issues and he said you should totally write a talk about that and we have a cfp open And I said well, I'll submit it mostly because I want to get Feedback from you and mary on the submission and then they accepted it and I went Shit Now I have to write this talk Uh, but it was uh, it was a really great journey to write this talk and I'm glad to be able to share it with you So first of all just a little bit of a content warning. So we'll be talking about There's some discussion of trauma and post-traumatic stress. There's no specific I don't talk about my specific trauma or anything like that, but Be aware that we are talking about post-traumatic stress a little bit Disclaimer the second so while I am a trauma survivor I am not a mental health professional. So none of this is Should be taken as you know direct medical advice or anything like that But these are some analogies and experiences from things that I learned Being a survivor of post-traumatic stress So with that, let's get into the talk So consider the zebra So if we have a zebra that's at rest, there's no threat coming on. He's just zebra's just chilling around being a zebra self Right, so this is operating the zebra is operating within what we call rest and digest This is the rest and digest functions of what's called the parasympathetic nervous system So there's no threat of predators. There's no stress As its name implies, you know zebras mostly think about eating and maybe about making more zebras depending upon You know what where what's going on with that zebra? Now We introduce a lion now. There's a predator. Okay, so now a lion is coming and chasing the zebra So there are drastic physiological changes That happen with the activation of the fight or flight response of the sympathetic nervous system So the zebra's heart rate increases its breathing increases its pupils dilate There's large amounts of stress hormones like cortisol and adrenaline are being released into the zebra's bloodstream Its blood pressure increases and any non-essential functions like digestion. They come to a stop The zebra's nervous system is preparing it to literally run for its life now Let's say the zebra gets caught. Sorry zebra I need to get you caught by a lion in order to make a point So say the zebra gets caught now the zebra's nervous system is overwhelmed. It has no more solutions So this is what's called the freeze response the zebra basically plays dead and it just freezes and it stops This is also a point that we might consider in a non zebra. We might consider this the inflection of trauma So Zebra gets caught by the lion plays dead lion drops the zebra kind of moves along maybe to go try to catch another zebra Now and so the zebra is maybe able to escape. What happens? What does the zebra do it actually literally shakes it off the zebra its body shakes And it's returning back to rest and digest. It's making a physiological change. So this is what happens When it comes to zebras now, I have if there's nothing else that you take away from this talk It's going to be this next point i'm about to make Which is that humans are not zebras So if there's a slide to take a picture of it's probably this one So all mammals have the autonomic Autonomic nervous system in common, right? So there that's the fight or flight Response this is this is common for all mammals Now as humans we have this thing called a prefrontal cortex Zebras don't have it lower mammals don't have it. It's something that's unique to humans And the prefrontal cortex is usually an advantage, right? This is uh, where executive function happens The prefrontal cortex is how we differentiate between good and bad between same and different How we understand consequences because we have this prefrontal cortex The disadvantage is this is where we mentally replay traumatic scenarios Which activates that sympathetic nervous system exactly like a real threat would here's the thing We all think we're humans and we're super smart.  Our brains can't tell the difference between real or perceived or remembered trauma So when something happens that is similar to a traumatic experience that happened We have the same physiological responses Zebras don't have this right other mammals don't have this they don't go and replay this trauma the zebra that got caught by a lion It's not going back and every time it sees a lion is is going and having this same response Because of the one time it got caught So uh, so dr. Peter Levine has said so animals in the wild they're not traumatized by routine threats to their lives Humans on the other hand We're readily overwhelmed and we're subject to the traumatic symptoms of hyper arousal shutdown and dysregulation Is one of the few slides I will actually just read out loud to you. I don't usually do that But this is a really important quote. So peter levine has done a lot of research on this fight flight or freeze response And I'll have some links later to some some work that he's done So, uh, let's consider a healthy nervous system. This is what something that we call The window of tolerance. So arousal or activation occurs, right? Something happens Lion right So we're going to do that that triggers a sympathetic nervous system, which causes a physiological response And then we settle which brings us back to the parasympathetic and then something will happen that might Cause the response again We kind of bring up and as long as we're staying within this thing that's called that's what we might say the Normal range it's the window of tolerance, right? So the window of tolerance is a zone of emotional arousal that's optimal for well-being and executive functioning So this is normal, right? You're not you're not steady state all the time Something happens that requires a response. You need to respond the trigger The difference is can you kind of come back down to settle when this happens? Now what happens when we talk about an actual traumatic event? This is when we start to see so in an undisturbed traumatic stress you have a dysregulated Nervous system and I know I'm not supposed to move but I'm going to because I want to point So we kind of sat there. This is our normal window of tolerance Our normal thing we get triggered something happens and then something really traumatic happens and we can see how Our response becomes very dysregulated. It can become very Up down all the order and we can either either be continually flipping back and forth or we can potentially be stuck on Or we can become stuck off So symptoms of when we're stuck on is when we are traditional symptoms around anxiety or panic, right? The inability to relax emotional flooding sleeplessness a digestive problem which shouldn't come as any big surprise because this is the opposite of rest and digest So our body is physiologically taking resources away from digestion Or we can be stuck off which are some of the typical symptoms around depression. You have lethargy Exhaustion when you have chronic fatigue. These are symptoms of being stuck off a lot of other things and again Poor digestion because now you're spending all your time in rested digest You should be able to be going kind of back and forth Hyper arousal, which is stuck on And similar to fight or flight right, which is sort of that up here Hypo arousal is similar to the freeze response where you're kind of stuck frozen So we're going to come back to the window of tolerance when we talk about organizations in a little bit Here's the thing trauma is not Simple there's a lot of nuance to it and a lot of things to keep in mind Which is why it's not a simple scenario and that this is why this is not a 10 minute talk And it's a little bit of a longer talk because there's some nuances. We're going to talk about So here's a couple things to think about right so first of all, you know the nervous system has been overwhelmed So it occurs when a solution the active response to a threat doesn't work Right, that's what causes something to be traumatic because things happen to us all the time That aren't necessarily trauma. They're just necessarily maybe there's bad things bad things happen to good people But if we have solutions to them, they're not trauma trauma is when we don't have a solution when What our nervous system has been overwhelmed by something Again as you talked about the nervous system activation is the same for real or imagined threats Right, so it can be a real or perceived threat And then finally and this is really important. It's subjective and relative, right? So what I perceive is overwhelming or my capacity for a solution may be different than what you perceive as overwhelming And having gone through Work around post-traumatic stress and everything It's quite common to for me to feel like wait a minute What i've gone through is nothing compared to what some other people have gone through but again, it's all relative And different people have different capacities And different experiences How does this actually apply to an organization? And one of those kind of key things that all of those pieces happen Right, like when we talk about the nuance of it and again bear in mind that just like different people Respond differently and have different capacity different organizations based upon the humans that make them up depending upon the business that they're in Depending upon regulations all sorts of other things Will have different capacity for for dealing with this. So let's kind of go back To our deregulated nervous system and see how this might apply to an organization when a traumatic event occurs traumatic event being an unplanned outage Aren't all outages really unplanned I guess really an incident something That is beyond the norm Right and not all incidents are traumatic for an organization either some of them And we're going to talk a little bit how we can keep them From causing us to go into an organizational state Of undisturbed traumatic stress or into an organizational deregulated quote-unquote nervous system of our organization So we'll think about hypo hyper arousal, right? So that's the fight or flight So if an organization this is being stuck on right? So if your organization organizations that are hyper aroused They display efforts effects around Constant vigilance and sounds like they're fighting Voldemort, right? They're hyper aware of threats And and this takes energy away from moving forward. This is kind of constantly being on the defense Right. These are organizations that are stuck on A lot of times this is reflected in how leadership approaches outages and issues We hear a lot of wartime metaphors a lot of military military metaphors a lot of we are we're in wartime right now Another symptom of this is when you have production support teams when you have teams whose entire job Is to be watching production And supporting incidents and outages, but they're not but it's not actually connected to the innovation of the organization These organizations have a really hard time moving forward because they're spending all their time Looking over their shoulders, right and and being you you need to be aware I'm not saying that you should just say, ah, you know, I mean shit happens. Whatever, right? But there's a there's a line We can also become hypo aroused, right, which this is when we're stuck in freeze These are organizations that are afraid to make any kind of a change because to be quite honest What causes incidents usually a change, right? So we can in an organization that it becomes hypo aroused Again have trouble innovating and moving forward because it's sort of the sphere of if we change something Something's going to go terribly terribly wrong. So we're kind of in this down state all of the time So bear in mind that signal again remember when we think about How trauma comes in just like our brains can't really tell the difference between real and perceived threat as organizations we often Try to pattern match, right? We say this looks similar to the last time we had this massive Cassandra outage So therefore because we see a signal that looks kind of like that It's probably everything's probably about to go completely shit So let's mobilize everything even though the signal might not necessarily be indicating the same kind of a thing So we're looking for pattern recognition because that's something we want to do with humans and And while the signals may seem the same as previous examples of seeing that we've seen The complexity of the systems that we work with mean that things are never that simple right The days First of all bear in mind, right? So one of the things to keep in mind is that our systems are always in some state of degradation Something is always broken, right? And depending upon what little part over here is broken versus this part versus this part versus this part Means that while this signal may have looked the same last time Doesn't mean that this thing is broken at the same time that this was broken that this was broken Which has actually caused the incident So how many people flew on an airplane to get here? How many people have flown on an airplane in the last 10 years? Okay, right so fair enough. So let's think about taking off our shoes Right, so in december 22nd of 2001 cat named richard reed He tried to ignite explosives hidden in his shoes on a flight from paris to miami. It did not work It was a bad idea for lots of reasons. It was poorly executed.  So we should probably check for them And then in 2006 the tsa mandated that all passengers must remove their shoes going through security And then you know a few years later said but if you give us some money, then you don't have to take off your shoes So the point being We saw a signal which was wearing shoes on an airplane Meant potentially destructive effect Therefore we're going to watch for all of them again and where this comes into it's kind of a phrase That someone you know said uh, so jennifer brea said, you know We have a saying in medicine that when you hear hoof beats the first thing that should come to mind is a horse Not a zebra and that this two cute by half phrases killed so many zebras So the this at its surface this sounds logical. It's like, okay, let's not look for edge cases Let's say this looks similar to the thing that happened before Inside of tech this is very similar to jumping on that first root cause This is sort of one of the the inherent problems with using the term root cause root cause implies there is one factor that caused the issue and the problem with that is It's rarely one thing and What happens here is when we find that first thing we see and we say that's our root cause Well, that's just thinking that everything was it was a it was a bunch of horses But maybe it was a zebra also. Hey look more zebras. So there's a theme So you want to identify your organization's window of tolerance? So what can we do to help kind of understand what that may be within our organization and how can we identify When we're becoming deregulated So I I put up a quote from dr. Peter Levine So I've got another quote from another very well regarded person in the industry Who has a lot of insight into that? me So my paraphrasing are kind of taking dr. Levine's points and applying it a little bit more To to tech is saying so a resilient organization. They're not traumatized by routine threats to their mission or business Um, I still have business spelled wrong in this slide. I gotta fix that So see and you notice this did not traumatize me. It was okay. I'm moving on Uh, non-resilient organizations are readily overwhelmed and subject to symptoms of overreaction shutdown and lack of Effort so that's many strength. I'm not a doctor. So what does this mean, right? So If your organization is resilient Doesn't mean that things don't go wrong It doesn't mean that you don't have incidents or outages But they don't cause trauma for your organization You're able to withstand them and that's a big kind of key thing about when we think about the idea of resiliency Right resiliency does not necessarily mean stability does not necessarily mean 25 9's it doesn't mean 100 uptime It means that when things not if but when things go wrong, you're able to respond in a practical regulated Normalization of effort and it doesn't cause everything to go parachaped right and so let's think about that So this so first we want to talk about regulation Not referencing war and g I might have a little bit of 90s stuck in my head because we saw Captain marvel last night so But how do humans and organizations take this deregulated state right? We talked about what it looks like to be deregulated How do we become regulated because it's one thing to understand that it's happening We talked about what some of those symptoms were we talked about that you might be hyper aroused or hypo aroused We're seeing that we don't want to move forward. We feel this stuff. That's great. You're saying great maddie We're we're not regulated, but I super want to become so what are so first of all we'll talk about how humans do it How we do it as individuals and then maybe how we can take these same metaphors and apply them to or an organization So one of the treatments for post-traumatic stress is something called eye movement desensitization and reprocessing also known as emdr How many people watch the netflix show russian doll recently? So there was actually a little emdr going on in russian doll. It was very cool. So there's uh the idea So patients difficult memories are offset with a positive association that's reinforced through external stimuli So that can be blinking lights. It can be little tappers in your hands It can be audio stimulus and so there was some of that in russian doll That's not really a spoiler if you haven't seen it, but Yeah, if you do watch russian doll and you see the part with kind of the lights You're like hey, that's that thing that maddie talked about so what's happening there is it's not Well, what we're doing is taking the the patient is going through kind of a safe journey But it's being reinforced by stimuli that's activating both sides of the brain Now we can't really do this For our organization We can't put giant strobe lights on either side of the office when things are happening and have somebody do a walk Through it doesn't quite work Literally the same But we can take the same principles which is taking Making a a positive association with memories of trauma Through some type of muscle memory stimuli And we can do that for our organizations. So here's a couple ways that we can go about doing this So one are the ideas of game days So what we're trying to do with this is we want to make an association of outages and issues with being safe With being okay So game days can help us do this. So a game day is when we're doing intentional Outages and we're exercising our practice of what happens when we have an incident what happens when a system goes down The thing is about game days they can be really helpful But they have to be done properly The idea of a game day is not to like kind of go psych out caught you Right because all we're doing there is creating extra stress for the participants We're trying to keep we want to keep them low stress and safe The point isn't to practice under pressure. It isn't to create a high stress environment So we get good practice on being under high stress. It's exactly the opposite It's to take this association of a thing that can be high stress and associate it with a physical Memory of low stress. We're associating response with a safe environment So when we're doing these explorations in a game day, there has to be guidance, right? This isn't just something we go due to our support team or to our SREs and to say we're just going to go break some Shit and yell at you till you fix it, right? We have to have guidance through this. It's a lot like meditation When we're when we're helping people who have post-traumatic stress through meditation So with some kinds of trauma meditation is really helpful But the problem is people can go into this inner what's called the inner landscape And if they aren't prepared and they're not guided Sooner or later, they're going to encounter The trauma right it's going to be there and Then what do they do? So they could be overwhelmed by it Or they find a way to go around it And similarly so if we don't have plans in place for our game day We're going to maybe get wound around the axle around the wrong bits of our process We're going to be too concerned about the physiological response. We have a stress We're going to start thinking that this is about training us to deal under pressure And the idea is not to make us better under pressure. The idea is to say that when these things happen, they're actually not pressure situations Also, we can in many and when we have meditation for trauma You can run into something that dr. Peter Levine called the bliss bypass Which is a way of avoiding the trauma which is the meditation is so peaceful and so calming That it just makes me kind of forget that the trauma existed, but it doesn't make it go away So if our game day is in following our usual rules of engagement around incidents, we've actually made it too safe Right, so we still need to follow our usual process and procedure And so if we're talking about things like plan failure injection, so if we're doing planned failure injection in a game day Run it like a real incident Here's the next time I talk about pager duty in my talk Not talking about pager duty the product but about how we do things at pager duty So we have these things we call failure fridays.  Because we actually broke it on purpose, right? But we practice This is creating an organizational association as incidents being a safe Normal place and in our case we actually run our failure fridays during business hours because It's actually the best time for us to Do it we don't have maintenance windows pager duty does not have a time when we can say well It's totally fine if people don't get alerted, you know from friday night from you know 8 to 10 p.m So that being said for us the best time to run failure injection is actually during business hours when we have everybody Already in the office on site. We know what we're doing This may be different for your organization You may do planned failure injection if you do and you may not necessarily do it in in production You can also do planned failure injection in a pre-production environment, but however you're doing it Run it like you're running a real incident This is another reason why you should run all incidents at their initial severity, right? So let's say An incident starts out and it comes in as a sub one You may decide later that this wasn't really a sub we may say, you know what this didn't actually Require our major incident response. It was a small thing Run it as if it was a sub one all the way through and the reason for this Is because it gives first of all it does two good things One that has nothing to do with the trauma stuff But I feel like preaching about it is this keeps us from litigating severity during an incident Which is really really important. It keeps us from saying we're going to spend time when we should be restoring service Arguing about whether or not this was a sub one or a sub two The other reason why it's good to do this is we don't want Sev ones to be is it going to sound controversial? We don't want them to be considered a bad thing They're just a thing that happens And if we run it as a sub one, it doesn't mean that we did anything wrong It doesn't mean that we're a bad organization. It doesn't mean that our systems are bad And all that happens is if we had an incident come in that we thought was a high severity to became a lower severity The worst thing that happened is we got to have good practice of our incident response process And it just went back to making us feel like this is business as usual. This is just the thing that we do It's no big deal Now we have to process Failure and this is really really key because post-traumatic stress happens when you have undischarged trauma When you have unprocessed trauma when you haven't been able to work through it So when we talk about processing failure and we're thinking about things like after incident reports and post mortems The ideas around blamelessness are really just the beginning Excuse me We need to process the failure or the outage throughout all the information that we have And and this is key the processing has to have a conclusion. It has to end Otherwise it's unprocessed trauma. We didn't finish So there's this misconception that in order to process trauma All you have to do is just get it out, right? If I can just talk about this enough Then I will be done with my trauma So that's not quite how it works. We need to integrate our experiences into a more coherent whole This includes telling our stories As well as changing our autonomic nervous systems Which in the case of an organization has to do with how we respond to outages and incidents So storytelling Is really really important We have to tell our stories Because that's how we process them That's happened for individuals. So this is part of Therapy for pts has to do with actually processing it But the larger organization needs to be able to tell this I did an episode of a podcast called greater than code and we talked about this particular topic And got into kind of this some some ideas around how Postmortems could really become a storytelling exercise As opposed to just a fill out a form because the form is sitting there in pager duty that we have to fill out And tick the boxes and say yep, I sure as heck did fill out the postmortem form You have to be able to tell the story and they need to become interactive A right only postmortem does not help anybody It has to be experienced So I'm going to talk a little bit about something called somatic experiencing and soma means body All right, so in a classic top-down therapy We start with the emotions Right, why do you feel the way that you do? This is very similar to an enduring it with an incident or an outage looking for root cause first Saying that's the first thing we're going to do is try to figure out what the root causes And the reality is that's kind of one of the last things you want to do is think about contributing factors because we're trying to restore service So in somatic experiencing the body is worked with first This enables a titrated approach to the emotions So we aren't necessarily saying we're going to think about all the emotions all at once We're going to think about the body's response first and that's going to surface Some of the understanding of where the trauma comes from and the letting us process it So similarly during an incident you are first working to restore service before we worry about And it's really important to look at the whole system When we're processing it not just starting with the quote-unquote surface emotions Which are the symptoms of the outage again when we talk back when we think about the symptoms of the outage Those are those signals that may point us down the wrong road that may make us think this is a more traumatic experience than it was Because small incidents and large incidents sometimes have very similar symptoms and signals You're going to spend a little time talking about these ideas of cognitive distortion So cognitive distortions are exactly what their name implies their distortions in our cognition So these are perspectives with bias They're irrational thoughts and beliefs that we unknowingly reinforce over time And some of these can be really really prevalent in how as an organization as a team as a squad how we approach incidents and outages So these patterns and systems of thought are often really subtle It's really difficult to recognize them when they are a regular feature of your day to day thoughts So there's as many as uh 16 generally accepted cognitive distortions I am not going to go through all 16 of them If you feel like looking them up google is a thing We're just going to touch on a few that might be specifically causing issues within your organization So polarized thinking This is also sometimes known as all or nothing Thinking this is seeing everything in extremes, right? So this is it's either perfection or total failure, right? So remember that our systems are always in some state Of failure. We don't have perfection Similarly things are not always completely asked Right something's probably working at any given time So we need to not look at these generalizations of it's either everything is beautiful and amazing or everything is shit Because what will happen is this this particular distortion Is going to affect your ability as an organization in two ways if you're either thinking about All you're going to be chasing a goal that you're never going to hit And you're going to think that you're not succeeding because You are not in this blissful state of a hundred percent uptime Or on the other side if you're thinking that everything is terrible everything things can feel very uh disinheartening very much like things are just awful and why should I even bother And over generalization This is where we take a single instance and we generalize it to be an overall pattern On a personal level. This could be something like well. I got a C on this test.  This can be well. We had a SEV one incident on this particular service So this service is unstable and awful Um when we combine this with another distortion called the mental filter This can manifest as seeing only the negative and eliminating all positives about a situation a person a team or a system Um fortune telling we do this a lot in it Um, we feel like if we have enough data We can be predictive. Hey machine learning, right? So if we only know enough We can predict the future Well, yes, we can start to get ideas on what is likely to happen But guess what? I'm sorry to say. I don't I don't care how much blockchain you throw at it You can't predict the future with certainty. So stop trying And here's the reason why our systems are you know, you're probably tired of hearing me say this our systems are complex We have to understand that our predictions are not Facts They're actually just one of several possible outcomes Does this mean you should not be looking at tools around ai ops and around machine learning and and how we can start to have some predictive analysis By no means these are all helpful. However, they are not Magical spells that will tell you what is exactly going to happen And then we have a control fallacy. So this manifests as one of two beliefs So one we have no control over our lives and we are helpless victims of fate or two We are in complete control of ourselves and our surroundings giving us the responsibility for the feelings and actions of those around us Both beliefs on either end of this spectrum are very damaging and both are equally inaccurate So no one is in complete control over everything that happens to them, right? And no one has absolutely no control over their situation So even in extreme situations when the individual seemingly has no choices in what they do or where they go or what they say There is a some Control over how you approach a situation mentally now as an organization within tech where this comes in is again If I could just keep those damn developers out of my system I could control everything, right? If we only had enough tests, right? It's like I can control all the things or on the other side That's not my fault everybody just f's everything up all the time And we have some somewhere in the middle So let's think about how we can make this better, right? So how many people in here Are currently part of some kind of on-call incident response rotation? And people have to deal with that. How many people have had this experience?  Okay, right. So hopefully be everybody. Okay So we wanted I'm just going to kind of finish up with some thoughts around self-care around things that we can do to make as individuals On-call an incident response less traumatic So why does it matter? Why am I even talking about this? So burnout is a real thing and We at okay, I guess it's the third time. I'm going to mention pagerd. We We did a survey about a year ago across. I think it was about 12 to 18 thousand different incident responders to understand um Who had who had left their position in the last 18 months left their company? We wanted to understand why? and For the vast majority the most important metric related to attrition was not money I mean some of them left because they wanted to make more money most of them didn't It wasn't even their manager, which is generally a true thing I mean sure we've heard the saying people don't quit a company. They quit their manager The most important and the most relevant metric related to attrition Had to do with how often they were paged How many and it wasn't being paged at all. It wasn't like well I left because I want to go somewhere. I never ever get paged But it had to do with how many nights were interrupted how many workdays were interrupted how many weekends were interrupted so this really really matters and It can matter to you as an incident responder because you don't want to get burnt out because you actually care about yourself as a human It could care you could matter to you as A leader in your organization because you care about the people who work for you Or you may be a leader in an organization who doesn't really care about the people who work for you But understand that it costs on average $400,000 to replace an engineer And it's a lot cheaper to keep people than to have to replace them. So let's try to not burn them out So a couple things that we can do When it comes to self care around on-call and incident response So one is doing some context switching Create a ritual around going on and off of call This helps your body understand that things are changing. So, uh, for example, like when I take out my contacts, I get sleepy And this is because over the last worth 20 something years or whatever my Body has associated it with when matty takes out his contacts It means it's time to go to bed and likewise when I put my contacts in I get a little bit more awake because that's the thing I do in the morning So my body has some expectations So there's things that you can do if you say like when I go on call I physically make this shift and I uh Was in an open space I think it was at devops days charlotte and someone came up with the idea and said like Maybe you have an on-call hoodie This is the hoodie that I wear when I'm on call And then I take it off and I never wear it if I'm not on call And of course I went and like oh that would be some good swag Right could have the on-call hoodie You could have things that have to do with like when I go off call I treat myself to this one particular tea now the trick is whatever these rituals are They need to only happen when you're going on or off call because you're creating this physiological response You need to allocate mental bandwidth So if you're on call, that's really has to be your number one priority Don't be trying to deliver high priority projects at the same time that you are the level one first Responder for instance for your organizations Having a good relationship with your manager is really key to this You have to work with her to help understand where you're coming from With this and help set expectations, which might say hey, I'm going to be on call this week Which doesn't mean I'm not doing any other work, but There's only so much mental bandwidth I have so I need to be able to shift and maybe I'm going to have more of a focus on Resolution and remediation type projects during this time than innovation type projects because I'm context switching treat yourself Have an awareness That on call is a stressful activity no matter how much amazing stuff you might do With your organizational EMDR and all the things I've talked about over the last 40 minutes It's always going to be stressful. I went I you know, I spent 20 years and various forms of system administration and Technical operations and most of those two decades I was on call almost all the time and then I went and worked for a software vendor for a few years and Blitzfully hung up my pager and then I joined pager v. Okay four times pager duty and now I am part of an on-call rotation and It was really stressful the first time Every time I go into my one little 24 rotation every two weeks of being innocent commander Just being on call is stressful to me so Knowing this knowing that on call is a stressful activity take some extra time to treat yourself during these times to counter that stress So I talked to Anna Medina from gremlin And she had some good suggestions and one that I really liked she said the weeks when she's on call She schedules wine or dinner with her girlfriends or a plan to kind of go out for her favorite dinner Because she knows that it's going to be a really rough week But what happens is it actually gives her something to look forward to so this is my on-call week And I know it means I'm going to go have my wine time now again It depends upon you know Maybe you can't go out and have wine with your friends while you're on call because maybe Your expectation is you can be on a terminal within five minutes But find something you can do or even if that's the case if you say hey I'm going to be on call for a week during one of those nights I'm going to take it I'm going to take a couple hours and go out with my friend Sonia And you know what I'm going to go to someone say can you take an override? Can you cover me for those two hours because I have to go do this thing Stay healthy So it's very tempting and common to fill ourselves with sugar and caffeine while we're on call because We're probably going to be up in the middle of the night, you know And it's