Hello, hello, hello. All right, that's it, you're done? Yeah, it was easy. Just on now. Testing, testing, testing, testing. All right, I guess we're right at it. So I guess we can probably kick this off. Can everybody hear me okay? Anything good? All right. Well, my name is Derek Morgan. I am with SpaceLift. We are an infrastructure as code management company. And it looks like that HDMI might be a little finicky. Just let me know if that disappears and I'll make sure we keep that going. Probably going to let you guys get the launch pretty quick. So do my best to help this go pretty quick. Basically, what we're going to talk about is managing infrastructure as code and kind of some of the things that you may not have thought about outside of running Terraform apply just from a command line. So obviously, a lot that goes into it once you start actually dealing with real scale and a lot of different developers, especially collaboration can be a huge pain. So let's take a quick look. My name is Derek. I started with a lot of on-prem stuff, worked in managed hosting, blah, blah, blah. I've dealt with everything from exchange, dealt with Ansible, puppet chef, all of that. I've created some online courses. Don't know if any of you have seen them, but got to recommend them. And basically, as most of you probably know, I assume that's why you're here. You know that infrastructure as code has basically revolutionized the industry. It has absolutely exploded. ClickOps, of course, is fun to do your first time through AWS. It's wonderful, but at the end of the day, it's absolutely not scalable. It's possible to maintain Drift and your state and everything else. Let's talk a little bit about those limitations. Now, today, we're going to mostly talk about Terraform since it is the most popular. Cloud formation, of course, is huge. Pulumi is big, but Terraform really is the elephant in the room. So resource management and cost, of course, is one of the largest issues, especially at scale. TF state management, since it does store pretty much everything that's important to your industry, secrets, everything else. That's also very difficult. Now, if you've got a lot of devs and they're all working from their local machines, you can have all sorts of issues. As we might have remembered a few years ago, when AWS S3 actually went down for several hours, it was a configuration error, just one wrong command, basically brought down the internet. So it's not hard to do that, especially when you're letting everybody just kind of go cowboy on the CLI. It's kind of a big deal. This is just a little example. I just grabbed from one of our blogs. Basically, infrastructure as code starts to get really complicated, what you do with a lot of stacks, start doing a lot of state files, a lot of devs battling each other, trying to deploy different things, have different dependencies, et cetera, et cetera. So, as we kind of mentioned, client compliance can get kind of crazy with security. You've got different teams trying to work on different things, and you've got to manage how those teams talk to each other. You got to ensure the security works, all of that. You know, kind of cover that in all these steps here. And then, of course, at the end, you've got cost. So I've got a lot of really cool tools that you can use. A lot of open source stuff out there. SpaceLift actually utilizes some of those open source ones if you'd like. But of course, SpaceLift does work pretty well. Not going to lie. I like it a lot. But really, the big issues you've got automation is, of course, huge. Just running from the CLI is not going to work when you've got hundreds of thousands of lines to terraform code. Policy as code is very important, ensuring that everything you run is managed properly and enforced and doesn't break any of the rules you have set. Collaboration. Devs working over each other. Got a lot of pull requests, merge issues, conflicts, that type of stuff becomes quite a nightmare. Resource management and drift. Again, you get that one dev that decides to hop into the console and start poking around.  Then, of course, as I've mentioned before, cost management. As we all know, that is a very, very, very big concern for most people. So some of the tools that you may or may not have used. GitHub, GitLab, SpaceLift, of course, I love. Atlantis, Jenkins. Those are really the heavy hitters in the industry. Jenkins is great. It's a little bit of a sledgehammer at times when maybe if you want to scalpel. Of course, you can do a lot of damage with Jenkins if you'd like, but it can also be used very well. Now, you've got policies code. Some of those tools. Open Policy Agent is great. TFSec is great. Sentinel, of course, is the Terraform or the HashiCorp managed policy as code. It gets a little expensive if you start using that. Then, of course, you've got collaboration and accountability issues. As you can see, there are some wonderful AWS keys. I'm sure at least one of you in here has seen that show up in a repository before. I know I have. Luckily, AWS let me off with a warning, but we don't have to go into that. Policy is code. It's pretty great to keep that type of stuff from happening. Again, accountability, that code history, maintaining all the pushes to your repositories, understanding who did what and how. Resource management. You can obviously use a lot of AWS tools, GCP tools, Azure, whatnot, to manage your resources, understand what's going on. You can analyze your state files using tools such as Drift CTL and Cloud Query that will allow you to see what has changed and what's caused it to change and even who made that change, which, of course, is very important. Then we've got cost. Infra cost and Terraform cost estimation are two of the main ones that you can find out there. Infra cost, of course, if you've used it, is phenomenal for taking a look at what's going to change and how much that is going to cost. Anyway, that's the slides. I'm not really a big slide guy, to be honest. We're going to hop in and we're going to do a little demo. Again, I'm going to use Spacelift, of course, because that's what's on my shirt, but we are going to poke around with Infra cost and a few others. These tools are ones that you can use. You can actually bolt together your own Spacelift if you want to. I do recommend using us, but again, this will kind of show you some of the ways that people do manage their infrastructure. Let's take a quick look. I've actually built a fun, if it opens, there we go. I actually built some Terraform code. It's not a whole lot here. We're basically just deploying an EC2 instance. Again, this is an exactly giant scale operation, but didn't really have the budget to launch thousands of instances here for a conference. I think I can. Yeah, let me take a look. That's my settings. Yeah, that's what I'm going to do. I was just checking to make sure that I was in the right resolution. Good. Perfect. All right. Again, got some code here. What I'm going to do is just push that over to a repository that I've set up for that. We're going to use Spacelift. We've got a stack here that's set up. Let me make that a little larger as well. What the stack is, is the stack is going to connect to that repository and it's going to manage my state for me, which again, one of the most important things that you can do is manage your state properly. S3 buckets are great, but of course, as we all know, people do not like to put the right permissions on S3 buckets and stuff gets leaked all the time. Let's go ahead and take a look. Let's just say I am a dev. I'm in my dev branch. We'll just change a thing. Let's just change the subnet here. I think my mouse is freezing. So one second here. Not something I expected to go wrong. So I guess that's what I get for using them back, huh? There we go. Okay. We are going again. Hey, Pavel, where do we get these laptops from? Yeah, it must just be with, I've been using this actually all day, so it must have something to do with the display or I just won't edit anything and just do an empty commit if I have to. It's not a really big deal. I'm sorry. Yeah. Yeah, I've actually, this is the first time I've used this laptop for a presentation and have never experienced this. So here we go. It's all part of the fun, right? Exactly. I want to say the rest of that comment right now, but I won't. All right. We got something. I'll just do an empty commit. I won't even bother with it. Keyboard froze up. That sounds like a good name. All right. So I'll head back over here. So if I head to my repository, I've got my branches here. I can actually open a pull request if I'd like. And as we can see, of course, everybody knows how Git works, I assume. I've got our pull request right there. Create that pull request. And right now what's happening is Spacelift. We'll check that out. We can actually go and see RPRs here, which is great. As you can see, it runs a plan, runs in for cost. I can see exactly what everything's going to cost right here in my output. So again, in for cost is something that is just absolutely phenomenally used whenever you're dealing with multiple devs trying to push multiple things. And I would take a look at my checks. We've got a cool little infracost table right there ready to go. So we can see exactly how much the stack is going to cost once I merge it in. So everything, as you can see, looks good. The plan was successful. I've actually got a plan policy right here that was written in Rego, which, what I hear is not something everybody likes, but I'm pretty adept at it at this point. So if we try to make anything too expensive, over 100 bucks, basically it's going to bomb the plan. Right now everything works out well, though. I can head back over to my stack and we can merge that request, confirm, and that puts it in the main and starts our run. There we go. Infracost is checked again. It will check against policies again, everything once again to make sure that it works. It shouldn't take too long here. Again, just a little EC2 instance should kick it off pretty quick. Perfect gives me an opportunity to confirm, make sure that everything's going to go well. You can also add comments on this if you'd like, confirm that. You can actually see a history of the runs, which again, that comes to that accountability thing. We can actually see who is running everything. We know who to yell at. If something goes wrong, something gets too expensive, etc. Now if you're wondering how I'm authenticating AWS, I've actually got a portable environment variable file here. Everything is of course encrypted, not just hashed, which is great. We'll go back over here to our run. Everything is applying just fine. While that's applying, we've got another dev who's going to hop in there and start causing some problems. Let's go ahead and maybe we'll get lucky here. I just need this one file. Come on, just one file. I was thinking about it, but we're so close. We're so close. I hope we get lucky here. Let's go ahead and just kick out a 2x large and see what happens with that. Almost done here. There it is. We've got a nice little EC2 instance. Everything is ready to go. Then we've got the cowboy dev. It is trying to deploy a 2x large, which again is against policy. Save that. Yep, everything finished perfectly fine. All good to go. We'll head back over and we'll create another pull request. Make a little comment right there. Create pull request. Let's take a look at it. DRs and exactly what I was looking for for once, a failure that I tried to include in this demo. So perfect. As you can see, our policy said, you know what, that was way too expensive. That is not going to work out at all. Right over to the checks, as you can see right there, denied by the plan policy, which is perfect. So again, something that is very, very important when you're running things at scale, ensuring that your developers are not going to start blowing everything up by themselves while again running cowboy. Yeah, you can actually put the comments on the PR request, everything else, close it, whatever you need to do. Rego, R-E-G-O. It's an open policy agent. That's how we do it. You know, I'm not sure. A company is called Styra actually, I believe does it. It's open source. Yeah, it's open source. Yep. So we can nuke that pull request. Oops, can't spill. So we can head back playing by the rules again. Creates that pull request. Leave that comment. Then we'll head back into space lift. And as you can see now, everything works perfectly. Everything is fine. So merge it. And of course, we haven't actually changed anything. So it'll merge in just fine. And we've got our new run prepared. And everything should kick off just like it needs to. No changes, exactly like we expected. So I'm not going to lie, actually kind of a short presentation today. I'm going to open up for questions. As some of you may have noticed in the brochure, I am still actually listed as Sean O'Dell. So this was kind of a last minute change. I want to apologize for not filling an entire hour here. But happy to answer any questions you have. Talk to Terraform. Go ahead. So I mean, it depends, of course, they've got to authenticate. One of the great things about like space lift is you can, or again, Atlantis and a lot of the others, is you can actually manage who has access to that state. You can manage, like for instance, these contexts. You can ensure your developers can't create context or access these keys in any way. So you would definitely want to make sure that that developer didn't have any access from their console at all. They would push to get. And that's really all they would be allowed to do. Yeah. So basically we connect to your GitHub Bitbucket GitLab repositories. And we do these things that we just showed you. We're basically running runners behind the scenes, which you saw, which will run Terraform. Actually, a little hush-hush. We've got Ansible on the way. We can run Kube CTL. We can do a lot of other things as well. Pulumi, CloudFormation, whatever. You can also set up your own custom runners to run kind of whatever you need to. But again, 10,000 feet, we're basically running and managing your infrastructure as code, allowing you to create policies around that, et cetera. I'm sorry. Oh, sure. We can set up a new stack if you'd like. So basically we have a concept called stacks. And each stack is connected to a state file. So create a new stack. Hop in. We're authenticated to GitHub here. So I can connect to any one of these repositories. Then choose your branch. And then you can also, of course, choose your project route. So if you've got like a Monorepo, or you've got a lot of different folders in one repository, you're able to do that. Then for the backend, you can choose... Y'all won't see Ansible right now, but we've got Ansible. We've got Kubernetes, CloudFormation, Pulumi, and Terraform, primarily. You can also run Terragrunt. You can mess with your versioning here. If you've got an old version of Terraform, hopefully you're not at .12, but we do support it. You can choose to manage a state here. You can connect to a state file in S3 or wherever you'd like to store your state, but it is very convenient, of course, that we do manage that state. It's probably S3. Yeah, everything's basically managed... Our entire stack's AWS, right? Yeah. And so the great thing about it, too, is you can define an administrative stack, which actually allows you to create more space with resources. We have a full-fledged Terraform provider that can do everything you see right here, just within Terraform. You can also set more advanced options, and you can customize. And this is how we got that infra cost going. So we actually added the infra cost API and everything that we needed and applied that during the planning space. So you can add different hooks you need. If you need to reach out to a vault or something like that, you can do all of that here with these custom hooks, and then basically just name it. We do have drift detection enabled. Unfortunately, I didn't have a chance to get that set up with this stack, but we do have drift detection, and you can set... Basically, you set up a cron job. Within it, you use cron expressions to determine how often you want to run, and what it does is it runs a plan against your stack, and then you can set it to auto-remediate if you'd like. How much do we have cloud formation drift? It has drift detection. It doesn't have the remediation, if I remember correctly. Right, but it does detect for sure. So yes, we do have that in as well. Yes. So can you chain multiple stacks together? Sorry, I keep forgetting to repeat your questions. Yes, you can. You can run trigger policies. One of the main things we're kind of seeing again with this Ansible rollout is a lot of people like to perform configuration management after they've run a Terraform stack. So you can set it up to automatically trigger another stack as soon as Terraform is done. It will, for instance, trigger Ansible to access that stack and configure whatever you see to instance as you've launched. Sure. Let's take a look. Yeah. So basically right here, walking through this rego policy, we've got our deny rule first. We've got a threshold set here, pretty straightforward. So what it does is it accesses the plan and it parses that plan. Threshold here is set at 100 monthly costs set right here. And what it does is it allows you to access that plan, which you can do in our rego builder. We actually have a rego builder that allows you to dive in and access an existing plan and test different policies, which again, is very helpful. Got to deny there. And then we also have a warn, which has a much lower threshold there. As you can see, if you get a warn, you'll pop up a warning and a lot of comment. You'll see that it happened, but you can of course push it through if you need to. We've got this sample down here. And what that sample is, is does it allow you to take a look at it after the fact? And let's go and take a look. All right. So we've got the plan policy here. That sample flag basically allows you to hop in and start analyzing it based on the input stack here. Now we do sanitize most of the attributes that come out of a plan. But we have a function but we have a function that you can use a sanitized function that will allow you to access these attributes whenever you need to. But you can kind of cut through this entire state here and you can play with the different attributes, everything else, and you can hit simulate and it'll tell you whether it passes or fails. Any other questions? Well, you want lunch? I think my team, they said they'd buy the whole room lunch. Is that right? Depends. Depends on which caviar you want. I hear Russian is kind of tough to get nowadays but we'll probably find some. Yeah, anything? All right. I think we're all set. Thanks. Can we get them to do more? Instead we should. Test. Yes, this is the session you want to see. It is. This will be awesome. I swear. Everyone in here is really excited about it. See, now I've kind of forced him in because he's like, oh, now I have to commit. See, my evil strategy is working already. Oh, see, there's someone else wandered by the room. You can come on in. There's plenty of room. Save the seat for you. I see familiar faces from my last talk so I must have not scared everyone away. In the last one, that's always good. It's a good feeling when you don't scare people away. All right. Well, we can get started, everyone. Thank you for coming. I appreciate it. This is going to be an unusual talk. This is my public service announcement. I've given this talk once before. It's an unusual talk because I'm not actually talking about fixing anything. I'm not talking about databases specifically. Oh, you want to come in? Yes. We have a seat just for you. See, it's just for you. We saved all these seats. Yes. See, now he's committed. It's like I said. See, it works. So, this is a bit of an unusual talk because this isn't really about any one specific topic, but it's a collection of stories and there are things that we see in the real-life space that really have some commonality with things that we do in the tech space. So, for those of you who haven't heard me talk yet, I'm Matt Yakovet, head of open-source strategy at Percona. So, we do open-source database software and services. If you're interested, you can follow me on Twitter. You can check out the podcast, the Haas. That's me, head of open-source strategy talks, Faas, free and open-source software. It rhymes and everything. That's cool. So, that's awesome. So, you should check that out. So, I wanted to start to tell you a little story about something that happened to me this week. So, I have a 20-year-old daughter who's soon to be 21. She works at a veterinary clinic. And I dropped her off last Saturday, like I normally do. And so, about an hour after I drop her off, I get this text. Okay? So, I don't know how many of you have kids, but when you get this type of text, you generally don't know how to respond initially, right? So, hey, can somebody get me? I might need to go to the urgent care. This could be something that is small. It could be something that's big. So, naturally, you don't know how to respond. But when you say, I might need to go, there's some doubt there, right? So, it doesn't really have the sense of urgency. So, okay, let's investigate further. Of course, we investigate a little more. What's wrong? Oh, a dog bit me pretty good. And it did break the skin. Now, my daughter gets bitten probably three times a week by dogs, generally not that bad, but she does get bit in her line of work. Okay. So, I'm on my way. So, it doesn't sound terribly urgent, but, you know, you do have some concern. And so, me and the wife pile in the car.  What do you think that we saw at the vet clinic? Anybody have any guesses? Well, I'll tell you what we saw. So, we saw animal control, because, hey, it's a dog bite. Okay, that makes sense. But what didn't make sense is the two police cars, or the ambulance pulling the stretcher out of the back, or the fire truck. These are all things that you don't want to see after you get that text, right?  So, of course, that heart attack pain, you're like, oh my gosh, I thought it was the cheeseburger's gonna kill me, but no, it's this, because it's so stressful.   This isn't for her. It's for the other person who got bit. So, wait, oh, there's two people who got bit now. Yes, there's two people who got bit. Okay, okay. Well, yeah, so the other one's worse.   The dog's name is Chaos. It really is Chaos, you know, and so she loves Chaos, and she doesn't want anything bad to happen to him.  Has anybody else had this happen? Well, maybe not this exact thing, but you do, right? So, as we've written software, as we've designed new applications, how many people have written kind of sloppy error messages or alerts in their pipelines, or have put out things that really don't, you know, put off the sense of urgency on why you do things, right?  Oh, see, you've had that happen, but that's where you have this type of thing happen, so that initial text, that initial, you know, response, you know, it doesn't give the sense of urgency, and so, you know, honestly, I wish I could reprogram my daughter to say like, hey, if you need to go to the hospital, it should be like, help take me to the hospital now, you know, or something like that, because that would send the sense of urgency as opposed to, you know, like, come on, you know, I think I might need to go, I don't know, so we want to see, you know, that type of thing as we're designing things and building out applications. Now, the fun thing is, that's an actual example that just happened, so that was added this week. You are the second crowd to see the rest of these slides, so thank you for beta testing, but there are awesome good and bad practices for all kinds of design principles all throughout the world. You just have to know what to look at it, and maybe I'm just a weirdo when I look at things and go like, wow, yeah, I designed something like this, and this is horrible, or wow, I should use this example in my, you know, real world work. So a lot of these examples, you know, come from people who are just trying to copy other things. A lot of us copy other tech companies, but we're not comfortable copying non-tech companies, right? How many people go to sessions and listen to Amazon or Meta or Google or Microsoft, they'll tell you how to like develop something cool, and you'll be like, okay, I'm going to go try that, right? You do that, but you never go to like McDonald's and say, I'm going to try to do this like McDonald's does, but you could, and I'm going to tell you why you might want to in a second. So let's talk about, you know, a few different things, and these are going to be rather random kind of conversations, so it's going to, you know, jut around a little bit, but I want to start with talking about the responsible selection of tools and components when you're looking at building out a stack, okay? And so where do I start with that? I start with the children because they are our future. Now, how many people here have kids? No judgment if you do or don't. I don't like to admit that I do most of the time. Maybe if you don't have kids, you also can commiserate with this because kids are annoying, aren't they? There's no kids in the crowd so I can say that. Yes, most children have times where they are awesome, but a lot of times where they just drive you absolutely nuts.  Does anybody know? We distract the children, right, with TV shows and with awesome things. We try to make them see the shiny objects to let us go do whatever, okay? This is a secret for those who don't have kids. When you do have kids, you will totally understand this. Now, for me and, you know, growing up, I was very different from my daughter so I had the cool cartoons, you know, like so I got the Transformers and the Thundercats and GI Joes. Unfortunately, my daughter grew up with teletubbies and Barney's which was just more annoying than not. But, you know, the thing with these shows is they really feed into that, you know, kids' imaginations. They really feed into what the kids are looking for and, you know, it turns kids a little bit into lemmings, right? You know, where they're going to follow that trend, whatever their cool kids are doing, whatever the trends are happening, whatever fads are going.   Now, some of them do become iconic, right? So I can actually sit down and stomach SpongeBob even today. I can admit that. I have watched SpongeBob without my daughter. That does not make me a bad person, okay? It doesn't make you a bad person either. But when we talk about choosing tools, choosing libraries, choosing frameworks, it's a similar situation where, you know, we all love cool things. Who does not love to listen to a session on new technology and something awesome that they've never heard before? And who wants to go out and try it in production, right? I mean, I think we all do. We want to see what happens, you know, morbid curiosity, if you will, right? And so, you know, here's the fun thing about that, though, is number one, we really have short attention spans. So I don't know about you, but me, I start things and I'll start to learn things and I'll get halfway through it and then I'll be like, okay, time to go to learn something else, right? If it didn't peak enough interest, I'm gone, okay? The problem is, a lot of people do that in the middle of projects. So they start with one framework, they start with one project and they go to the next, right? And so you end up with this weird sprawl, this tapestry of uniqueness in an environment that is just strange, right? And what's fun about that is a lot of these, you know, are really hard to hire for.  How many Haskell programmers out there, you know? You've got a few Haskell program. It's hard to find Haskell programmers, right? I'm not saying Haskell's bad, it's just, it's hard to hire for when, you know, new technologies, new trends come along. But these development tools, they come, they go, and how do you handle these, right? And, you know, it's that, you know, herd mentality where we want to follow other cool people who are doing awesome things, you know? And, you know, not all trends though are going to last.  Old school cool is a thing. So you can choose technologies that are tried and true and still be cool. You know, that's something that's, you know, important to realize. I give you permission. But remember, those trendy things do come and go. I'm old, so I remember, you know, everybody was all, Adobe Flash. Nobody even, like, mocks me for that anymore. Hardware appliances, the network is the desktop. Remember the, you know, Google Glasses, we're going to replace all everybody's phones, right? Like, all of these trends happen and they go away, right? So these are things that are happening on a regular basis. So just be mindful. When you are choosing your technologies, is this going to be Barney the dinosaur or SpongeBob? Okay? Because it is a critical distinction. It really is. Now, when you do choose those components, okay? We are all here at a conference that loves to celebrate open source. Most of the stuff here is open source. But my challenge to you is as we choose these components are which version of open source are you choosing? Now, I had the fortunate or unfortunate circumstance where I was in Vegas for four weeks last year, or maybe it's two years ago. It was like conference, conference, conference. It was just pre-COVID. So it was a couple of years ago. And so it was like, after four weeks of Vegas, your brain starts to do weird things, okay? I don't know if anybody has had their Vegas brain just do weird things. But of course, when we talk about open source, I, you know, was thinking about this as I was wandering the Vegas Strip and as I was trying to find someplace to eat. And of course, what is Vegas known for? The buffets, right? So many buffets, so little time. And after four weeks, you don't want any more buffets. Like, you're just, please, no, I don't want any more. But, you know, you have so many different choices of buffets that are out there, right?  Other ones, you get the free continental breakfast. But if you want the eggs, then you got to pay extra. The other one is, you know, you have the free, you know, with the minimum drink purchase, or buy three buffets, get one free. You could pay per pound. There's all kinds of different ways that these buffets advertise. But all the buffets eventually do have a cost. And why this struck me is so weird is this is very similar to the open source, you know, ecosystem right now, right?  But then you've got open core, which is that continental breakfast, but you pay for the hot bar, right? And then SSPL, which is you're open with restrictions. So when you have a company like Mongo where elastic, where it's like, you know, hey, you can use our stuff for free if you buy for something over here, or you follow these specific restrictions, exit the gift store on the exit, right? Or exit through the gift store. You know, the source available license, which allows you to look at the buffet before you purchase, or the eventually open buffet, which is you buy three buffets, and then you will eventually get one for free, or the freemium sass cloud, which is you pay per pound, so that's the cloud model, right? So you pay per pound of food, you pay per instance, you pay per instance hour. And then of course, the open source compatible, which is saying, hey, you know that buffet across the street, we're just as good as them, but you should pay us, right? So they are all over there. And we have so many choices, okay, more choices than ever. And just like there are some weird fusion food choices out there, we have that with the buffet option of licenses as well.   So important to not follow all the trends. But also if you do pick on technologies, and you do choose components, you should make sure you're choosing the components that are fitting to your buffet lifestyle. Okay. Now, now that we've picked some components, and we're talking about building an application, we're talking about building something, let's talk about success criteria. Now, how many people before they've designed anything think success, what does success look like, or what's my requirements, or what's the outcome look like, right? Some people do, some people don't, right? But how about when you build a house? What do you expect when you build a house? Okay, you probably have some expectations, right, that's reasonable, right, that it doesn't fall down, that you can live in it, that you can get electricity, that you can, you know, use the plumbing, whatever. And so this is a house that I had purchased a few years back. And brand new house, builder's spec, beautiful home, had all of the appropriate number of fixtures, and everything else. And so we moved in. And when we moved in, everything was perfect for the first couple of years. About two years in, we started to notice in the bathroom, one of the floors started to buckle a little bit. And we're like, huh, that's weird. So we called up a plumber and the plumber said, oh, by the way, your bathroom has a leak underneath the floor because they forgot to put an O ring on the toilet. Okay, so it worked, and it worked okay for two years, but in the end, it didn't work, right? Because let's be honest, when we say, ship it, it works, it's not a good thing, right? So yes, the toilet flushed, yes, the plumbing worked, but it's not enough to just say that it works. And that's where there is a completely fine line when we think about success criteria, right? Where is that success? What does it look like? What are we looking for? And it works is never something anyone strives for. It's something we all compromise to, right?   And it works is great, but you want it to be at least it works and it's good enough to be sustainable, right? That's an important distinction. You want it to not fall apart within a year or two. You want it to not just be that, you know, ugly mess of an issue that's sitting underneath the floorboards, getting all gross and ready for someone to come and rip it all out and start over again. Okay. And even if you are focused on good enough, okay, I want to leave you with this thought here on this point, which is, you know, while an idea can inspire you a good enough project generally isn't going to be something that's going to lead to commercially viable success either. So if this is part of somebody else's infrastructure, maybe good enough is okay. But if you're building your own product, this is your lifeblood of your revenue stream, it's something that you're going to want to make sure that it's better than good enough. Okay. The internet is littered with projects that are good enough. If you're gonna get hub, there's a lot of projects out there.  So keep that in mind as well. Now that we've decided on buildings and foundations, we decided on success criteria. What comes next? Well, there's a lot of different components, but I like to talk about concurrency and scale. Ah, yes, concurrency and scale. So when you are building a system that can handle millions of users or billions of users, who should you look to for an example? You might say Amazon, you might say Meta. I'm going to tell you to look at the fast food industry. Yes, because guess what McDonald's does serve billions and billions, but I'm going to talk to you specifically about Chick-fil-A. Now, how many people here have been to Chick-fil-A? Oh, come on. We all know that you love Chick-fil-A. So I gave this an Amsterdam and no one's like, what's Chick-fil-A? You don't know what Chick-fil-A is. Oh my gosh. So think about Chick-fil-A. Let's start with how they design their menus. They are simple, uncomplicated, right? They have chicken. You can't order a burger. You can't order a hot dog. You can't order pizza.  They have a few common components and they make those components reusable across all of their different items. And so that means that they can get an efficiency of scale. So in the back, they know how to make a chicken sandwich. They know how to put the chicken chunks in the box, right? There's not that many components, so they have optimized for that. But where their truly unique genius comes into play is in the drive-thru, okay? Now, how many people have been to the Chick-fil-A drive-thru? Okay. How many people have seen the Chick-fil-A drive-thru like this? Yes. Oh, so you don't go, right? Because most people will turn around. In fact, some Chick-fil-A drive-thru's look like this, right? Because they are popular. Now, here's the fun thing. Most people look at a drive-thru like this and they're like, nope, nope, I'm going somewhere else. I'm going to McDonald's. But Chick-fil-A has this weird thing. They have designed their queuing system, their drive-thru system to solve scale problems. So if you see a line like that, you're going to get out in 10 minutes or less. And that's weird because it doesn't matter how long that line is.  It's going to be 10 minutes or less. And they have this really unconventional practice where they'll send out workers into the lines and they'll take your order and then they'll run your orders out to your car before you even get to the drive-thru window, before you get to the order taking spot. So they're trying to optimize the entire process. In fact, excuse my three-year-old drawing. This is my three-year-old self trying to draw what a Chick-fil-A drive-thru looks like. But they do multiple lanes. They have like a fast lane for call ahead orders. They have mobile order takers where they'll run out and then you could park over here and then you can have them run out your food. So I mean, they have optimized this process for those long queues to be able to scale. And in fact, this has been so successful. I don't know if anybody saw this, but the governments, city government, state government started to ask Chick-fil-A to come optimize their COVID testing lines, right? I don't know if anybody had to do those mobile COVID testing where you're just like in the line for like 12 hours. They're like, oh my god, this is a mess. You know who we need to call? Chick-fil-A. That's right, Chick-fil-A because they know what to do. But when we look at this example, they tell us a lot about things we should do. Whoa, that is just weird. Look at that thing go fast. Okay, yes, Poltergeist in the system. So what can we learn from this, right? So we can learn a lot from the system on the system side. Things like, hey, we should pre-validate our requests, right? So they're running out and they're ordering from the lines ahead of time before you actually get to the place where you're supposed to order. They're pre-fetching your food and running it out to you. Just like we should be pre-validating our data needs or what we need, we should be doing as much as possible. They speed the delivery of data, right? We need to do that. Segregate different types of requests when we're building the application, right? Don't have everything go through a single channel. If you need something that is immediately time bound, have a queue for that. Have another queue for something else that's slower. Have direct access if you need it so you can bypass some of the overhead. Know when things are slow. You can always make changes. So have the flexibility to make those changes. Because the thing about that queue with Chick-fil-A, they shut down lanes when they're not busy. They don't send workers out. So they know as soon as you reach a certain level, they span up, they shrink down. And it sounds a little like some of our architectures now, right? When we're talking about the cloud native space. So you can learn a lot by looking at how some of the real world examples like Chick-fil-A build their systems. Because there is a mirror on how we should be building systems to handle the same sort of traffic and workflow. But we all know that even if we're building it, we're all moving faster than we ever wanted to. I mean, I know no one ever says, oh, just take another three or four weeks to get that deadline done. We're good with that, right? They're always like, we need it sooner rather than later. And so we have this move fast requirement. And that often has unintended consequences. So look at the growth of industrialism. Yes, I'm working a lot of different references into this talk. Impressive, right? So when we look in the 1800s, and we look even into the last century, the amount of progress across the world has been insane. And it's all been driven by industry. It's been driven by factories. It's been driven by new innovations. But one of the consequences that happened, and it really came to light in the 1970s, especially here in this US, was the pollution that a lot of this was causing, right? So we had rampant pollution. Now anybody here from Cleveland, anybody here ever go through Cleveland? Okay, you know, have you ever heard the term the mistake by the lake? Okay, well, they used to call Cleveland the mistake by the lake. The Cuyahoga river in the 1960s and 70s looked something like this. Okay?  So what happened with this is they've had so much industrial work. All those are factories down here. They call this the factory district along this edge. So all these factories dumped all their pollution into the water.  And one of the problems was with fires. But it was fires on the river, right? Not once, but a dozen times. Okay?  So it's that unintended consequences, right?  Okay? So we have that, but I like to take this PSA real quick. The law of unintended consequences. Anybody working on machine learning AI stuff? Okay, a few. Yeah, please don't make Skynet. Thank you very much. Okay, now, how do we go through and clean the rivers? So there was not only pollution and garbage in the rivers. There was also a whole bunch of invasive plants that started to, you know, show up. And so a lot of enterprising or really smart scientists came along with this idea. You know, we should use something natural to clean up the waterways, as opposed to trying to do something that could make things worse. So you know what's really good at getting rid of garbage in the yard? Right? They eat everything. Yes. So we know that goats could be lawn mowers and they can also be, you know, effective garbage cleaners. So how do we do this for the water? Anybody have a guess? It's carp into the carbs. Yes, yes.  But you know, these really smart people didn't actually see one potential big problem. That is, carps like to eat. And they don't stop eating. And they're not limited to eating the stuff that you want them to eat. They will eat everything. And so we actually have a pretty bad issue right now, where we've had invasive carp that are killing off all of the native fish. They're destroying the ecosystems.  Actually, some people say it's worse than the pollution problem. Okay? And so how do you deal with this? That's a question we haven't solved yet. But this is really an interesting case study and, you know, thinking about tech debt and tech regret. Right? Because we often have solutions out here that we're like, okay, we need to solve this problem. This is a problem that is real. How do we fix it? And so we go out there and we find the new widget, we find the new software, we find the new library, we find some new way of doing things. And we decide, ah, we're going to introduce this. Without thinking, could this make our systems measurably worse? Right? And so we'll introduce that into the ecosystem. And a lot of times, it just doesn't work right. It causes more problems. And there is, you know, a process you should go through before trying to solve these problems. It's think about what these, you know, new solutions, what you're going to try and do. What's the outcome going to be? What are those potential unintended consequences? And what you're introducing, if you're doing another integration, you're introducing some new technology, some new thing, you know, has it been vetted? Who else is using it? You know, it won't always work for you. How do you control this? And there is a difference between tech regret and debt. Okay? Now, anybody here understand the difference? I can explain.  Tech regret is where I just feel that I can do better. And like I'm disappointed in the work that I have done. Okay? So there is a difference between the two. You know, and tech regret is that process of thinking like, you know, oh, I wrote that code when I was, you know, 10 years ago and my God, if I knew what I knew today, I would write it differently. And you know what? I'm going to redo it. It's not always a good thing because sometimes you can't, you know, capture that magic again. Like I love movies. Okay? I do. I don't know if anybody else loves movies. I love movies. But you know what? Eh, sequels to most movies aren't that great. I don't know. Teen Wolf 2, anybody? New, Jaws 3D, new, you know. It's just not great. But there are some ones that are better. And so the question is if you are looking to revise or fix things, is it because you are regretting what you built and you think you could do it better? Or is it because you actually have an issue you're trying to solve? Because those are two very distinctly different things. I mean, I'm tempted to redo a lot of stuff that I do. It doesn't necessarily mean I should do it. Especially if it's working and meeting the customers and the users' needs. Okay?  Okay, a few people. Okay?  Or if they're not getting any new features, right? You know, it's like, yeah, they have no idea what changed. But I spent a year and it is way better. Good use of my time, right? So will it enable the users to have some benefit? You know, is this going to be worth it? Those are questions you have to ask. And it's, you know, similar to back when you have the pollution side of things, you know, that's a tech debt that you need to solve. Okay? So that needs to be resolved. But when you talk about regret, that's saying like, ah, I wish I would have done something different to fix it because, yeah, you know, it's, it's, it's a bigger deal. Now, the carp on the other hand, that's also tech debt and regret together. So there you go. I regret what I did. Now I really need to fix it. So how many people have bought into this new concept called DevOps? Oh, yes, the DevOps. Yes. So let's talk about the best DevOps example that there is, the Cold War. Yes, this is the epitome of awesomeness for DevOps and automation. Of course it is. And I'm going to talk to you about that dark time of, you know, our lives when we all lived under the fear of nuclear annihilation. Yes, that wonderful time when strategic air command was all the rage. Now, if no one is familiar with what strategic air command is, let me give you a rundown. Strategic air command had the responsibility of being able to blow up anybody within 15 minutes. That's their job, right? So, you know, they decided that they were going to have lots of different locations and they were going to be able to get anywhere within 15 minutes. They had to have tens of thousands of employees distributed all over these bases and they had to be responsible for the most lethal killing thing in the entire world. Okay, so they had a pretty big job. But the problem with strategic air command was initially when they launched it, they started to do these very routine tests and they found like, oh, they got proficient on everything because their test was something like take off and land. That seems like a good thing, you know, like taking off and landing. But then they really didn't test anything else. So they found out, like when they started to increase the tests and what they were looking for, that they were good within a five mile radius when they were going to drop a bomb. So somewhere within five miles, you know, they could hit, which is probably not as accurate as you want it to be. But they also found that they couldn't get anywhere in the world in 15 minutes. It would actually take them about three weeks. Kind of defeats the purpose of the whole thing.  And, you know, he came up with these four issues, right? Inflated scores and exercises. So as they did those tests, they were getting like all these great reviews and they were completely controlled tests that had no real world applicability, right?  They also didn't have any sort of accuracy, like I said. They also had this insanely high accident rate, right? So people were dying within like, you know, just test flights and stuff and things were falling apart because no one actually maintained the systems and no one actually maintained the planes. They would just take them out for their test runs and if they flew, then wonderful. And they also had almost no routine and preventative maintenance, right? So they had some pretty significant issues. So they brought in this new general to take ownership of this and he really started with this philosophy of, you know, first consistency standardization and leadership from the top, right? So we want to work as a team. We want everything to work together. So leaders and crew are all part of the same, you know, group and they should be treated the same. No one should be treated differently. But we're going to have checklists. So when we're talk about those failure rates, we're going to have 600, you know, different checks.  These are checks that they started back here. Okay, this is how like, you know, flight started to like get standardized. So they check everything to make sure it works.  So we were looking at ways to see what you could do to increase, you know, that consistency and efficiency. But they also built everything for repeatable scale. So knowing that they would have to potentially fight in a war, they started to make everything universal. So it was like, oh, we want every missile silo to be the exact same because if the power goes out, then people know how to get around because they won't have the lights.  And if they know where the parts are, they can go get the parts, they know where the generator is. So every missile silo that they built was the exact same, had the same process. And everyone was able to train on anyone else's job so they could fail over people in the case of an issue. Now, when we look at this and we look at the parallels in the current market space, right? I mean, how do you draw Cold War back into the current infrastructure space? We talk about automation. We talk about process. We talk about checklists. But we still know that these still have failures. But we have built the systems to go through and check our tests. We're doing a lot of test-driven development. We're doing automated testing. So we're looking for those problems that occur. We're doing our own 600-point checklists. But we're doing it automatically. We're also reusing a lot of our components. As we start to use microservices, we're trying to copy the same infrastructure so we can have that consistency. So when there is an issue, we know how to fix it. When there is a new release, we know that it is rolled out to everybody else. We also verify that every one of these things is following the process. So we focus on that verification and we hold each other accountable that way. But I mean, there are some things that we don't do, right? So one of the big things that we've seen rise in the last few years is like chaos testing. And that's something that we haven't done enough.  This is not something that's good. But we do see that people do take those types of tests seriously when they do do them. So these are types of things that we can learn from this particular space. Now, a few final thoughts. Look, we're all designing new systems. We're all looking for inspiration from everybody else. It's great to come to a conference here and listen to how other people are talking and doing things. But hey, as you're driving through the drive-through, as you're going to get your coffee at Starbucks, as you're looking at that river that's on fire, you can gain inspiration from those and you can figure out new ways to potentially do some really cool things. There are tons of great examples out there. And one of the great things about these is a lot of times, I don't know if you get this, but you talk to somebody who's not in the tech space and they're like, oh, you're working tech. Can you explain to me what you do? And you're like, no. So you can take these lessons and apply them and talk to people who might not be as technical and show them like, oh, you know how this thing happened? Yeah, that's what we're trying to solve. We're trying to prevent the river from catching on fire and not releasing the carp. But it is something that you need to be mindful of. And so think about these types of things and hopefully you found it interesting. So we can learn from all those examples all around. So that's my talk for today, everybody. I hope that you appreciated this fun ride through history and complete nonsense. So the question is, should automated testing be augmented with manual testing? I do think that once in a while you should go through some manual tests, yes, because I think that we rely too much on automation to save us and it often overlooks some minimal things. So I'm not saying you have to manually test every week, every quarter, every so often, you have to do something. For instance, testing backups, I'm a database guy. So we often just script the backups to run, but we never test them. And so you don't necessarily understand what you're seeing there. So going through that process of testing things manually as well as the automation, it's a good practice to go through a couple of times a year maybe or maybe even more often than that. It just depends on what your application is and how big it is. I don't think you can do it at mass scale, but there's so much that you can overlook if you're not looking at it. Because your focus is just on the tests you've written. And if you haven't written tests for it and it's causing an issue, then how do you catch it? Right? You wait until somebody catches it in production. So, yeah. Any other questions? Cool. All right. Thank you very much. I hope you enjoyed this. You are beta crowd number two on this talk. So hopefully you liked it. Check it out. Sounds good. Enjoy. Hi, everybody. We're still a few minutes from getting started, but feel free to grab a seat. Feel free to grab some ice cream or some coffee or some snacks from the back of the room as a sweet way to end scale. So if you are in the back of the room again, there are plenty of seats. Come grab a spot even socially distant. We are good. And again, enjoy some ice cream in the back of the room. We'll get started promptly at three o'clock, which is just a few minutes away. Good afternoon, everybody. I know we're all super excited for the presentation in just a moment, so I'd ask if we can start to settle down, start to grab some seats. There is not many sessions that will pack a room at three o'clock on a Sunday. I know we're all super excited about this one, so thanks for coming out. Again, at the risk of you all running out of your seats one last time, which I don't encourage, there is still some ice cream in the back of the room for those of you that are looking for some. Maybe it's gone. There is definitely some coffee and some other things. Again, thank you to our amazing break sponsors that supported the event, VMware MatterMost, as what happens after four days of being here. I will get that in a second and I will thank them. Hopefully they're not watching the recording. Data Bricks, yes, sorry, Data Bricks as well. Thank you again. We are super excited to have everybody back here together after that two and a half year break. For those of you that were here in the morning, it's maybe closer to a four or five hour break, but thanks again. We've got a few administrative things and then we'll go ahead and get started with our main attraction with Dr. Server. How many of you participated in the Expo floor passport program slash scavenger hunt? How many of you feel like you got all the answers right? Less fewer hands? Turns out only one did. He was tenacious in doing so. He was tracking me down, asking me for suggestions, tips, trying to figure out if I knew the answers. I did not because I didn't put together the program and it looked hard so I didn't personally do it. I also don't think I'm allowed to give myself a prize. We've actually got, I see him in the room here. Kyle, do you want to come join me on stage and tell us a little bit about what they have won and then I will tell you who it is. Hey everybody. So what we have for the Rappel winner today is a fully decked out Libram 14 laptop with, oh what makes it special. So one of the things, so when we made it we looked into what are all the things that we learned from all the past laptops we made that we think would make this our dream laptop. So in addition to having hardware kill switches that shut off the web him a microphone and the Wi-Fi and Bluetooth, it has, it runs core boot firmware. It also runs in an open source embedded controller firmware as well that we developed. And in general it's just like the best laptop that we've been able to make so far. Thank you very much Kyle. So I'll give you one fun fact on Kyle. Kyle is the only scale speaker that I know of in the last 20 years that we've been running the conference to have a literal explosion occur in their talk. Please, I know it might sound hilarious but please don't leave cans of Coke directly behind the hot bands on projectors in the session rooms. I'm sure nobody intended to do it but it made for a fun surprise for Kyle and a sticky situation for the hotel. So I'll stop keeping you all wondering if you won. The one person who did win is Dr. Sam Coleman in the back of the room. Come on up. And for those of you that don't know Sam, you've been coming to scale since what 2004? 2004? And you were at the time you helped us run the education track among other things of our call correctly? The time we're to open source software and education and so at the time I don't think you were Dr. Coleman, at the time I think you were just Sam. But congratulations. I hope you enjoy this free software powered laptop open from again courtesy of the purism team and Kyle. And before you take off today we'll want to do a quick picture for the purism folks as a thank you for that. Awesome. So again for those of, thank you for coming out for the closing keynote. This is the first time we're doing this but we thought it was an important opportunity given we had, we had what's had soon a suspicious speaker joining us. And so reminder just the last few things. Again we will be back here well not here but in Pasadena a few miles away next year March 9th through the 12th returning to our returning to our home after all of this COVID fun. I appreciate you all sticking with us through the masks, through the vaccine mandates, through the tests and everything else. I know it's not fun. I hope that by next year we can skip a lot of this no promises. It's not pandemics are not in my control but I do know that we will be back together all things all things. So with that in mind my next speaker doesn't I don't think needs an introduction but I first heard of him in high school I was reading a book called where the wizards stay up late the history of the internet. If you've not read it it's a fantastic overview of just all of the early days of the internet how things came together at various institutions to build what we now most of us run our careers on today. Everything from how this conference was organized to what we did through the pandemic to how I get cat litter through the mail all organized and coordinated over at TCPIP. I don't know if Dr. Surf had that in mind when he did it but that was interesting. So yeah we're super excited. The other thing that interesting piece here is Dr. Surf did this while he was at UCLA and for those of you that don't know when scale started 20 years ago that was done by students at UCLA and USC and UCSB and CSUN and a few other local schools when we were all just but we children in our freshman and sophomore years of college thinking that running conferences would be easy. One thing I can tell you is that after 20 years the only thing I have learned is that it is not easy but but yes so we're super excited to have Dr. Surf and one of the things that has been a pleasure for me over the last 20 years is just getting to meet my my my internet and sort of CS heroes as we bring them as we bring them on stage here for key notes and for other activities and so without further ado I give you I give you Dr. Surf is joining us all the way from Virginia and appreciate appreciate him making the time as well as for joining us. Thank you Dr. Surf. Thank you all very much you know when people clap before you said anything my first reaction is to sit down because it won't get any better than that. Now you all I clearly understand the tactic that's been employed here you wait until the last session of the four days of ruling fantastic I looked at the at the list of sessions holy moly you guys have got an enormous amount of useful information and I'm sure in the corridors in between lots of other very useful ideas that are flying back and forth so what you do on the last day is you get the talking dinosaur to come out and and it doesn't matter what the dinosaur says the fact that he can still talk is amazing so I was I think I've been given a few suggestions though about things to talk about and I understand what to get to be my age you tell anecdotes so so we're gonna go looking to the past and look at a few anecdotes and I see more than a few gray hairs in the audience which means that I don't have any left but my beard okay uh so for some of you this will be I hope a kind of a fun reminiscent of things past that have taken us to where we are today and where we are today is a big challenge and I'll try to close my talk with some thoughts about things that we might collectively do to meet some of those challenges and what I find most appealing I think about you in particular is that you've been doing this conference now for 20 years and there is this fellow feeling of collaborative sense of responsibility that you bring to the table that I wish everyone who writes software would bring to the table so we're going to come to that in the last few thoughts of this talk but let me start out by jumping into the time machine for a second take you back to 1969 UCLA not very far away there were four notes by the end of December 1969 in operation at these four locations which were selected by the Defense Advanced Research Projects Agency for particular reasons UCLA was selected as the first note because Len Kleinhoff who is still a professor in UCLA ran the network measurement center and my job as a graduate student there was to write software for the Sigma 7 machine to measure the performance of the ARPANET and compare with the queuing theoretic models that Len Kleinhoff students were developing so we could compare what the predictions were from the queuing theory models and see what actually came back from the measurements and I can tell you that the queuing theoretic models were always but they didn't always predict what actually happened in the real network and I suspected every one of you knows exactly why that's the case so UCLA is network measurement center then SRI International is the second note and that was because Doug Engelbart was at the Augmentation Research Center at SRI International his belief and the belief of J.C.R. Lechweider who was the who was running the information processing techniques office at Dartmouth believed that computers could be a way of augmenting our capabilities and I would say that they were right our capabilities today what we do every single day are often augmented by the kinds of software that you write and other people write to help us do things that we could do on our own not least of which for example searching the entire world why why so that was the second note and the third one was UCSB because they were doing some really interesting work on presentation of complex functions on the screen so computations were producing and finally University of Utah because they were very much involved in computer graphics at the time you can imagine computer graphics in 1969 are not like what they are today but one of the guides at Utah invented one of the first hidden line removal algorithms so that they were doing 3D rendering of what the surface looked like he had to remove the parts that couldn't be seen so he found it was John Mornock who some of you know Adobe some years later so that's 1960 1969 many years ago and this is a picture that was taken in 1994 it was the 25th anniversary of the ARPANET on the left you see John Pustell who sadly passed away in 1998 in the middle of Steve Crocker who led the network working group that did all of the host to host protocols and telnet protocols the FTP and eventually SMTP for email so he was the leader of that group and he was at UCLA as was John Pustell all three of us went to the same high school Van Nuys High in the San Fernando Valley I don't know it must have been something in the air but but we we became good friends at UCLA as graduate students it took us all day to organize the shoot for Newsweek magazine because we had to draw all those pictures of the backdrop hanging here then we had to go find the zucchinis and the yellow squash and the five five pound tins of coffee and then string it all together now this is Newsweek magazine 1994 and we thought we would put up a geek joke for those who understood it so if you look carefully at this network you notice that it's mount the mount in year to year but there's no mount to year this network would never work so that was our little geek joke in 1994 but we were celebrating the that anniversary of the ARBANET because without it we would not have even bothered to do the internet so now I'm going to skip very very fast forward here past the original early designs of GCP and then later GCP IP this was a demonstration that I called for after I moved to Virginia in 1976 to run the internet program for the defense department and I wanted very very much to show that the GCP IP protocols would really work across different kinds of networks or considerable distances so we had a mobile packet radio network in the San Francisco Bay area driving up up and down 101 radiating packets like crazy through a gateway which had been configured to send those packets all the way across the ARPANET which by that time extended into Europe over an internal satellite hot from ETAM West Virginia to and then a landline to Norway to the Norwegian defense research establishment and another landline down to University College London and then popped out of the ARPANET at that point into another gateway that led to the Atlantic packet satellite net which was based on the IdleSat 4a hanging over the Atlantic with multiple ground stations all vying for access to one single satellite channel so it's like having an Ethernet channel in the sky and then through the packet packet satellite network to ETAM West Virginia down to another ground station back into the ARPANET all the way across the US again to USC Information Sciences Institute so if you do the math of course going from the Roving vehicle in San Francisco down to Marina Del Rey it's about 400 miles but the packets have gone through two synchronous satellite tops back and forth so about a hundred thousand miles and I remember when we did it and it worked I was jumping around in my office saying it works it works you know like it couldn't possibly have worked listen if it's you know you know that if it works if software works it's a miracle so so so that was a really important demonstration from my point of view as the program manager for this now we'll skip forward now we're in the early and mid 1980s after the ARPANET and the and those three network demonstration was done and further standardization was done then we implemented the TCP-IP protocols on every operating system we could get our hands on in 1982 John Postel announced that we are going to switch over from the NCP host to host protocols of the ARPANET to the TCP-IP protocols of the internet on January 1 1983 so this is January of 82 we make this announcement and everybody sort of wobbles a bit you know everything seems to be working on so why do we have to do that and say well you know if you don't do that they won't fund your program next year oh okay I get that so I had Dan Lynch and somebody who knows the founder of Interop was measuring how many implementations of TCP-IP could detect on the ARPANET and so he would report to me you know once every couple of weeks and I could see that you know the curve was going up and then somewhere around the summertime it flattens out so you know I'm a bureaucrat at this point so what do you do well you clearly create incentives so I called the defense communications agency now called DISA and I said will you shut off the capability of the ARPANET to carry anything except TCP-IP they could do that it turns out there was a way to do that and they said turn it off for a day so they did of course the phone bringing off the hook what's the matter with you you blankity blank you know can't can't get an email you know files nowhere you and I said I just want you to know I can do that so so you know oh okay so curve starts going up again then somewhere around late summer you know October or something it flattens out again so I called DCA and turned it off for two days phone rings off the hook everybody made it on January 1983 except for two guys and they tweeted for some kind of mercy and he said okay we'll give you another much to do so everybody made it up now how many everybody's were there there were only 400 computers on the internet at that point 400 as opposed to 400 million or whatever the numbers are today if you include the IoT devices in the mobiles we're into the multiple billions so now so it's now 1983 NSF is starting to invest in internet technology for hooking computer science departments up to the ARPANET and then to the NSF network backbone and they quickly run out of gas they essentially sent out an RFP asking for a higher speed network now remember the backbone speeds of the ARPANET were 50 kilobits a second okay that was that was broadband in 1969 so NSF net launches 50 kilobits and basically runs out of gas it switches to 1.5 megabits and lasts for a little while and then it switches to higher speeds and we kept just kept going you know eventually we end up in multi gigabit range so NSF makes this big investment they build a backbone network they build about a dozen intermediate level networks to serve about 3,000 universities the purpose for which is to help those universities get access to the five supercomputers centers that NSF and the Department of Energy are investing in so the NSF net comes up and the supercomputers centers come up and not long thereafter NASA and the Department of Energy say this looks like a good time so DOE builds the ES net backbone and NASA builds the NSI or NASA science internet backbone so during the mid to late 1980s we're starting to see substantial implementation of course if we go to look at the internet today now some of you will know I'm cheating right because this picture was actually generated from the BGP backbone back in 1999 but it was so colorful that I thought I would keep it because it lets me illustrate something that you know and that is that the internet is very dispersed there are literally tens of thousands of more networks running each operator fix hardware and software to run just who to connect to you know what terms and conditions there were no central decisions about your business model who you connected with the only thing we have of everybody is please run the same protocol so that you could interoperate that we wanted to encourage Bob Cohn and I to encourage people to build pieces of internet find someone to connect to and let the system grow so that's what it looks like today except bigger and more colorful it is true that there is only one centralization element and that is trying to coordinate IP address allocation and domain name assignment so that they're unique and that's pretty important but ICANN doesn't dictate what is done with those it just tries to make sure that if you want a domain name that it's assigned uniquely to one organization or IP address block to an autonomous system so this is just my little memory of milestones I'm sure that every one of you who's been engaged in this game have your own favorite milestones of by no means should you take this as the definitive list of important milestones they're just the ones that I happen to remember one very important one comes after the internet goes operational in 93 and Cisco systems figures out that they can make money by selling routers to universities they want to get hooked up to the net and put local area networks in place oh by the way I did leave out something important 1973 Bob Khan comes to visit me in my office at Stanford and says we have a problem and he says and he says well the ARPA network and now the defense department wants to see if it can use computers and command and control but he right away realizes if you want to do that the computers are going to be in mobile vehicles and ships and sea and airplanes and the ARPA net was built out of dedicated telephone lines connecting everything together you can't connect the ships together that way because they get all tangled up the tanks run over the tanks run over the wires and they break and the airplanes never make it off the map so he was already starting to work on the packet radio net and the packet satellite net at the time that we met so we started working on the gcp something about a mile or two away Bob McCount and David Boggs are busy inventing internet heads there on the park so that was the fourth packet switching technology that was born during this 1970s early 1970s period so Cisco was the first to start commercializing this stuff the way we used to build routers was to find a computer and a graduate student you wrap the graduate student around the computer turn it into a router that the problem with that is we were running out of graduate students so Cisco figures that out also in 84 it becomes very clear that this thing is scaling up and that we can't go keep sending the host.txt file around the map the domain names into ip addresses so we needed something that was more scalable so Paul Macho Tetris and John Puffstahl invented an ass that evolves over time but it certainly has scaled dramatically so around 1988 by this time Dan Lynch is doing interop and it's up in the San Francisco Bay area and I walk in in 1988 he started in 86 but then it was an exhibit well and the deal was you can't exhibit unless you show that you've been interworked with everybody else so they bring out the show net this big fat yellow voxel cable you all had to plug into it and then show that you could talk to everybody else in the show so Eric Venemo who was then the CEO of three con company that Bob McAfs started to sell each and I walked into the show the first thing we encountered is a two-story Cisco thing you know display and so I turned to Eric I said Eric how much did those displays cost pieces about $250,000 this is 1988 and I'm sort of sitting here saying that's a lot of money and he said that doesn't count the people who had to stay there and so I'm just standing there and my jaw is dropping thinking somebody thinks they can make money out of the internet so so at at this point I'm starting to wonder because you know how are we going to let the rest of the general population get access to this thing because up until that time all the networks were funded by the government agencies so there was an appropriate use policy and said no commercial traffic will flow on a government sponsored backbone and you could imagine the rationale for that they didn't want government resources to support commercial activity but it became increasingly clear that even the people who were doing research under government actually needed access to commercial services that could be reached on the internet if it were permitted so at this point I'm working with Bob Conn and his company the corporation for national research initiatives and I had taken a little break from the internet from roughly 82 86 to build something called MCI mail which is a commercial email service so in 88 I'm sitting here thinking okay so what could I say to the federal government that would let me break the appropriate use policy without really appearing to one so I called the federal networking council which at the time was mostly NASA DARPA NSF policy so I called them and I said would it be okay if I tried to connect the MCI mail system to the internet see if we could get the email stuff to inter work and to my surprise they said well okay you know you know for a year and so by summer so 1989 we announced we have a gateway between the MCI mail system and as soon as we make that announcement all of the other commercial email service providers which are islands into themselves like you can't talk to anybody unless you all have the account so things like telemail and on time computer yes all of those I'd say wait a minute these MCI guys can't have this special treat we want on too and so the government says okay so they all get hooked up and two things happen first thing that happens is they all discover that all of their customers who used to be trapped in a wall of garden can talk to all the other customers of their competitors plus everybody on the internet because they were all compatible through the internet mail protocols and that was a little surprised and then later mail becomes almost free so much for that business model but the second thing that happens about the same time maybe a little later in the years three commercial internet services pop up because we've just broken the AUP limitation so you have net in Virginia and PSI net in Virginia and surf net in San Diego all gets started in 1989 okay yeah this is anecdote number surf net used to be spelled S U R F net and of course you do that you're in San Diego what else would you do so so they had a whole campaign all lay down t-shirts you know surf the internet and all that stuff and then a couple weeks before they actually launch somebody discovers that there's an organization in the Netherlands called surf net which is you know it's a Dutch acronym and they are building a network to connect the universities in another one so they can't call themselves S U R F net so the Susan Ostrato is the executive director so somebody says why don't we change our name to the California Educational Research Foundation network but you know it sounds the same and then somebody says maybe we should call them in so they call me up and can we call it surf net and my first reaction was you know they screw this up am I going to be embarrassed and I thought about it this morning wait a minute now people name their kids after other people and if the kids don't come out right they don't blame the people they name them so I said sure so I flew out to California in 19th Susan and I had a plastic bottle full of glitter and we smashed it on a Cisco router and launched the surf net so by that time we're starting to see real commercial services pop up uh Rick Adams was the founder of media so we're talking 1989 so 1997 he sold the company for two billion dollars which on the same day so far I will come for 14 billion dollars so so uh just you know picking a few more of these things uh the big deal after commercialization forces went temporarily and houses the worldwide web which is late December as I recall and I don't think too many people actually know that he was doing it on the next machine which is very cool at the same but not too many people noticed except for these two kindness Mark and Jason and Eric Mina at the National Center for Supercomputer Applications in Urbana Champaign and they look at the text-based interface and they say comes out around 1993 and everybody notices because suddenly the internet looks like a magazine with formatted text imagery and eventually streaming audio and video so that was a big deal and Jim Clark the founder of Silicon Graphics takes one look at the mosaic browser and he says this is a big part of Silicon Graphics which turned out to be based on another chip that ARPA funded called the Geometry Engine so he flies out and he takes Mark and Jason and Eric Mina and a few other people back to the west coast and start Netscape Communications in 1994 and by that time I had left Bob Conn's organization to rejoin MCI to put them in the internet business and the first thing they wanted to do is build the MCI mall okay so so I fly out to Netscape and buy seven million dollars worth of licenses for Netscapes browser and server and first thing I asked them to do was to figure out how to avoid having my servers filled with partial transactions that won't ever get cleared away and I won't know when to get rid of them so please store the partial transactions on the user's computer and so they went away and came back with cookies so if you're wondering where cookies come from so um so that you know of course they go public in 1995 the stock goes to the roof and the dot boom is on the venture capitalist in San Francisco were throwing money at anything that looks like it might have something to do with the internet and this goes on for a while 1998 in the midst of the dot boom google gets started by Sergi Grimmie and Larry Page Yahoo gets started a little bit before that to surf the internet the interesting thing about the arrival of the World Wide Web is that it triggered an avalanche of content that flows into the net so interesting people who are not looking for money for the content they just wanted to know it was useful for somebody else kind of like what you do so this avalanche pours in pretty soon nobody can find anything because there's so much of it so they need uh they need a search engine some of you will remember Alta Vista which came and then Yahoo comes along which was kind of more manual I think than some of the others were and then Google had a strategy called page rank which was very successful it didn't have a business model to start with by the way but not very long after it got started and after they bought Eric Schmidt and it's the CEO sort of adult supervision then this three-way business model evolved which was quite successful ICANN gets started in 1998 and the original idea was that John Costello would be the chief naming system and IP address allocations through the regional internet registries as they say John passed away in September of 98 but ICANN was really needed so now they proceeded now one thing I have not done is to tell you about the 1978 to 1993 protocol wars between the open systems interconnection model and TCP IP to say nothing of x25 and x75 and x29 and so on that would take too long so I won't do that but a lot of people imagined that this is to smooth sailing and it was just step-by-step and everything was all planned out and yet nope it was storm storm and wrong for many many years and it's still storm and wrong today so just pushing a little further into time here a few other things the dot bust happens in 2000s around April a big lesson there a lot of the startup CEOs apparently didn't understand the difference between revenue and capital and in economics 101 says you have a finite amount of capital to get your revenue engine going and if you don't get the revenue engine going you will run out of capital and then what so lots of dead bodies startups in 2000 but the internet kept going the demand for that capability was still very strong so youtube gets started in 2005 amazon web services comes up in 2006 the iphone shows up I want to emphasize the iphone improvement because all of you I'm sure know what a transforming event that was but now this is anecdote number 17 or 18 I guess um some of you may know that the mobile phone was working for Motorola at the time what you might not know is that they started working on it in 1973 so um so anyway this this thing gets started in uh in the same year as 73 uh and it gets turned on in 1983 the same year that the internet gets turned on so Danny Cohen another name you might know very much so Danny calls me up in early 1983 says come have lunch I have something to show you so I show up and he's got this thing sitting on the table it's got this tall it's got a whip and pen on it and it weighs about two and a half pounds and uh what's that he says it's a phone where are the wires there aren't any how does it work so we talked about it for a while I don't know the answer to that why don't you call the guy that invented it so I called Marty Cooper on a Motorola break the first question I asked Marty was how long does the battery last and he said it's about 20 minutes but you can't hold the phone up longer than that so so Marty bless his heart you know and presses on he's still around he's still around in his 90s now he's just written a book about the whole story of the but the iPhone is really triggering is that when jobs figured out something that none of us realized that we wanted which is a device that had a camera it had access to the internet it had access to the telephones it had a touch sensitive display it's got all of these amazing features all of which existed as a technology never put together in such an interesting way that transforms everything because suddenly the internet is more accessible anywhere you get a mobile so you can get the internet and of course the mobile phone gets more useful because it gets access to all the applications that are running on the internet so that you are mutually reinforcing really powerful man so 2007 is a big deal in the world that we now inhabit in 2008 2009 several developments came out of the academic world this is basically software to find networking which really has transformed fate and in fact in 2010 started a company there's lots more I'm not the last 10 years of development and you've lived those last 10 years anyway so you know so somebody asked me to look back and say what would you do differently and so I decided I put a little list together first one thought this right I would have done IPv6 first instead of IPv4 but because it's been damn hard to take to to cause a incompatible introduction of it in our own defense though we actually did a calculation when we were doing the original design of TCP to see how much address space we ought to have to remember it was an experiment and we didn't know if it was going to work so we said okay it has to work everywhere in the world because it's going to be supporting the defense department command and control system so we said okay so how many countries are there how many how many networks per country and we thought well how about two then we said how many countries are there and there wasn't any google time to ask so we guessed that 128 because that's the power of two and and you know we did the math 256 that's eight bits okay so we know how many networks we got to be with 256 networks and then how many computers for network how about 16 million at the time there were millions of dollars and they didn't move anywhere they were in air conditioned rooms and they were hooked together with wires but we thought what the heck besides that rounds it out to 32 bits which is cool so and we thought you know that's 4.3 billion terminations if you if you could allocate them densely of course you never would but if you could and that was more than there were people in the world at the time so we thought that ought to be enough for an experiment now i want you to imagine you know that you're a young vince started in 1973 and and and the future itself goes back and whispers in your ear and says 128 bits of the dress space and your younger self says wtf that's that's 3.4 times 10 to the 38th addresses and he said yeah and i don't think i can sell that it doesn't pass the red face test your network has never even been demonstrated and you're telling me it needed 10 to the 38th addresses so i don't know if i would have gotten away with that now there is a huge mistake that i want for mobility support uh this this one it's amazing how you can fool yourself into thinking solve the problem when you actually have it i remember splitting tcp and ip and then we had to figure out how the tcp identifiers would work on an end to end basis so we created a pseudo headers sucking the ip addresses up out of the ip layer into the tcp layer and use that for socket identification and since i had an operating mobile radio network at the time i thought we had dealt with mobility except for one thing i didn't think about the possibility that your mobile would move from one network to a different network that had a different ip address space and at the time i was thinking okay i know i can put another address space at the tcp layer so that it's okay to switch ip out from under the tcp and those of you who know about quick know that the quick protocols establish a cryptographic shared variable at the upper layer at the quick layer so and i if an ip address changes not both but if one of them changes you can reconstruct the network if they both change of course they can't figure out who to talk to so that doesn't work anyway i thought that the mobile problem had to be solved clearly it has not been solved so that was a big mistake and i regret that but at the time i was patting myself on the back for saving bits in the header so you know be careful what you congratulate yourself for it the other thing that has occurred to me is that radio has wonderful features one of which is broadcast thinking of satellites a bunch of stuff gets sent to lots and lots of receivers and the guys that didn't get it to raise their hand and say please send me another copy but we never really built any protocol another thing that that we could have done is put crypto into the system sooner than we did and a lot of people come and say you know wanky wanky at the beginning we wouldn't be in such a mess that we are today i don't actually think that's true but at the time in 1976 when Widdiffy and Marty helman published their first paper on new directions in cryptography it was stunning and i'm sure the guys in the uk were especially stunned because they invented this idea in 1974 but they didn't but they didn't tell anybody because they didn't want anybody to know about how clever this was so anyway the paper gets published and the next year or so 78 or so um the rsa algorithm to implement this idea and it gets invented now you could say why the hell didn't you just immediately and my reaction remember is 1977 i'm trying to get the damn thing demonstrated and implemented on as many operating systems as possible and i look at the rsa idea and i said this is retrofitable i can't put this in anytime and so i wanted to get the system up and running and demonstrated first so we did that we were actually working using bs which is a conventional symmetric key to build a cryptographically secure system and we were demonstrating that with a program called black crypto red which is the red side of the head is a sensitive side the black side is post-crypto and we stuck the es in the middle but key distribution is not not nearly as nice uh with the symmetric key system as it is but anyway so we were we were clearly working on on the crypto uh support and of course the nsa was visually doing some of its uh development work as well but i remember thinking at the time okay if i were serious about insisting on cryptographic implementations who are the users of this thing it's graduate students and i don't mean any events if i was a graduate student was too but i can't imagine graduate students being really good about key management and all the other things that you have to do so it kind of was comfortable not doing that and not insisting on it and of course multi-factor authentication would also have been a good thing to have thought about if we had any technology to support it because even then so here we are we really and so from my point of view we really just make a dumb mistake configuring something wrong almost all the really bad stuff that happens most of the time somebody's just making a mistake sometimes it's not somebody did it on purpose rpki is another thing which ditto dms and so those are all things that we should be working on i also think it's more strong authentication system all the way down to identifying on hardware being able to something half a million webcams that were hijacked which had cascade effects because dying was doing everybody's main resolution for a lot of very important companies and they all disappeared off the net because dying fell over which by the way hang on to that thought for just a second because if you're like me because if you think about how dependent we are on certain key parts that we are to see when they don't work their cascade failures if your mobile doesn't work rather instead can't get a signal for some other problem then you can't do two-factor authentication maybe so that doesn't feel you were about to close doesn't happen more than once even just over this weekend i had to go through a whole series of login and authentication steps some of which involve mobile and now i've got two devices that i have to make sure work everybody in this room knows that the probability of success is multiplied multiplicative so 90 successful times 90 successful is 81 successful if you rely on both things working and if there's a third thing it just gets worse so we are steering right now into a very fragile future thinking about software development this isn't just a question of more security now some people have criticized the basic assumption that we made that every device on the internet should be able to talk to every other device and the reason that we chose that as our principle didn't know which devices were going to need to talk to which devices so we had no rationale for inhibiting communications now today we look back and realize that since everything can talk to everything then the bad guys can talk to everything and they do and they cause trouble so it could very well be that we should reconsider how to hide parties from exposure and there are some suggestions so i'm also very conscious of the fact that read my own watch now those of you who have good eyes will know this is a Ronald McDonald one and i only have 10 minutes left but i got this for teaching a class in networking at Hamburg university just outside of Illinois i am not kidding there is a university they teach people how to run McDonald's and they were about to network them all together so they could keep track of the sales and the use of resources and everything else instead of giving me a Ronald McDonald watch to commemorate the thing so anyway let me just riff for a second on open source and open standards because that's what you do and i tell you what you do is really important and i still consider it to be one of the central engines of the evolution sharing your think about what happened with the world wide web there was no class in being a web master what you got to do with browser was a show source so you can see how did they make this really cool web page because you can see the hdml and so lots and lots of people learn how to be web masters from each other and that's what you can learn from it accelerates the pace of it also open source also gives you an opportunity to find bugs now there's a little problem with this sometimes open source leads people to think it's open source everybody's already found the bugs because it's open source therefore you don't need to look yes you can stop laughing now so i do worry about that and i have pushed very hard wherever i can to argue we need to support people like you more so that you can help us make more resilient and more stable and safe software it's really tough because the curation of this software is really important some of the bugs that some of you know have fly around for 20 there's lots of supply chain risk factors that are involved because the open source software can be anywhere in the stack and a bug that gets in deliberately or so there's a real challenge for us as open source software in a sustainable way so i just want you to know a big fan of trying to we need this stuff the standards and the open source for interoperability some people say standards inhibit inter a competition but i don't agree with that i think that having commonality and interoperability allows for competition and of course we all need to require software to whatever new platforms that come along this is a slide which i don't have time to talk about but i want to just pause and ask you to look at it for a second because these are all problem spots they are not full list they're just some of the challenges that some of them are really tricky because they involve international agreements of some kind whether it's treaties or norms or something else think for a minute about the world that we wish we lived in one where accountability is enforceable and that agency is given to you and in order to make that work you may have to give up some anonymity because if you can't hold people you can't hold anonymous people accountable and so if they are deliberately doing harm it's a little bit like license flights it's not a perfect analogy but you know license flights are gobbledygook except mine which that serves up so most of them that most of them are just random stuff but the police department is permitted to penetrate the veil and find out who owns the car it may not be the person who was driving the car but if they can penetrate that veil because it's their job