Good morning everybody. We're in Pillager, Minnesota. Diesel, how'd you sleep? Eh? You good sleep? Having diesel with me this trip means much more housekeeping in the truck. That's okay. We have a good little vacuum cleaner. I've been feeling left out and neglected for the past few trips when I didn't have diesel with me. Getting good use out of it now. It's really quite a shame all the litter we see at truck stops. This one's not so bad, but I mean, you get all kinds of stuff. Look at this back here. Whatever that is. You got all this paper garbage here. You got a tire here. Got a tarp here. Got a tire over there. I know that it's not the majority of truckers who leave their garbage out here. It's just a very select few. But those few make such a big mess. Eleven years old, but he's still a puppy at heart. Breakfast, I've had got my coffee. I haven't had it yet, but I've got it in my possession. That's all that matters. Let's get going. We need to get this freight off my trailer. Let's roll. The process of taking the tarps off the load. Got one tarp off.  Believe me, this is very difficult when there's wind. That doesn't mean you can't get your job done. You've still got to get it down. So I'm gonna put you down over here. I'll show you how I pulled this out. First of all, you can see that the flap at the front that goes over the front of the load. I flipped over on top of it there, right? But the nice one, I leave my straps on. I'm gonna roll inside the building, take the straps off in there.  Looks like I might be going to Fargo, North Dakota for a reload. Find that out in a bit. My work is done. Got all my equipment put away. Straps, tarps, everything. All put away. I'm just waiting for them to take the last couple of lifts off my trailer. They're doing it right now. I have two more lifts on there and then they got to sign some paperwork and then stone. We're that little cabin is right over there. The car just drove through. That's the main road going through town here. This is Emily, Minnesota. Diesel. Found ourselves a little pathway through to this little gravel road back here. We're gonna go see where it heads to. Go explore town a little bit and actually possibly get some exercise in. Diesel, can you come here? You sneaky little guy. Sneaky little guy, come here. Hey, there you go. There you go. Come on, put it on. Now don't eat it. Diesel, come here. Diesel, come on. Come here. Are you a waniak? Are you a waniak? Diesel, we're in a strange town. Technically a foreign country. Okay, I don't want you getting lost here. And I don't want you getting hit by anything either. I don't know. There might be some ATVs or something come flying out of this trail here. I wonder where this trail goes. I wonder if it's private property or not. This is the US. If you don't know if it's private property, it's best just to not even go find out. Now we're on the other side of that main highway. You'll see them driving through there in just a second. On the other side of town. There they go. Same road we were talking about before. We're just on the other side. We found this beautiful park here. Should have come this way first, eh Diesel? And behind all these trees here, there's a beautiful lake down there, yet that I had no idea even existed. I'm gonna go check it out. I don't know if there's a public access point. I would assume so because this lake is right there. This is a public park. These are all private cottages on the lake. You see that? There's got to be like a public, either a beach or a a boat dock or something around here. If we can just go to the edge of the bush here, this part of the park hasn't been mowed. I don't know if it's still a part of the park or not. Yeah, I think so because here you've got these horseshoe games. I wonder why this area of the park isn't mowed down like that area. And what if people want to come here and play some horseshoes? Diesel, I think this is the way down to the water. We'll check it out. Maybe I should have put my swim trunks on. I've got my bull's ear sets in so I can listen to some music quietly. Actually, very nice because they still allow me to hear what's going on around me. I can have some music playing that way. I kind of want to. I mean, I haven't really been listening to music because when I go for a walk, I like hearing the sounds of nature anyway. You know what? I'm going to take them out. Hold on, Diesel. Hold on. I'm going to take my headsets out. I didn't take their case with me. Oh, that's much nicer. Oh, that is much nicer. I could hear everything before, but this is like genuine nature sounds. Come on, buddy. Diesel, come. Go check out the water. I have my trunks in my bag. Maybe I should have put them on. Makes me want to go swimming. You can smell the water. It smells good. I wonder what that's for. That's where they used to hang the pirates. Pirates ye be warned. The lake pirates. I'm joking for those of you. I don't know what this is for. It's probably not for hanging pirates, but you never know. Maybe. Diesel, hey, hey, don't go too far in there. I don't know what's in there, man. Oh, it's an old dock not being used. Look at that. That is Minnesota. Beautiful day out, too. It would be such a good day to be out on the water. Oh, looks like there's a, oh, we just had to go around a little bit. We came down the grassy way. We could have gone down through there, I guess. That part's all mowed down. Isn't that public? I don't know if that's private or not. Like I was saying before, in the U.S., you want to be careful where you walk. If it's a private property, you don't want to just go walking onto it. It's a good way to get a bullet inside you. Not that that would happen. Most likely you'd just get yelled at and then you leave.  It's not like people just go out firing at random people. It doesn't happen. That's all just the news media hyping things up. There's not one point in my life where I've been in the United States as a Canadian and have not felt safe. I always feel safe here. I know the news likes to make it sound like it's just this crazy dangerous place. And I'm sure there are neighborhoods like that in the U.S. Chicago comes to mind, but I don't walk through those neighborhoods. I walk through the countryside and everyone I have ever met in the U.S. and the countryside have been good, kind, generous people. They just don't want you walking onto their property without their permission. Fair enough, right? I don't like that either. And I'm out of breath. This is why we're doing these walks. This is why we're doing these walks. I need to get that exercise. There's another dog over here. Diesel already spotted him. That dog over there is not on a leash. And I always get frustrated when dog owners sort of don't pay attention to these little rules because maybe your dog is very well behaved and maybe it doesn't need to be on a leash in your mind.  And he wants to go running. And you know when two new dogs meet, you sort of have to supervise it just to make sure everything will go good.  Diesel doesn't always like every dog. He's never attacked a dog or anything, but he doesn't like every dog. Sometimes he bullies them a little bit. And that's why I have to be careful around him with him when he first meets a new dog and make sure they drive, right? Well, when they have the rules that say, please keep your pet on a leash, it's not always for your pet. It's also for the other pets. I mean, diesel starts misbehaving like crazy on his leash, which is something we're working on. But you know, fair enough, he sees another dog running loose, playing fetch. He wants to go run and chase him and play fetch too, right? Well, it's not fair to diesel that we're listening to the rules and they're not. He doesn't understand that bugs me a little bit when I see people have their dogs off leash in public spaces, especially at truck stops. I sort of just cringe because, you know, one gopher runs, runs across there and diesel would be gone chasing the gopher. And if I don't have him on a leash, well, your dog might be well behaved, but he sees diesel take off across the parking lot after a gopher. Chances are he's probably going to want to chase him as dogs do as other dogs do. They all run. And if that happens and you're not paying attention for that split second, dogs are just gone across the lot. And you know, these NASCAR drivers that come into the truck stops unannounced all times of day and night, 60 mile an hour through the pumps. It's just so risky. So I always just bugs me. I mean, he's got a pretty long lead. This is an eight meter lead, which is really long. Diesel show him how long it is. Show him how long it is. Keep going. You're good. And it's a very loose spring in here too. So it's as if he was loose and he has almost at the end here. He has all of that radius around me. Now, when we're by ourselves or in a dog park, sure, I'll let him off leash because he needs to really run sometimes, but only if we're alone. As soon as I see another dog, I would leash him up because I don't know the other dog. I guess it all depends. Some people are okay with it. Some people are definitely not okay with it. I'm sort of in the middle depends on the dog, right? And the owner has a lot on the owner. I just try to be respectful of everybody. And this was the park we were in here. He needs a little bit of paint, but Emily City Park. And if you didn't know, there's a lake right over there, like I showed you. There we are. Time to get back to trucking. We're going back to Canora for another run. Another load of lumber. Same thing. Pick it up in the morning and try to get that loaded like first thing in the morning. And if we're lucky, we might get it unloaded tomorrow afternoon yet. Looking good, blue. You miss me? You miss me? Do you think he missed us? Are you giving it we missed you? We're back over on the Canadian side for Francis, Ontario. Well, careful, bud. Don't, don't jump out into the street there. I thought that guy was going to jump right out into the street. I don't know what he's doing to standing there. Made me a little nervous. I always lock my doors when I'm in town, just in case people jump up on the truck and try to open the door. It's happened before. It's not like it's common, but it's happened. Sometimes they want to try and sell you stuff. That actually happened in the US. It's never happened in Canada. You never know. Fear would be that they want to rob you or something, right? You never want to leave your doors unlocked in traffic. People can just jump in, point a gun at you and be like, what are you going to do? Port Francis is very confusing. The US side makes it a little bit more straightforward, but I think they're currently making it better because everything's under construction and that's what kind of makes it confusing. Everything's under construction right around the border, right? Right around the port of entry. Come on, once again, we're going to try to get this done as quickly as possible. There's one other driver that's Park Tier as well from another company that's going to want to get loaded first thing in the morning as well. We'll see who's more on the ball first thing. That's whoever gets to the gate when they open, I guess. We'll see. See what happens. I believe we can get in at 7 or 7.30. I'll be ready to go in at 7. Quickly run in there, get loaded. I'll have to tarp it again. We're going back to Brainerd. Same run all over again. And if it's as easy as the last one to tarp, there's a really good chance we can get it unloaded tomorrow afternoon yet, but I'm not going to hold my breath because things happen, delays happen, and it'll probably end up being first thing the next morning. But we'll see what happens. Thanks for hanging out with me today, everybody. And thanks for hanging out with diesel as well. We've been having a good time putting everything onto the computer here right now and we'll see you in the morning. Right Diesel? So tired.


Why you need to cite sources in this tutorial will introduce citation and tell you some of the reasons you need to cite the sources You use in your research papers and assignments When you do research for your papers and projects you always cite your sources to cite means to point to the evidence You need to show where you found the information you used in your research What to cite? It doesn't matter what type of source it is. It can be print or online. It can be an image or a conversation In fact any place where you find someone else's ideas Examples include data journal articles charts or graphs newspaper articles books ideas theories and concepts course lectures emails and phone conversations blogs and wikis the web and more Quoting and paraphrasing are two important concepts to learn Before you write a research paper you need to cite when you quote a source and when you paraphrase a source To quote means to use another author's exact words in your text Quotes from another author should be in quotation marks To paraphrase means to restate the words of another author using your own words When you restate another author's original ideas in your document you don't use quotation marks But remember you still have to cite your source Why cite you might wonder why you need to cite resources that are free and posted openly on the internet Well, all information has value whether it's found in books or journals or freely available on the web People work to create it and that work should be acknowledged. It does not matter where you got the information It does not matter if you paid for it or not. You always have to say where you got it There are several reasons why you must cite your sources One is to let the reader know what the original source of your information is When you cite a source you are given credit to the original author or creator of the ideas or content you are using It shows that you respect and acknowledge the work and ideas of others by citing you acknowledge How other people's work ideas or experiences have influenced your own thinking Citing is a simple yet powerful way to lend authority to your work and let the reader know that you did your research When you cite reputable sources to support your statements, you're essentially saying I'm not making this up Experts or credible sources back me up This can make your argument more convincing Another reason to cite is to help your reader identify and retrieve sources that you found and follow the path of your information To verify the sources you've used Clear and traceable references provide the essential roadmap for others to explore Validate and built upon the ideas and discoveries presented in your work Citing is also the best way to avoid plagiarism You must give credit to the person whose ideas you have quoted summarized or paraphrased Not doing so constitutes plagiarism a serious form of academic misconduct and a violation of copyright law Where to cite there are two places in your paper where you need to cite your sources You need to cite your source in the body of your document. These are called in-text citations You also need to cite in a list of sources at the end of your paper Your list of sources at the end of your paper can be called a bibliography work cited list or reference list Style manuals and more help Style manuals are guides that tell you where and how to properly cite your sources There are several different style manuals. For example, MLA or APA Always ask your instructor which citation style to use on your assignments Have more questions about incorporating sources? Explore the learning portal or contact a librarian at your college library Happy citing


Down Dodge City and in the territory on West, there's just one way to handle the killers and the spoilers, and that's with the U.S. Marshal and the smell of gun smoke. Gun smoke, starring William Conrad, the story of the violence that moved West with Young America, and the story of a man who moved with it. I'm that man, Matt Dillon, the United States Marshal, the first man they look for and the last they want to meet. It's a chance they job, but it makes a man watchful, and a little lonely. Talking about Harvey, it's the far piece you had to reckon. If you ask me, it's about as far away as a man can get without starting back again. Yeah, and all this heat. Ain't this the dangest weather you ever seen? I don't know. I've seen dangers. Hey, looky there. Where? I'm appointment, ain't I? Oh. Why, it's a wagon and a bunch of people. Cooking food. It's only a man and a woman, Merle. Probably is right. Well, they're cooking food. Harvey, we ain't it in two days.  Come on. Here's my wife, Linda. Howdy. How do? What's that here cooking in the stew pot? Oh, just potatoes and beans. Mighty little of that. Sorry we can't offer you none. I wouldn't mind potatoes and beans, mister. I ain't it in two days. My twin brother here ain't it either. Sure am sorry, but there ain't even enough for me and the woman as it is. Had real bad luck finding beans. We was kind of counting on all that or something to eat, mister. And today's Sunday, too. Huh? Here. I'll get my dad with the bill. No. It just ain't enough, I tell you. Joe, maybe we'd better get him something. Merle, what, Harvey? There's too many miles to feed around here. That's the sure truth, mister. Now, mighty sorry. You better saddle up and move on. I don't know as we're ready. Look, mister. Now, don't reach for that rifle. It ain't fitting to handle weapons on a Sunday. Well, turn around, mister, and get out. I shot him. I reckon I did. I hurt him. He had no call to poke his weapon at us, had we. All we want is dinner. Joe. Oh, Joe.  Are we going to eat, Harvey? Joe. Might as well. Lady, lady, please. I'm kind of sorry about how things turned out. I'm kind of sorry. It makes us feel bad to hear you carrying on, Joe. I'm kind of sorry. Hear that, Joe? I'm kind of sorry. Sorry you're dead. Oh, now, lady. I just want to thank you, Joe. I just got so sorry, too. Where are you going, lady? Lady? The woman's going, Merle. She didn't tell me, Harvey. She just kept talking to herself and walked away. She's just as well. One thing I can't stand is not as a woman. Me, too. Hey, that looks good. No, dear, please. We can eat our fill now. Uh-huh. What about that team of horses? Tied up out over there. And we're all going to see them. Oh, never did hold for stealing. Remember? Oh, I wouldn't steal nothing, Harvey. I know that, but we can't leave them four beasts tied up. They'll be down right cool. Well, then turn them loose. I don't reckon these people don't need them any more, anyhow.  It's mighty tasty. Oh, hard to. That little marker fur grave, though. Somehow it kind of helps, don't it? Yeah, it does. Look down there, Mr. Dillon. Must be another immigrant heading west. That's a funny place to make camp, though.  Nobody wouldn't do a thing like that, Mr. Dillon. Would they? Let's take a look in the wagon beyond me. Ain't sure it don't make sense, does it? No, no, no. I think we can do nows and get him buried. Took you long enough. I don't want to have to go back for more in the dark, Harvey. Don't you like this camp, Murrow? Oh, it's a tolerable fair camp. Why? Well, what's that? That I've had by on him some little while. Good thing. Get down, stranger. Harvey and Murrow, see me? We're twins. Where you headed for, cowboy? Dodge. My golly, we've been trying to find dodge for days and days. Where is it at, anyways? What? About 40 miles, oh, your honor? Never mind. Where's Dodge? You seen any Indians, Grant? Ended. Any kind of Indians. Take it, there. Erwin, Murrow? Nah, Indians is all I like to us. What are you talking about?  Oh, never mind him, Harvey. Nobody out on the prairie knows nothing about where there's Indians. What do you want Indians for? I don't understand. You leave it to us what we want Indians for. You hungry? Oh, yeah. I reckon I could eat something. We shot a antelope this afternoon. It's been boiling in the pot there for quite a spell. Be ready for it all. Well, it might have good of you. Sit down, Grant. I'll get me a knife and poke around the pot some. Meat might be done already. I better see to my horse first. Oh, I'll steak him out for you, Grant. You're probably plum-tuckered out, right, Norway? Just pour yourself some coffee. Thank you. It used to hospitality like this. Being Sundays, we ain't hardly done a thing all day, Grant. That's right. That's what I forgot about Sunday. Man, all the rest one day a week. Our Paul always said so. He must have been a right religious man, your Paul. Yep. Sundays, he was always sleeping off a drone. He didn't hold a fighting and killing on the Sabbath. No. Well, last Sunday, somebody sure must have got drunk or something. Why? A family got wiped out east of here. Some pilgrims traveling to California. A man and his wife. Indians, huh? Dump, gun it. Why didn't you tell me they were the Indians around? I didn't say it was Indians. Fact is, I heard there wasn't a scalp to nothing. No. I declare, you can be plum disappointment. All I said was it must have been somebody drunk or crazy. Are you saying it was us, Janet? Me and Merle? No. Look here, man. I didn't say nothing like that. Well, it was. You did it? You killed him? Only killed a man. He ran off. Must have died. You're crazy. You knowed it all along, didn't you? That's why you come here. No, that ain't true. I didn't know anything. Look at me, boy. Got shot in his belly. He's killing him. You don't? Well, I had to, Merle. He found out we killed that fella last week. Don't know how he figured it, but he knowed too much. Got to talking real loose. I wish you'd told me, Harvey. Now I got his horse staked out and everything. Well, go turn him loose. Suffer is about ready. All right. Well, don't you start eating before I get back. I won't. What's it on? Thank you. Guess you heard about that cowboy they found out on the ferry, huh? Yeah. He was nearby and brought them both in today. There's nothing missing on that, kiddie. Not even the $30 he had in his pocket. Thank you. Maybe it was the same murderer who killed that man and his wife a couple of weeks ago, man. I don't know, kiddie. Three people murdered and there doesn't seem to be any reason for it. Well, it doesn't leave you much to go on, does it? Yeah, it sure doesn't. I sure did. What? We want to talk to you. Sit down, Merle. All right, Harvey. My name is Harvey Finney. My twin brother's name is Merle Finney. Nobody in this town knows nothing, Marshall. Well, I bet you two could teach him a lot. Look at her. I sure ain't never going to get married. You old ape. What do you men want anyway? Indians. We can't find no Indians nowhere, Marshall. No where at all. What are you going to do, join them? No, we ain't going to join them. We're going to kill them.  You've got to help us find some. Nobody else will. Uh-huh. Oh, maybe they got a reason not to, mister. Just because they're dumb.  I don't know what your reasons are, but I'm telling you to forget about killing Indians and go on back home where you belong. Oh, he ain't no more use in the rest of them, Harvey. Oh, wait a minute. I'm curious. Why do you men want to go Indian hunting anyway? We read it in a book. You what? Now that's a lie, Murrow. You know Paul never did hold with lying. Well, all right. We can't read, but a fellow back home, he told us about this book.  You came out here to murder Indians, is that it? No. Come on, Murrow.   Well, they act even worse than they look. Yeah. But you don't think you have to worry about anybody as brainless as they are, because they're subtle. I sure do, Kitty. It's the brainless ones that are the most dangerous.  I'm not charging you for this visit. People were supposed to socialize on Sunday. It just by luck to draw...  Chronicles. Chronicles. Well, are you holding something back, man? Did you used to be a preacher? What did I ever tell you, Doc? Why'd you quit? I lost my battle with the devil, Doc. The play was too small to support my gambling habit. That's fine, Tom. From the uphold of the law, the figure of justice, the civil of virtue... Let's go get a drink, then. Oh, that's a silly sign. How's it going? Yeah, what's the matter, Chester? Oh, them penny twins. Harvey and Murrow, there's going to be trouble, sure. What are they up to? Well, you know how they've been talking about finding an Indian's kill. Well, for a joke, some darn fool told them Moss Grimmie, the full-blooded Cherokee I'd just heard about. Oh, Moss Grimmie! You got the stable? It ain't funny, Doc.  Yeah, you're right, Chester. Let's get over there. I swear on anything to do with you.  She's on there up there and hanged off. They got a rope around Moss Grimmie's neck and up over that beam there fixing to push him off and hang him. Okay, Harvey. Your rope's just died. Let's get closer. You can't do this, Major. I've got a big family. Then your family ain't going to get no bigger. You murdered Cherokee. I'm not a Cherokee. I'm a Cherokee. Can't you do this? It don't matter what drives. Wrongs are Indians. Push him off, Harvey. Wait. Wait. You don't understand. He's got his own mixed up.  All right. Hold it, Harvey. You touch him and I'll shoot. Master. Master, thank heaven. I mean what I say, Harvey.  Yeah, I know. He was no good. Siding with the Indians. Moss Grimmie's no more rendian than you are. Now you got that rope off his neck. I ain't going to death.  You'll die for it, Harvey. No, you'll probably miss me from there. He's going to do it, Marshal. Harvey, now you'll listen to me. I'll make a deal with you. You come out to the edge of the law so I can talk to you. We were right.  Ain't too many guys. How could I be a wretch? Indians is powerful tricky. We know all about him. Now hold your breath. You're going now. Harvey, wait. Too late, Marshal. I said hold it, both of you! Did you hit him both, Mr. Dillon? I only winged him. Don't shoot no more. We're coming down. All right, take the rope off that man and hurry it up or I'll put another bullet on you. All right, all right. I'm taking it off. You let him come down first. Hey, maybe he weren't an Indian. Moss, are you okay? He's just a mite nervous. You better sit down over there. I'll catch you two. Come on, get on down here. All right. He busted my arm. We're going to jail? Yeah, your Indian hunting days are over for good, Merrill. Well, Doug's on. Then he made this whole trip for nothing. Never killed one Indian, Harvey. Just a few white people. What's that?  Why did you tell him that? I wasn't talking to him, Harvey. I was talking to you. You're the man who shot those immigrants, son. Well, Merrill done towards you and no use denying it now. That cowboy. You killed him too, didn't you? I didn't say that. You don't make no difference now, Merrill. It sure doesn't. You're going to hang anyway. Moss, I think I'd sooner not be hung if I had my sooners. But you haven't. Hey, wait a minute. Merrill, I just figured out what we'd done wrong. What, Harvey? We flung forgot about how it was Sunday, both times. That man and woman and then that cowboy. You know, Paul never did hold for killing people on Sunday by golly, you're right. And today's Sunday, too, Harvey. Well, I've all the dog-gun bad luck. All right, you two. The jail's on down the street. Just walk straight ahead. For Houston, directed by Norman McDonald, stars William Conrad as Matt Dillon, U.S. Marshall. The story was specially written for guns forked by John Messon. Featured in the cast were Joseph Kearn, Ralph Moody, Jack Moyles, James Nussar, Gene Bates, and Sam Edwards. Harley Bear as Chester, Howard McNeer as Doc, and Georgia Ellis as Kitty. This is George Walsh inviting you to join us again next week for another story on gun smoke.


Welcome everybody, thanks for being here. I'd like to start by saying that this is my first time at DEF CON, so I'm really happy to be here and really happy to be able to talk to you. I know that the title is a little bit generic, using the IoT in smart buildings, but that's intentional, that's because I want to take you through a journey of the research that we did in smart buildings that is culminating in, let's say, consumer grade IoT devices in smart buildings, but it started with industrial protocols with BACnet and all of that, and I'll take you through all of that, by showing you some of the attacks that we developed, and that's the main focus of the talk today. But first, a little bit about me. So I have a bachelor's and a master's in PhD in computer science, so I have an academic background, I have more than five years now in security doing some pen testing and mostly academic research also. Right now I'm a senior researcher at Forescout, and I do vulnerability research for OT and IoT environments and vices, and I also work with network monitoring and intrusion detection for OT and IoT. This presentation, of course, is not just my work, it's the result of more or less a couple of years of research with many collaborators at Forescout and also at the University of Eindhoven where I used to work before, so just bear in mind that this is not something that only I did. So a little bit of motivation, why are we talking about IoT in smart buildings and IoT in general? So what are the risks from IoT devices, right? So the number of devices in organizations is rapidly increasing and I think everybody is aware about that. IoT has already grown a lot in the past decades and is expected to reach, the numbers vary there, but people say between 20, 30, up to 50 billion connected devices in the next few years. Those devices are mostly unmanaged. They come from different vendors. It's not just the usual IT world anymore where we had Windows and Linux and maybe MacOS. So we have non-standard operating systems. We have a diversity of protocols. Many of them are insecure. There are infamous cases of insecure protocols in building automation. Some of these devices may dynamically connect to other devices. It could be inside or outside of the organization's network. And there's also another statistic there that by 2020, next year, more than 25% of identified attacks and enterprises will involve the IoT. This is a projection, but it's something that could happen. And there was this report released by Microsoft just last week about nation state attackers using IoT devices to infiltrate in corporations. They're using printers and voiceover IP phones and stuff like that and IP cameras, mostly. But let's talk then about smart buildings. So smart buildings is one of the places where the IoT is rapidly proliferating. So in the smart building, we have several functions. We have lifts. We have surveillance systems. We have fire alarm systems. We have lighting systems, HVAC, thermal stats, fans. We have data centers that have to be cooled. We have surveillance rooms to control the surveillance systems. And all of these are connected to a network and connected to building controllers. And why would anybody attack smart buildings? Well, first of all, smart buildings are not just shopping malls or residential buildings or something like that. They could be very critical buildings like airports, data centers, hospitals, public spaces, even factories. We're talking about, we're in the ICS village, right? So the factories that host the ICS devices can also be critical buildings that are smart buildings. These building automation systems and smart building networks are many times legacy systems. So some of those systems are decades old. There's many times no encryption or authentication. It's the same story as in ICS. But what's happening nowadays is that connectivity is increasing, right? So there's an increasing attack surface. There are default protocols and open ports, default passwords, weak passwords used in these devices. And all of that creates a perfect storm, let's say, for attackers. So OT, IT, and IoT are all converging in smart buildings. So if you look at a typical smart building network, you have a management layer where you have workstations, you have IoT platforms, and you have building management workstations that are connected and used to manage all of the field devices and the controller devices and all of that. And then you have separate subsystems, right? You have, for instance, video surveillance with IP cameras and NVRs and displays. And then you have the access control systems with badge readers and door locks and physical access control controllers. You have the HVAC system. You have the smart lighting system. And then you also have, in this whole scenario, the consumer grade IoT entry. So you have IoT gateways. You have things like smart plugs, smart TVs, wearables. In hospitals, you have medical devices, which many times are in the same networks as the building automation devices. You have other kinds of sensors and actuators and smart everything, basically. And although, we will see that later on, but although ideally, these networks would not be just flat networks and things should be isolated and all of that, in many cases, in practice, what we do see is just a flat network where everything is able to communicate to everything else, which is an ideal scenario for an attack. So what is the focus of our research? So we wanted to study vulnerabilities and attacks in smart buildings. And there are many attacks, attack vectors in smart buildings. And we were looking first at building automation systems of the legacy, protocols and devices, and then later on, the consumer grade IoT stuff. And then some of the attacks and areas that we imagined and some of the things that we implemented are, for instance, in HVAC, changing the temperature set points to, I think there was a presentation before talking about backnet and from the McAfee guys about backnet and how you can change temperature set points and HVAC devices so that basically data centers are taken offline or in hospitals, biological or medical material can suffer severe consequences. So you also have in the access control systems, you can add or delete authorized users. In the surveillance system, you can, for instance, delete or replay footage so that physical intrusions can happen. For the smart lighting systems, you can switch lights on or off or continuously on and off and basically make them unusable. And for other IoT systems, you can do things like denial of service. You can use them for information gathering, for pivoting in the network and all of that. So on the left-hand side there, what you can see is the lab that we built for this research. And you can see that there is a mix of legacy built-in automation stuff, so workstations and industrial-grade built-in automation controllers using protocols like backnet and Modbus and KNX and Longtalk and others. And you also have IP cameras and Philips Hue and Wi-Fi access points and stuff that are more consumer-grade IoT. So now I'll go through a list of attack scenarios and stuff that we implemented so that we can show some of these scenarios in practice, what are their effects and how to achieve those things. So we started as almost everybody who starts looking at smart buildings by looking at the backnet protocol, because it's really one of the major protocols used for interoperability in these systems. And it's one of the most used protocols in building automation. Nowadays it's infamous, because it's a clear-text protocol, there's mostly no authentication. Because of that, there are all the security threats, right? Or you can use it for reconnaissance. You can spoof devices. You can temper with variables, for instance, temperature set points or inputs and outputs and all that. You can do denial of service easily in many of these devices. So if you look at the right-hand side there, it's one of the parts of the lab that I showed before. And the little dollhouse is simulating a smart building where on the upper floor you have a motion sensor in the light, and on the lower floor you have a thermostat and a fan. And on the right-hand side you can see a lot of devices there. There's a network switch, and there's an HMI that's used to program the backnet devices. And there's another touchscreen device, which is actually a Raspberry Pi, simulating an attacker. And then on top you can see some backnet controllers. So the idea is that the usual functioning there is, yeah, you have a temperature set point, and then if the temperature ever goes above the set point, the fan starts working. And then if it lowers down, the fan stops working. And when there's presence detected in the house, the light goes off. But you can easily DOS the backnet router, which is the blue device in the middle there on the top. And the effect that you have when you run a simple DOS, so you're basically just flooding the device with many packets. And you can see that the packet's being sent there by the attacker. And you will see that basically the router will stop working, and then the error LED will go on. And yeah, that means that the device is not active anymore, and then it should be controlling the light. But it doesn't, right? So the light doesn't work anymore. So in this case it's controlling the lights, but it could be controlling HVAC or any other of the subsystems that I mentioned before. So OK, we can do that with using the backnet protocol and with building automation devices. But then we wanted to look at another attack scenario, which is basically, OK, if we have many vulnerable devices, can we create a chain on attack path, for instance, where a remote attacker could reach the devices that are actually controlling, for instance, the access control function in this case? So another thing we did is, in that lab, we did some vulnerability research. We disclosed some CVs. We found around, from not mistaken, eight CVs. And most of them were actually related to the web applications running in these devices. So I think almost all of them had cross-site scripting, and some of them had command injections. This is very common in this kind of environment. And we also found a buffer overflow that allowed for remote code execution. So we basically changed an attack where the steps are those on the bottom there. So first, we gained access remotely using an IP camera. Then we performed some lateral movement to the management workstation. Then from there, we moved to the access control PLC using the buffer overflow. We execute our payload over there. And then the idea is to persist also in that. And that was achieved basically by exploiting mostly the access control PLC, which has a web application and a specific building automation framework, Java framework running on that, and runs on top of the QNX operating system. So once we have that buffer overflow, we can basically leave off the land there on the device and use some of the devices on functions to upload our malware and persist. And then implement our final attack, which is to open or close doors depending on what we want to achieve. So for instance, here you can allow and authorize people to access sensitive locations, imagine that in a hospital or in an airport or something like that. Or you could also not allow somebody to access a specific location. So for instance, you combine both attacks, and you have an HVAC not working in the data center, and then you lock the door, and then nobody can go in and fix the issue. So then we actually looked at how many of those devices can be found online. And this is data from last year. So this we did last year. And we saw around 23,000 devices of the ones, the same models that we were looking at in our research. And we saw that around 40% of those, almost 40% of those, were vulnerable to the remote code execution, the buffer overflow and the remote code execution that we found. So that means that there are more than 9,000 devices available on the internet that's from Shodan that could be completely owned from the outside. So then we saw, OK, so we have problems with the protocols. We have problems with the devices in the building automation side, right, in the older stuff, let's say. And we have a lot of those available online. But what we have, what we really can easily find online are IoT devices and consumer-grade IoT devices and smart cameras, for instance, and such kind of devices. So we wanted to look at these IoT devices and what are the impacts that they have on smart buildings. So in this attack that I described now, we use the IP camera, for instance, as an entry point. But what else could we do with the IP camera, right? Or with the lighting system and all the other systems that I described. So the next three scenarios are about the IoT smart buildings. So our goal was to investigate the IoT protocols and devices used in the smart building. So some of the examples of the protocols and devices that I'll be talking about in the next scenarios are RTP and RTSP for video surveillance, the Philips Hill Smart Light for the lighting system, and MQTT for the IoT system. And we divided our findings there in three categories. So things that we found on the field. So basically by talking to customers and partners and by looking at the networks of these customers. Things that we found on the web, mostly on Shodan, and things that we found on our lab. So the setup of the lab is this is just a simplified version of the same figure that we had in the beginning.  So let's talk about the findings on the field. So when we look at smart building networks with consumer grade IoT devices, how are these networks configured in practice? So first thing we realize is that many times these internal networks are not well segmented. So you do have one switch that has IP cameras and IT devices and an OT and building automation and everything talking to each other, not well segmented at all. So you have unwanted communication links between all these categories, right, IT, OT, IoT caused by misconfigurations. There are many unwanted services and insecure protocols enabled. So it's easy to find Telnet, FTP, UPNP. And many times the facility managers or the people who are managing those networks are not even aware that those things are activated because they come activated by default. The insecure protocols come activated by default and that's it. There are many times weak passwords to access devices. There are devices with no vulnerabilities. And there's a lot of use of unencrypted traffic. So I already mentioned RTP and RTSP for the video streams, and I'll talk about this in details. And you also have, for instance, HTTP and MQTT in clear text for IoT devices. So then what do we find on the web? So we were talking about the unencrypted protocols, right? So if we look at devices with clear text or TSP on the web, we find more than 4.5 million devices. And those are mostly IP cameras, but there are also other devices that use this kind of protocol. If we look specifically for the Philips Hue, for instance, for the smart lighting system, we found almost 10,000 devices online. And if you look at clear text MQTT, which is used for general IoT, you have more than 75,000 devices. And these data are from this year, actually, just from last month, because this is one of the latest things that we had been looking at. So let's go in details in these three next-to-text scenarios, right? So just in general, what are the findings of the labs? So again, this is the lab that we had before. Now we have an internal attacker. And what we want to achieve there is, for the video surveillance, we want to do a footage replace or a Hollywood style, highest movie kind of scenario where we are showing one thing to the security operator. But actually, the camera is streaming something else. So you can steal something from the bank.  You can also use, in the case of the Philips Hue, something that's interesting, you can just reconfigure the platform to use that as a pivot point inside of the network. And in the case of MQTT and IoT devices, you can use that for, for instance, information gathering and denial of service. And these are the attacks that we implemented. So for the attack scenario number three for the video surveillance system, again, this is just a simplified version of the other picture. You have the IP cameras that are using RTP and RTSP to communicate with the network video recorder. And the network video recorder is connected to the network switch, and then you have a storage server and then local monitoring to look at the video being streamed. So what's the problem with RTSP and RTP? So they are streaming protocols. One is usually over TCP. The other one over UDP. One is to send the data. The other is to control the session, basically. RTSP is very similar to HTTP. And it's used to control the stream. And it's basically mandatory to establish a session before you start streaming via RTP. And RTP is designed for a real-time transfer of usually audio and video data. It's unidirectional from a server, which is the camera to a client, which is a network video recorder. And there are secure versions. And you can use the GLS, for instance. But this is rarely used in practice. So what is the anatomy of an attack here? Because you're using these unencrypted protocols and you are, in turn, in the network, you can just establish a man in the middle. If you drop the traffic and record the video footage, it's unencrypted. It's a clear text. You can replace a heartbeat RTSP command. Usually it's a get-per-end command with a tear down. So you stop the connection between the camera and the NVR.  So we do replay a pre-captured stream to the NVR. So in practice, this is how it looks like. So you have the surveillance footage. And this is what the camera is normally streaming. And then in the attacker machine, you run attrcap to do a man in the middle. So you select the hosts. And then we have another script, which is actually recording the footage and replaying the pre-recorded footage when the session is stopped. Yeah, and now basically the session with the camera has stopped. And then now we are replaying something that was captured before. And then on the bigger screen, you can see what is actually going on. And on the smaller screen, you can see what the security operator or the surveillance guard would be looking at. So this is another attack scenario, attacking the Phillips Hue. And the Phillips Hue is interesting because it's a very common IoT device, which until last year didn't support HTTPS at all. It was only supporting HTTP. And a lot of people knew about that. And it was very easy to attack a Phillips Hue. And then they issued a former update last year. But basically, what is used to validate your communication with the Hue is an authorization token. And that can be just sniffed also from the network because it's in clear text. And then an attacker can just control the camera. So you just send a put request there for the state of the camera so you can do on or off. I just saw that I have like three minutes left, so I'll run through the slides. So you can just switch the lights on or off. And then as a result, you can have something like this. So it's basically that the light is completely unusable. So the last attack scenario that you have is MQTT, which is a published subscribe protocol used for IoT devices where you have a broker that coordinates the communication between a set of publishers and subscribers. And that's used to exchange sensory information, telemetry information, and also to issue commands to some devices. And again, it's usually used in clear text. So you can just passively sniff the traffic. Or you can subscribe to some topics. And you can use that for information gathering to understand the devices that are in the network. You can subscribe to wildcard topics. So you can see basically every topic that a broker is handling. It's very easy to do denial of service also, because you have quality of service that can be increased. So you amplify the attack with the responses that are required. And you can use very heavy payload. So it's a very common protocol in IoT. But again, if used insecurely, it's very easy to attack it. So in mind, with all these scenarios of attacks, let's very quickly talk about defending the IoT. So the things that we should do so that those attacks are mitigated or cannot happen. So the first thing is that we need network visibility. So you need to know what devices are on the network so that you can know if they're vulnerable or not or what kinds of protocols they are talking, what devices are communicating to what other devices, and so on, which allows you to do proper asset management, so proper configuration and patching of the devices. Network segmentation is extremely important, so you can isolate problems and reduce the attack surface. Sometimes there are devices that are difficult to patch, so at least they should be segmented. And as a last point, network monitoring and intrusion detection, you should basically know what's going on at the network in real time and be able to spot anomalies and attacks happening. So more details about these attacks and about the defenses and all of that are in four papers that we have over there. The top two papers are actually academic publications, and they deal mostly with the backnet stuff. And they deal mostly actually with the defensive side, so how to do intrusion detection for backnet. But we also describe the attacks. And the two white papers on the bottom are commercial white papers. So they describe most of these attacks and areas that I showed here, and also a little bit about defense. So what are the key takeaways from this talk? Abusing the IoT in smart buildings, it's easy, it's cheap, can have severe consequences, right? Like we've seen with HVAC, with Access Control, where you can let the attacker basically do physical intrusions. And defending the IoT in smart buildings, it's doable, and it starts with understanding what's in the network and monitor, right? And as a last takeaway, I just want to say that the building automation technologies is old, but we are moving to more connected smart cities and connected everything, so we should always keep in mind that more connection means an increase that our surface and the consequences of these attacks also. So sorry if I went over time. Thanks.


Hello everyone, thanks for coming My name is Pablo. I'm one of the funders and the CTO at Octeto Octeto is a development platform to deploy remote development environments on Kubernetes and we will talk about that a little bit later And hi, I'm Vinay. I work for future way technologies It's the research armor for parent company Huawei and I've been working on Kubernetes my interests are in Kubernetes network and computer and lately ebpf and We hope to have a good talk with you today. Thank you Cool. So this is the agenda of the talk We are going to introduce the idea of cloud native development environments The problems that they solve and also some of their challenges. One of them is to make them a cost effective So we will analyze the different challenges there and then we will introduce a feature coming soon It's in place pot resize and the idea with this feature is that you can modify the request and the limits of a running pot without restarting the pot After that, we will see a demo using in place pot resize and ebpf to optimize the infrastructure Utilization of cloud native development environments and we will finalize with some takeaways Okay, so let's talk about cloud native development environments and first. Let's see the current state of the earth so most companies are moving to Kubernetes and microservices So most companies are moving to Kubernetes and microservices for the right reasons, right? They solve many problems in production environments But they also come with some challenges and one of them is that They make make it harder to mimic your production environment in your local Dev environment Even with tools like Docker local Kubernetes distributions like mini cube You need to install this software run all these microservices So there are several issues there one of them is that you make run out of CPU and memory in your laptop So things go very slow or even stop working You Need to maintain local configurations. There is something wrong in your Dev environment. It's very difficult to replicate The same problem in other laptop or it's very hard for anyone else to troubleshoot. What is the issue? And if you are using for example multi repo, it's not trivial to orchestrate The build push and deploy of all these containers running locally So what most people do is to assume an environment disparity between Dev environments and production And I think this is wrong Because at the end what you are doing is shifting right your testing efforts Ideally a developer should be able to deploy a Dev environment and test end-to-end all the changes that they are doing before even sending a pull request but If you are not able to do that you need to send a PR Wait for the continuous integration job to validate your changes or even worse You need to merge and wait for an staging or integration environment to be accredited and then do the final end-to-end test there And if there is an issue you need to start again Work on your local Dev environments and appear and all those things so all this cycle Reduces the developer productivity a lot and also the developer happiness So the solution that we propose Is cloud native development environments and this is a high-level view of this methodology The idea is to have a single Kubernetes cluster served by all the developers and Every developer is working on a different name a space And you can have as much isolation you need between them a spaces using the standard Kubernetes objects So the idea is that on each name a space the developer is able to deploy a full replica of the application Which is much more realistic because this is running in Kubernetes with your hand charts Using your network configuration security policies the same thing that you do in production And it's fully replicable because it doesn't depend on your local configuration You should be able to deploy any commit from any git repository and everything happens in the cluster in the cloud So it doesn't depend on your local configuration and anyone in your team can go there and check it out because it's available for everyone So in order to Adop this methodology is very important that for the developer the deaf experience is the same and to do that there are several open source projects like Otheto the le presence till garden and many more and More or less the goal of these deaf tools is to provide this experience. So the developer Keeps working locally on their IDE And the application is hot reloading on remote immediately and you can even set Configure your debugger set breakpoints and all those things. So that's key for developer adoption If you need to change the developer workflow people is not going to adopt it, but With these tools you can have realistic replicable and Fast iteration remote deaf environments So that's the solution and I'm going to talk about one of the problems with this approach and is that You need to run all these environments in the cloud and it could be an impact in your cloud build, right? Which is scary But good news is that Kubernetes is very good To optimize resource allocation. That means that all the containers running in the same Note they may serve the CPU and the memory available in the note So for example, if you are building your application and you need an spike for CPU and memory The memory and the CPU of the note is available for your container And when you are done the same CPU and memory is available for other containers So that is very helpful to optimize your Utilization and also if you need more notes in your cluster the cluster will allow the scale up and down as needed automatically So that's good news, but let's analyze different use cases for development environments to see if this is really helpful in our use case so this is a workload in production and It's more or less using two CPUs all the time So we are able to set the CPU request to two CPUs And in this case is working very good If there are more incoming requests the deployment will scale of it horizontally. So that's good But for development is a little bit different what usually happens is that the when the application is booting You need more CPU and memory to put your to bootstrap your your application and then after a while In development, you don't usually have that many incoming requests, right? So after a while you need less CPU and memory to keep your application running So in this case there is already The vertical pod out the scaler in the Kubernetes community And the idea is that the vertical pod out the scaler monitors your container CPU and memory usage And then it will update the requests and limits of your containers based on the real usage of your container Here I'm assuming That in place pod resize is available because otherwise Here when the requests are updated the container will restart and then you would have the booting time Booting CPU again, but let's assume that in place pod resize is available But even with that when you are developing There are random spikes and they tend to be short in time For example, you are working on your application editing your code. You don't need too much CPU to do that But then you need to build your application with for example Make command so for the main command you need more CPU But the vertical pod out the scaler is reactive So there is a delay between just start using more CPU and Until the VP and the vertical pod out the scaler updates the request of the pot. So in this case I don't have Enough requests here to run my main command then the requests are updated, but I'm not using this CPU here Then the vertical pod out the scaler updated request again, and I don't have two CPUs available for the main command So there is a delay. So it's not really working for for this scenario Ideally would we would like something like this something that is proactive and updates the CPU requests of my container in real time basically or milliseconds So in our case when we are started up there, though, we basically run a dedicated cluster for every customer and when we started of that, though, we were using a standard Kubernetes request and limits and Using that we were able to run eight pods per note Which is not cool because We are using VMs with four CPUs and 32 gigabytes of memory and we were not utilizing all the resources So what we did is an ad hoc solution for this is a custom scheduler basically and with that solution We thought I don't have time to talk about but with that solution We were able to run 80 ports per note without affecting the developer experience. So this is huge is 10x infra savings and And there is a lot of potential to solve this problem properly our ad hoc solution is not ideal It requires a lot of effort to maintain the solution and to configure the solution for different customers But it shows as the potential of solving this problem So the question is if there is a better way to solve this and with that in mind I'm gonna pass to Vinay to talk about in place for resize and ebpf Thank You Pablo those great So in place for resize Can I get a show off hands? How many people here are aware that this feature is in the works? A few okay, then those few probably know that this PR is moving at light speeds Okay, well not exactly light speeds we're being extra cautious and there are good reasons for that the PR is big And it touches critical components and Kubernetes and mistakes can be costly So it's imperative that we stage it in a responsible way. What really matters is that we get across that finish line Hopefully in 126, I don't know We just released the continuity 169 which is needed for this and so this PR could merge any day now in the next few years If you guys are really nice to me, I will leave this PR to you in my will Okay enough grief for my PR. So let's take a look at what really changed The first thing we did was we made you have the container spec in that there's a resources field We made it mutable for CPU and memory when you do that What it lets you do is it lets you update send a patch to your part spec saying I want the container I started out with one CPU, but I want to that's you're expressing desired resources for the part and then Kubernetes goes to work doing what it does best, which is you know get the actual state equal to the desired state so With that with the ability to specify a desired state of resources We need a way to signal the user who's specified the request What's the status of the request and for that we introduced a new field called resize in the pod status? This status holds one of these four values when you have a pending request for resize The interesting one is in progress where which should be the default case That means that the kubelet was able to allocate the memory or CPU that you wanted And it's working on it with the runtime to make it happen Proposed as the initial state where when your API server looks at your request it make sure that the request is valid you're not exceeding like your requests are not exceeding limits and stuff like that and Infeasible is the user may not know okay there is a node that has four CPUs They may ask okay my part needs five. It's never gonna happen so that's a signal that you may need to Evict your part and ask the scheduler to schedule a new instance to another node where you can get five CPUs and Deferred is the case where the node has six CPUs, but another part is using two CPUs there and It's possible, but just not now. It's your choice. Do you want to wait or do you want to evict and get five CPUs elsewhere? We added a couple of more fields the resources allocated field in the container status It's a persistent way of telling what is it that the in the in progress pod status is driving towards and The resources field in the container status I know there are a lot of these fields in here But the resources field in the container status tells you the actual state as reported to us by the runtime So that's what is actually on the container on your containers Lastly we have a new field called resize policy. This was introduced for one reason There are some legacy applications where like Java applications which are using the xmx flag They're not able to take advantage of increased capacity of the pod without restarting. It's just the those applications need them So you have we want to give the users a way to specify. Hey, this is My application needs to be restarted The default is restart not required where we will try to we will try to resize your pod without restarting It doesn't guarantee. It's not a guarantee, but Kubernetes will try its best So what does it really involve? I touched upon this earlier. We have changes to the API server cooblet scheduler and runtime There is a lot to go into here, but I will focus on one thing the cooblet So cooblet admission of a resize request is interesting here because this PR introduces a new race condition where the resize request is racing with the pod that may just have been scheduled to that node and Cooblet is the ultimate authority and it will it's a gatekeeper It checks to make sure that at any point of time the requested resources is not exceeding what's available So one of the two requests might fail if there is a contention Going to one of the best ways to really look at how this works is to go through it step-by-step. So let's do that now Consider this example here what we have here is a node where you have a pod with 40 mcp use allocated to it Now you desire to give it 80 mcp use so it starts with a patch to the pod spec the API server validates this pod spec The patch and then updates the object store in this case We update the CD now the next step is watch is triggered for the scheduler and cooblet the scheduler Takes that looks at that watch and updates its pod cache and uses that to do the Do the max of desired and actual so that it doesn't compete with resize that's going that's in progress And in this case, let's assume that the cooblet was able to successfully allocate the the resources that was requested 80 mcp use so it does the admit pod resize it succeeds and It immediately patches the pod status with the pod status resize field saying I was able to give you 80 mcp use and The status of your resize is in progress and I'm going to be working with the runtime to make it happen now in this case we are increasing the CPU so the pod see group see see group settings are initial are Set first and then it goes and talks to container D the runtime Why are the update container resources CRI API asking container D to allocate 80 mcp use for the pods containers and now the container D goes to work it updates the configuration for the pod and Next time a container status CRI API comes in it reports back to the cooblet saying yes Your pod its containers have 80 mcp use that triggers a generation of update status update to the pod Where it patches the pod status saying the resize is now complete the pod has 80 mcp use and we have come full circle So that is the happy golden path now One there is a nuance to this that I want to touch upon your pod may have more than one container and You may be resizing more than one container at the same time You may be giving more resources to some and taking more taking resources away from other containers In all these cases what we do is we order the resize such that the decreases for container resize are done before the increases are invoked and If there is a net increase to the pod see group values due to the resize then the pod see groups are ordered first And then the containers are resized and why simmers are Why is this important? Let's take the scenario you have a system where you have two gig of memory available to the pods and You have one single pod running on that system It has two containers there they take one gig each and we desire to give C1 1.5 gig and cut memory for C2 make it 0.5 gig now the pod Hasn't changed the pods resources are the same But if we did C1 and then C2 we are oversubscribed and this request will fail and the pod will end up in a bad state It's for this reason that we want the runtimes to not only support update container resources CRI API but do so in a synchronous and transactional manner if we don't do this then The down downstream request to update the policy groups might fail and the part will be in a bad state And when I say synchronous and transactional what I mean here is that don't queue a task and say okay I will apply this later We need that yeah or nay whether it succeeded or it failed and if it failed What's the reason for failure in the context of that update container resources CRI API call? one other thing that got introduced to the To the container status CRI API is a resources field This is what allows the container runtime to tell us to tell Kublet what resources are actually configured on the pod I guess you're starting to see why this feature is a little complex, right? And Lastly there is C group V2. It's here New more and more OSS are shifting to C group V2. There are a bunch of desirable features to this For us in particular we're interested in the ability to specify memory requests at the container level We did not have that in C group V1. There is another talk. I believe it's on Friday It's by David Porter and Rinal Patel about C group V2. I highly recommend attending that if you can Okay, so this is the fun stuff we What can we do with eBPF for in-place pottery size as Pablo has laid out we have a problem here We have the use case here with the remote depth environments There is this workload where there are spikes and the reactive approach that we have with VPA is Not good enough. So How do we how can eBPF help? Let's take a look Consider this pod here. It's called kubil pod with one container kubil container and as you might guess we use this spot to Exec into the pod and edit code and build code It's a pretty good Example that mimics the remote dev environment except that in the remote dev you might do our sync of local code to remote What's interesting here is that when we look at the requests the resources that we have requested We're requesting for CPUs, which is good enough to you know edit code and build But we have curated and cherry picked a value of 50 meg for memory That is sufficient to edit code and edit code with a lightweight editor like vi But not enough to build if you try to hit the make command with this much of memory The oom killer will come along and take care of things It's gonna kill your process or it's gonna run really slow. So What can eBPF do? Well, let's take a look What you're looking at here is this 20 odd lines of Python code. That's all we need. That's it You go to the node on which the pod is running and tell Python. Hey run this for me and you're set Looking more closely. There are two parts to it. The first is the eBPF code itself This code attaches to the exec exec ve system call That's the system call through which all the commands in the system that are executed go through you do LS It goes through exec you do make it goes through LS it goes through exec and The second part is well when this When this eBPF program sees those commands it traces it it traces it to a trace file and the second part of this is that we Watch the trace file and if we see make in the trace file, then we resize our pod to 5 gig That's it. It's very simple, isn't it? Well, not so much This the only good thing about this code is that it fits on one PowerPoint slide it's It's very inefficient everything that you're doing the system is getting traced with trace file I actually tried it. It's really slow and it's of limited use So you're in your container. You're happy writing code somebody else doesn't make another container in your resize. That's not great, right?  So BPF also offers this facility of maps Which is a way for Talking to the eBPF program and having it be more configurable than this What we're gonna do is we're gonna use the BPF maps to tell the eBPF program to focus on only specific containers And in that container focus on specific commands. How do we do that? The containers have C group ID. So we specify the C group ID as a key in the eBPF in the eBPF maps and In the value we specify the list of commands that we're interested in tracing This will let us only trace the commands from the containers that we are interested in That's good, but how does the user tell this to us? Well We resort to the good old annotations We've defined annotation called eBPF resize which contains C name the container name that we're interested in and The commands that we're interested in in this case make and of course what do you want to resize to? So we specify these three things with these three things the pod watcher thread or process can configure the BPF maps And the program can trace exactly what we need With that all we need to do is initiate the resize when we see the trace and Of course, we are lazy Kubernetes people We are gonna go to each node and say hey Python run this code. Hey Python run this code We're gonna ask Kubernetes to do it we hand it to Damon said and Kubernetes does the hard work for us. I Have code up on GitHub. It's demo code only so don't run it in production So you can take a look at it and see how this works. It's a prototype It brings us to the showtime Okay, I'm gonna switch to the The screen here Can everyone see the the terminal screens? Okay all the way in the back. This is the hardest part to get okay great Okay, what you're looking at here is is a local all these terminals are SSH into a local VM called eBPF resize and We're running a local Kubernetes cluster a local cluster here with in-place pod resize feature gate enable That means we can resize our pods without restarting the power containers The first step is to of course deploy the build pod and I'm doing that here with this YAML deployed It's gonna take a moment. There it is. So we have the pod running here the could build pod We can execute into this part eight code and build code and we have this Window up here shows the stats. It's using it's using 600 It's using under one a be a memory and then you have we can do one more thing here We get the cube build pod in json format and query the status resources allocated the new field that we introduced for this feature that tells us What the cubelet has allocated for your pods pod in its containers in this case? We are focusing on the build container that's container zero now Let's take a look at Let's edit some code so I'm gonna exec into the build pod and LS this is the Kubernetes 125 release branch that I've gotten the pod for this demo Get status shows us that if you're looking at the memory values. We are at 42 meg It's under what we so we have 50. We're under that and we use lightweight editor like Wim and Edit some code So this does not look right. Kubernetes also known as k8s. We need to fix this. So let's fix it All right, that looks much better, isn't it? Okay, so the memory usage is 30 mag. We're good. So what happens when you try to build this? Let's try. There's one way to find out So it's gonna run really. Oh, so it got killed. Well, that was fast This is expected, right? We have 50 mag and make is trying to allocate more than 50 and we just don't have it And the own killer came along came along and pretty much told us to go kick rocks So what can we do? We have choices. We have options. We can use We can schedule the pod with 5 gig and you won't have to worry about it. Of course your bank account won't be happy and You could use you could rely on VPA when VPA sees these womb events out of memory events It will resize the pod for you, but it's not great developer experience So there is another way fortunately, let's take a look at what eBPF can do for us BPF tool map list just shows that map entries. I'm gonna use this later But first is to deploy the demon set. So with this command, I'm gonna deploy the demon set It's gonna show up here. It takes a moment to initialize now It's running we can know that the program is up and running by looking using the BPF tool to list the map entries So I'm gonna run that again. So there is a map entry here now We can get into the details of this by doing a dump of this map entry 67 So what we see here is a key one four seven five two That is the C group ID of this container in which I just executed the make command which you know caught killed by own killer and It's telling the BPF program. Hey, if you see this container execute a make command then please trace it So now let's try this again. Let's see if it works. Oh Please keep an eye on this window the top left top right corner window with the 15 meg value That's being watched as I hit the make command. Here we go So we're at 5 gig and the container seems to be happy It's allocating whatever it needs and the make is making much better progress than it did before So that my friends is the magic of ebpf for you Okay, let me take a moment to thank the demo gods here for saving the surprises for another day and So what did we what did we see today? We saw that? We have this use case as Pablo described where make spikes that happen in the development environment They need a much more responsive resize and that's not always possible with that what with what we currently have and We saw that ebpf programs can help here by almost instantaneously resizing the pod for you to your needs and So to recap We feel that cloud native development environments are the future and because they're cost-effective there They give you a production like environment and all teams work on the same config which saves on testing costs and production issues we we Want the runtimes to be able to support? In-place resize if you're working with runtime or if you're a maintainer Please consider adding support for in place resize to your runtime We don't want the user to have to worry about whether whether this works with my runtime or not We want it to just work. It shouldn't even be something that people think about that's where we want to be and Try this feature gate out when this makes it in please turn on the feature gate hammer on it beat up on the feature We want to find those corner case issues. It's important to find it's important to handle the use case as well but it's also important to gracefully handle the abuse cases and lastly try out ebpf Look, I'm I'm no ebpf expert Not even close, but we managed to cobble together this little improvement in a short time and This improvement got us from being Reactive to being proactive for this use case it took us from you know tens of hundreds of seconds of response time to like a sub second response time and And that little improvement is actually a paradigm shift now And it's great when small improvements lead to amplified gains. So yeah, try out ebpf. It just might solve problems for you We have one more slide to go and It's the one that everyone is waiting for The title says it all right so Please scan this QR code It will take you to a place where you can live you can leave feedback for us and please tell us what you found useful and more importantly what we could do better and We love to hear from you. We love the feedback So and thank you for being here. Thank you for being such a great audience. It's been a pleasure With that I will conclude this talk


Oh dude, that's a good one! Oh my god! Oh my god! I've got to bring my wood in, because it is about to start storming. I can actually hear the rain across the road right now, so it's going to start pouring down any moment. I want to go fishing today, and I wasn't going to go really early, but I saw the weather, and obviously I was going to get poured on, so I'm going to wait till after. I wonder what the fish are going to be doing after a thunderstorm. It hasn't rained in forever. Hopefully the raindrops are going to be cool, and they're just going to be coming up to the top, just to try to get a little sweet nectar of cooler water after being boiled for a couple of weeks. Look at the leaves falling for goodness sakes. Alright, I've got to hurry. It is finally clearing up. Y'all know what that means. It is time to go do some fishing. Just fishing, period. I've been working on wood. I've been doing home projects, honey-dos. I'm ready to just wet a line. So Lunkers is on his way right now. He's caught some bluegill. We're going to use some bluegill as some baitfish. We're going to do a little bass fishing, some jugging, and some scouting of just new summer fishing spots. Thank you, Lord, for not being 116 degrees today. Let's go enjoy the outdoors, y'all.  It's the special tail, guys. He's caught bass. He's been bitten, but he was hanging on like a monkey there. Hope you guys are catching Mondos on these Mondo worms. Yep, that's the first time I've ever seen a wing-wing detector on the deck of the boat. That's a nice addition. Just in case we run out of hooks, we're going to find some. Time to go fishing them wrong leg. You know it's a good mood when you're singing. Like it. Like it, Steven. Getting down. Got the crew here. So we got our lines. We got our jugs. Lunkers was nice enough to get us some delicious catfish candy. Catfish, they love themselves bluegill. That's like a nice juicy steak. We're going to head out and do a little bit of that. Maybe a little bass fishing. Do a mix. I don't know. We're just going fishing. Guys, I left the plug on. Oh, no. What just happened? Yeah. Well, someone left it. Oh, yeah. He's coming in. He's coming in hot. Look at this guy. He's like, Hey, guys, we're backing up. Don't worry. We just had a mishap. Forgot to put the plug in. Yeah. I'm more worried about the wood. That guy was doing a full set. He was full. He was like, Oh, yeah. This is a five-lane boat ramp. I'm going to take this one spot. He's going straight after where he's at. Your thing is sticky. I know. It's sticky. I pulled two four-inch long pieces of fiberglass out of it the other day. That's the thing about a new boat. Are you recording? Yeah. I'm still recording. So the thing about new boats is like, all the fiberglass that they had, like the boats made of fiberglass, it's all still down in. They're still trying to come out because it's still a new boat. And when you put it in, it's like. That sound like Popeye. For sure. That nice new boat smell. It works its way back to the bung hole of the boat. Oh, what a good day. Well, yeah. Anything else you'd like to add to this moment? No, this is a good start on that. It's so much fun. Like a stick of gum. Okay. I need to move this metal, Zachar. Here, take that. Yeah. That'll be fine. Yeah, they're coming up. Oh my. Sorry. I'm just, I'm excited. Look at him. Look, look, look, look, look. Oh my God. Oh, those are some decent pops in there. Those are some doosies over there. I'm seeing them everywhere, dude. Even the birds getting out there. He's like, holy crap, I got to get me some of that. Let me get off this stump. And look at this guy. He's like full speed ahead. Like I got to get, look at the shad. Look at the shad, dude. Am I recording? I am recording. Yes. Get down. Oh my. Whoa. Smoked. We've got too many things going on. These are nice keepers though. I mean, we're out here to get meat, so. You want to eat them? These are nice ones, dude. I'm not going to turn it down. Oh, Alex has got one. White bass king is here. I'm here. First time? First time. Is this your first time? Yeah. Awesome. Sand bass king is here. You're getting to experience some good stuff, everyone. Hitting it hard. I can feel their little mouths trying to wallop it. Feel small ones trying to get it. Ooh, that's a big shad getting busted there. Wow. Wow. Look at that. That's so cool. Look at that. He's doing backflips. I got one. Oh, my gosh. That is so awesome, man. Backflip and shad out of the water. I got a little guy here. Oh, wow. Yeah. He housed it too. This is how I caught my first fish ever. I just slapped my Snoopy pole rod on top of the water and caught a fish. There. This is a keeper. Whoa, what about the water? Yeah. Everywhere. Hey, all around. 360. Oh, this is a catch. It's a 360 catch. Oh, I got one too. Oh, that's a good one. I got a good one. Oh, you did get a good one. Yeah. Okay, we're doing things. I said when I woke up this morning, I was like, I've been working on wood for so long. I just want to go fishing. I don't care what's biting. Everything's biting. Look at it. Look at the bounty. He's going to get out the crappie special. Look at this.  It's more fun just watching them smash. Oh, I got a fish on the team and tying it around. I got a Marlin. Look at that. Wow. That is an actual fight. Oh, come on. Hit my bait. Dude, look at it. Look at here, man. Whoa, look at that. Four pound test. Look at it.  Look at that. I got to get crushed. Can you guys hear this audio? I mean, it's just suction. It's all it is. It's suction. I'm hooked up, baby. Come on. Come on. That's a good one. That's a good one. Hey, Alex has got a great one, too. Oh, party fell.  They only have to be 10 inches to keep. So we're catching like real nice ones. Oh, I got a runner. He's not very good. I mean, I'm throwing the tiniest thing. I shouldn't really expect much. Oh, my God. Freshwater Bonita. It is. Freshwater Bonita. Oh, my gosh. Look at this little rod, man. Oh, my gosh. Look at this bait ball.  Oh, my God. Look at him. Wow. That is insane. They're coming right towards us. Wow. I got it. I've never seen that. Look at that. Wow. Look at that bait ball right there. I'm hooked up. Oh, I can't even lift this guy. I just caught that one on four pound test, dude. Dude, look at that bait ball right there. Oh, my God. Look at it behind us. They're everywhere like a squirrel. Just hang tight. Dude, I can't even set the hook with this rod. It's just like they got a... He's weird. I mean, normally it's like sundown, sunup kind of thing. This is like just broad daylight. Wow. This one's definitely not a keeper. Oh, yeah. Yompa four pound test. This is probably the funnest rod I've ever experienced. It is a good one. Boat flipping him on the yomps. Can't do it. Yeah, we were going to have deer meat tacos, but this is just going to take precedence. Last little nugget. There he is. Oh, is it going to be a keeper or not? Either way, I enjoy his presence. Oron. I think if there was one lure, I had to bring to a survival situation. It'd be a spoon. A spoon? A spoon. Oh, my gosh. Look at the Mondo-ness. That's a toad. We're going to have to keep you. Just popping shads. Dude, look at him jumping shad out of the water. That is a good one. Hey, that's a keeper. We have absolutely skewered the wet bass. I'm bringing that word back. Skewered. That was so fun. I needed that when I woke up this morning. It's like, I just need to go catch something. I just need to go fishing. So incredible fun times. They're still biting, but we're going to move on to the catfish game. So we've got these bluegills, sunfish species. We're going to chop some up. We're going to put them on jugs. I think this is going to be a really great area because all these white bass and hybrids feeding around here, what do the catfish do? They go down to the bottom. They wait for those easy scraps because they're lazy sons of guns. So that's the theory anyway. We're going to see if that works out. There are thousands of... I just can't stop looking at them. They're everywhere. It's amazing. It feels like we're in a tune-up. This is crazy. It just keeps going. The catfish lines are out. We can go fish for white bass because we're so much fun and tasty. I'm probably going to metal the deck. Rob's probably going to metal the deck. That's where he's at in life. That's okay. I've worked on wood projects. That's it. Okay, good. We're just going to go three sections. The filet mignon of the catfish world. That's a porterhouse, T-bone, tomahawk steak. Just going to go with that circle hook right in there. Right through the back. I don't want to fill up the gap of that circle hook too much because that's how they get hooked. Okay, bombs away. I think that might be perfect. That is perfect. Perfect depth. Win. You've got to be kidding me. I just threw my weight in. Wonderful. I forgot to snap my thing. Just went through the bottom. Hey, there was no life-giving moment right there. Always snap your snaps. It's kind of what they're for. It's time for Rob to go do his thing. Do what he does. He does the wands, looks for things. So we're trying to find him a spot where he can go do that while we're waiting for the catfish line to go out. Go down. It's a win-win.  All right. Let's put the hammer down. Hey, good luck, man. Thank you. You need like a good saying like wand on or something like that. Wand on. Good. It's going to need a flying leap. One of those good lunker originals. That's more than enough. Oh, I'd say you did well, man. Hook this gizzard shad right in the back. Let's see what happens here. Just throw him on out. Well, Rob is still doing his detecting. His wanding. Still wanding on. Finding a ton of weights and stuff like that. Anyway, you can go check out his treasures on his channel, I suppose. So here's that dime I was talking about. This color is silver. I didn't even look at it. I didn't look at the year because look at the coloring on it. What year is it? 92. Is it really 92? It is. Dang. Hopefully we have some KitKats to be found. Clump KitKats, little joys. Free floater jug. I don't think it's moved. So I don't think it's got anything on it. But we do have to check our other lines in there in prime locales. We got a storm rolling in behind us, too. I think it's time to roll and see if we got any surprises. Jug lining and metal detecting both have surprises on the ends of the units.  It's just got crustacean stuff all over it. I was going to let you grab it. I was going to go ahead and just start wrapping it up. Yeah, just go ahead and wrap that thing up if you don't mind. It was a long shot, but we figured we'd give it a go while you were wanding. Dude, it got taken! Woo! It did get taken. How does that make you feel? Dang, that sucks, bro.  Steven, what's your excitement level? Um, can't wait to see it and then hear it. Are you ready to check them? My face is ready. See if anything's on there. You feel it? Yep, there is. You do feel? Yeah, there is. He just tugged 114%. Is it still? Um, it's very heavy. It's a fish. It's a catfish on there. Oh yeah, it's a blue. It's a catfish. And we got a blue, baby. It's a blue catfish. That's awesome. Okay, I got... Ow! Did you get hooked? Oh, underneath the fingernail. Okay. Oh, listen to him honk. Okay, guys, this is a nice catfish that is under the 30 inch to 45 inch slot, so he is going in with our... We have a plethora going on here, a fish. A few jug lines working out, guys. We didn't even have them out here this that long. Just a couple of hours. And they were getting dangled hard. That one just went down. It wasn't down before. Are you ready for this one? There it is. Yeah. Are those fish? Those are fish. And we have the dangler. Guys, this one just went down. Like we were getting the other one off and now this one's an automatic little... You think? I don't know. I mean, it could be. Oh, there's fish. There's fish? Oh, I see it. It's moving. Dude! We are stroking them. These circle hooks, they just don't go anywhere. They're ruthless. Croak for me. Woo, baby. Oh, my gosh. I'm going to be working the play knife tonight. Give that one a little check. There's not a single ounce of movement on it. No movement, huh? Oh, dang it, dude. I'm going to reset it over here. Just want to let you guys know when you open your dip bait and there's maggots in there, that's extra goodies. Okay, don't be discouraged. Just go in and give it a plunge. I think I have one in my shoe. It's in my toe. I'm serious. It's in between my toes. While I'm staring at them, I'm definitely going to take a gander. Marie Kitty, how many white bass do you think we got in there? Mm-hmm. Yeah, that's your put in the cooler. Oh, I don't know. You've got to give you guys one last dangle update. We've got a couple of cats and we got a piece of cut bait out on a regular rod and reel. And we're just waiting for some of the other ones to go active. We're going to stay out here a little bit past dark and see if any of the other ones go down. But we've got so much meat, you guys. Great, fantastic fishing day. Been incredible. Lots of dangling. Lots of meat. Oh, my gosh. What mic was it on? Your mic wasn't on. It wasn't on. Hey, guys. My name is Subtitle. What's going on? Large excitement. Woo! I could literally hear it. I was like, what is that knocking noise? Clink. That thing has moved probably 40 yards. I'm going to get it. Come on. Come on. There could be something serious on here. I got it. Here we go. What do you feel? Like what your initial reactions? There's head shakes. Yeah, we're in 28 foot here. I don't know if it's giant, though. It doesn't feel huge, not real heavy.  He must have just moved it. Maybe he was just a super active cat. Oh, my gosh.  That cat is the one that moved all this. That's a decent little channel, though. He's a tasty one. That is a good eater, though, right there, man. That's a seizure. That is some flair. That's some flair action. Look at that. Toss her in. Here we go, kitties. Is it hooked in your trolling motor? No. Uh-uh. He can't hold it down. Did you see that? No, absolutely. He took it down good. You see, he heard the trolling motor. Dude, we've got more cats on here. Look at him. Come on now, baby. Is he going to go down? He took that under for a while. This might be a decent one. Come on, be decent. The last one tricked us. I think there's one on there, buddy. I'm going to move this out of your way. There has to be one on there. There's no way it dove that far. I'm tired now. He's on there's fishing. Oh, yeah, there's fishing. Let me start a wrap here. You have the wrap going? Do you want me to do this? Yeah. Oh, gosh, yeah. That's definitely a fish. Oh, dude. That's a good one. Oh, my God. Oh, my God. Oh, my God.  What? Oh, my God.  No. What did I just say? No.   I said, I'll cry if we caught one like that. Dude, this is a giant. That's a 10-pounder, bro. Oh, my God. Ragley, no way that that just happened. That's a 10. Oh, my God. Got the line out so I can see if I can get it. Oh, my gosh, bro. Look at that thing. That is a freaking huge one. It is huge. Okay, we got it. That's a bucket, dude.  We got it. Oh, that's a nine-plus. Oh, my God. Absolutely. He's still, he's still, he's way out. Okay, he's good. Yeah, he's good. Okay, I'm going to reset. Oh, my gosh, dude. That is a hammer. That's a nine, not a nine-pounder at least. I think so. 9.21. Dude, it's heavy. 9.19. 9.19. 9.19. 9.19 is what it locked in as. This is going to go in the Guggen Cave somewhere. This is hilarious. Let's get quick measurements. He's short. He's a 23. 23. Actually, yeah, he's 23. 23. Then let's get girth.  He's not messing around in the summertime. 18 inches around. 23.18. Okay, here we go. 23.18. We're all about fish conservation. There you go. So far, we're good.  He's good. Get back down. Come on, swim bladder. I think he's going to be fine. He's going to kick. He's kicking strong. He's kicking strong. He's good. Swim bladder. Keep that light on there. It's gone. It's gone. It's gone. Into the night. Insane. I cannot believe we almost had a double-digit bass on a Dagum drugline. It ate cut bait. Cut bait. That one's getting mounted. Don't you say, Rob? I'm going to hit you with that right now. Just give me. 1 billion percent that's getting mounted. 100 billion percent. It's going to be a funny, funny mount because that's the biggest bass I've ever caught on a Dagum line. But we still got a lot of work to do. We got to go back to the house and we got to get the knives out and really start cutting up all the fish we've got. We've got probably 20 pounds of fish.  Incredible day. Just a day. So this is the bounty. It's been one of the best meat hauls we slayed. It wasn't even really catching him. It wasn't even fishing. Touched it. I'm a full sniff. I had to get in there close. I have nicked my finger, ladies and gentlemen. This knife is a little sharp. That look is just had to leave. It is midnight. Still cleaning fish, y'all. Still going strong. Got a few more left. I'll tell you what. The best way to do these is let them sit. Let them sit overnight or at least like four or five hours on ice. Meat is a lot easier to clean when it's chilled. If you haven't seen the Buffalo recipe, stay tuned because we're going to do that on another video. A couple more here. We're going to be done. Look at all these filets right here. And there we go.  We absolutely crushed them today. I'm going to do the kitty cats, but I'm going to let them chill overnight. They survive in the lab well much longer. We're actually going to stop this video right here, guys, because there's so much insanity that has gone on. Boom! After the rainstorm, fish are just schooling everywhere.  We already had some traw lines ready to go from Rob's bluegill catches. So we got those out there. It didn't take long for those to get hit. We started getting the catfish and then a giant bass came up and hit one of those bluegills. I'm almost embarrassed to say that's my biggest bass out of that lake. We caught it on a dagum jugline. And it's actually not legal to keep game fish like largemouth and strippers that you catch on juglines. Obviously we're going to let that fish go anyway. Just letting you guys know if you catch fish like that, you got to let them go if you're catching them on a jugline. You have to, have to, have to hit the like button. Make sure to leave a comment down below. Just tell us how crazy you think that was. Oh man, it's just an awesome day guys. Thanks for being here for it and I'll see ya.