45 minutes I know it's so serene above the moon Above the moon These shots are why it took two years Yeah It's gonna be a jagged pipe up above No! No! No leverage Just in the castle, right?   I know It's gonna be Brown Throw the javelin in there Let's get out of there You gotta throw it Don't do it He knows Person is gonna lie Because he just did it I swear the heck Rayna scripted television show in history of TV Did anybody in any way, shape or form think that the war against the Night Walkers would end with this episode? No, absolutely not Not freaking women I thought they'd possibly win I thought this would end with them possibly winning I thought everyone was going to retreat Yeah, retreat to King's Landing To King's Landing But then watching this I was like, how the heck are you supposed to get away from this? Yeah, I didn't That was an epic RSA I thought Arya was dead When she turned to grab her I was like, bye Arya It's always the little ones So we didn't lose anybody except Jorah We lost Jorah And we lost Melisandre She wasn't huge We still haven't lost A huge character yet Jorah is the biggest Jorah is the biggest loss If we lose a dragon, I couldn't tell I don't know No, because It was Drodon who was being covered up and he got away and came back So Jorah's fine Yeah, they were fighting each other You wanted it, Jorah Did the green one get killed? So Jorah fell off, right? I don't know Holy cow, my heart is like Oh, that was So much stress There's three episodes left I don't know I can't imagine I can never guess It's not the most high stress one I don't know how we're all going to do this This was way more stressful than the Battle of the Bastards It was stressful from the time it started Well, it's because he was going up against something that was non-human With everybody you love in potential death And there wasn't any breaks There was never a break This was an entire war Holy cow This was a war movie It was just a movie about war I can guarantee I've never been more stressed or invested in 90 minutes of anything caught on film I've never I've never been more invested for 90 minutes and stressed out And there's three episodes left I told you this show is going to send people in a therapy This was the first Lord of the Rings and there's three more It's the greatest TV show that's ever been created The theory about Cersei is that one of her brothers will kill her It'll either be It doesn't matter There's a prophecy Just like it said that her three will be the hand of her brother and her three children will be covered in golden so the prophecy of her three dead kids has come true But I'm excited because it's a lot more excited than it could be It could be more impactful if it's genuine But then who's going to sit on the throne John Because you know Danny isn't going to let it be anybody but Danny You would think I mean the rifle air is John We all know that He's the bloodline Well my theory is done after this episode Yes it is because your theory is stupid The Night King I thought it was going to be total desolation there would be no throne everything would be destroyed but the Night King would die and Westeros would have to just start to scratch I cannot believe Cersei sitting off in Westeros just like oh it's not a big deal Wait a minute We also lost Theon Yeah we lost Theon That sucked And Brandon you're such a Brandon is a dick But wait a minute He just sat there the whole time No because Brandon What was he doing? What was he doing? What was he doing? He looked into the future Guys he knew how everything was going to end Where did the birds go?  We won't know now If she was Or did she just know that's what he said Because she is Or is it the Melisandre that died? Maybe We still did not I thought for sure that we were going to reveal That she was At that moment Either she did Either she did And they just let her die and they didn't reveal Or they're maybe Or we'll find out later That would be stupid She said the exact same The same stuff Before Arya died And in a way that was almost like she had insider information They had a few looks Like they'd said it to each other before Yeah But Arya had never said that to Arya No, but Arya looked at her like anything And what do we just exactly like he did And Arya didn't have There was nothing She didn't flinch How did she know that? Three or four This was third There's six episodes, three left Wow, that sucks What's going to happen? There's going to be a battle for the body Someone's going to sit on the throne There's going to be a battle for Three This freaking episode Who else here fought Sansa and Tyrion? No, I did too My map was dying I was dying You cannot kill Tyrion until the last episode If you watch Tyrion and Sansa die, I will be a mess That's the only way I'll be finding Tyrion dying As if it's in the last episode Of the episode Of the last episode That was going to piss me off I'm still going strong But holy cow That was my favorite episode ever That was insane I don't know how many times I can watch that episode I know I so much want to watch that With people who go through it for the first time And watch their reactions That's the brilliance of the show Is that people who have never seen it I want to watch it I thought so many characters died when they didn't I thought Brienne and Jaime were going to die at that wall I thought they both died multiple times Which they would have Arya hadn't killed Guys, Arya has killed a freaking night king Arya is going to be the one on the throne She deserves it She deserves it I know she has no claim But she deserves it If she wants a claim, she's going to keep it Jon and Daenerys at the end Are going to get on the throne Beco regions and look at each other and turn to Arya Because this actually belongs to you We're going to have a little finger death It's going to be like And she'll pull up her mouth


Going on everyone Sam here with late sports cards. We're in 23 Bowman's best baseball hobby 8 box case number 35 Pick your team style read off the list for names that are teams. Yes g-men exactly I will as soon as I'm done reading the names I'll type in like I'll type in guesses start now and hit enter when you see that come through and chat Just anybody in the checklist just guess drop the name if we happen to pull a super that person Then you will get a hundred fifty dollars break credit So you got Adam M with the Brewers Alan R twins Anton H. Padres Brandon S. Reds Astros Charles F. Cubs Chris H. Angels Christian Rangers Christopher M with the athletics Daniel F. Mariners Daniel F. O. Mariners Daniel F. E. with the Royals Harold S has the White Sox Rockies Pirates Jack S with the Guardians Blue Jays Geron T Rays Jesse S. Cardinals John G. Phillies Josh S. Braves Larry S with the Marlins We've got Lucas R. D. Bax and Orioles Mitchell K. Yanks Robert S. Matt Scott F. Giants and Dodgers Shagon R. U. Red Sox Steve B. with the Nats and William T. Tigers So I'm gonna say like I said again everybody guesses starts now when I type it in you will have 30 seconds to guess a Superfracture in the checklist of Bowman's best get them in get them in now Come D. With a leaf art a sports pop. There we go. I love it All right, a lot a lot of uh guesses coming in. This is awesome. A couple seconds left guys about 10 seconds to go everybody 10 seconds All right, I'm going to type in guesses are closed when I type that in Guesses will be closed so those five four three two and one there you go We are really we are actually we are overdue for a super to be honest A bit make sure you hit the like button too if you guys are enjoying the stream. So here we go Bowman's best time Good luck everybody. Yeah, I'll be a little behind going to the next one I should be able to catch up on Don Russ and do that fairly quickly and then we'll have Bowman's best university and phoenix divisions everybody The super Chris Bryant's gonna get full where you're not in chat probably Probably that's just going to be a sign to stop guessing at that point of on Guess Christian Yelich the break before we fold a Christian Yelich super Really? Yeah. Also, what day in November is your birthday? The sixth the sixth one not the 24th All right We're close That same month All right, let's do it I think I shan't share a birthday with Drake. I think I don't know pretty cool There's many diamonds. Davis Martin champagne poppy Bass and bass that's follower of blatant sports cars Is that's ice? Yes, pateese. That's Corbin carol 32 of 50 Lava randy johnson auto congrats arizona. That's nasty man. What a start great start. That's so freaking cool hell, yeah 32 of 50 Corbin get that straight to the mag. That's really sick Real to real of Juan soto green a volt be to 99 and von grissen. It still cracks me up randy johnson's like, hey This big glove here. Let's take pictures. Yeah, sit there. Everybody line up Corbin you take a seat Yeah, it's a nice card man. That is a really nice card ring cones to uh, the fillies nice one there fillies and Sure, yeah, I'll take it Riley green even uh, what's it called and he said these sell like 500 bucks a piece That's pretty awesome. That's clean dude. That is clean. Those are cool, dude Beautiful. I love when we pull the autos. That's obviously that is by far the best one Not a big glove just a little man. Yeah, that's actually a regular size glove Selective max clerk Gee man, man, is he actually moves that's pretty cool Bass auto of jacob melton for the astros got drew jones Wave of ryan rockio spencer jones That is davis martin blue gunner henderson to 150 Oh nice Do you want to have a similar 99s and down or uh, yeah, that's fine refractor auto cam collier going to the reds Went through the shrinking machine at willy wilkis factory That is christopher morrell and Abbey's One in 705 packs will take that plus to make a Corbin two man. They're hard to hit Honey, I shrunk the Corbin shellac of matt Shaw Through a brady auto in here by an accident. Honestly steve that would bode well for you That's two of ten louise guanipa going to the braves on the red auto Good stuff there. That's awesome, dude make that one I would say so. Yeah Very nice. Congrats on that one. Uh, bravo Very good first, uh overall box. Oh excellent. Okay, gotcha gotcha gotcha cam collier. That makes sense though, timmy the 250 is manumichado and joshyum Yeah, his case is solid, man Great start jd with miami st. John f a u. Thank you, brother And then george l picking up a couple of mixer spots later tonight on says and christian yellich base auto willy fannis for the mets and jeremy panya And zikil tovah, maybe next year griffy will get his photo set Now he's a photographer Who is it griffy? Yeah, that's what uh, well vaughn said that apparently next year they're gonna have his stuff in there Take your time there. Yeah, that's kind of cool. Yeah base auto of josh casavich for blue jays and zack netto Wave is anthony voltby rookie j rod Him is gunner henderson and that's on that. Let's go man. There you go orioles I think uh, I forget who it is. I think it was michael's something had the diamond backs and the orioles Uh, that's max clarkton 99 and cj abram. So let me double check on the name. Uh, no, it's lucas r lucas r Excuse me had the uh orioles and the diamond backs in this break So good stuff lucas base auto lubbing espinoza going to the cubs Solid case so far. Yeah, very solid case Very solid case again seven to go and don russe everybody. That's random team style manual avonia and rike bradfield Random team stile Yeah, one of 192 so about a case about a case case and a half when you get a hen, which is nice We have seen cases with none of them in it With no hens in it, which is good. Those are the uh, super short prints and this year's bowman's best uh, gorman camilo diaz base auto For the astros adley rushman refractor real real freddy freeman and corbin carol refractor if he as well That mervis oscar colas the 299 and matt mervis mervis swerve Base auto of sam zavala. Nice one there for the padres. That's drays And joey vato Corbin carol masterpieces a jordan walker and louisa rise And sec everybody base and base That is canard elgato base auto for the inks And jusson crawford As a wave of rogic arias and one soto And juvonne and air judge leover baguero Uh, two one 99 that is nice riley green astral projections auto 82 and 99 for the tigers Congrats on that and oswald peraza iruiz and uri perez purple is To 250 at michael harris to 150 carlos whorehead for the reds and anthony bolby Drew jones and james outman James outman Caleb killian and christen for morel base auto javier osario for the tigers nice one there detroit trish and cassis cassis michael harris rookie and ron rocunia michael harris astral projections rookie that is And you know, it'll probably be about 10 minutes or so behind going into dawn rest, but We'll be able to pretty much catch right back off And for to go and then eat to go as well for to go and then eat to go as romantes francisco lendoa Gold lava to 75 carlos whore. Hey, congrats reds. Cool one there Take that all day man.  Hold on everybody.  So We are two to go on dawn rest everybody two to go on dawn rest. That is it. That is all two three and eight Gabriel gonzalez real real of spencer jones to 50 on the lava and brian rock. Yeah Is jackson holliday tovar and max clerk base auto michael arroyo for the mariners and roger arias Ziko tovar and give gonzalez anthony arizzo The randy. Yeah, if we make latent latent employee cards, it'll all be randy johnson poses ambior stavaris and the braves. We're gonna all sit on that club together. Yeah Pops gloves real real gladi jr. My trout green mini diamonds two 99s colson montgomery. We got base auto of d on whorehead and then a hunter brown mullets and jeans Shohei masterpieces base and base that is aqua lava two at one 99 adley ruchman and gunner refractor rookie Yeah, there you go turtle neck chain and fanny pack like the rock i'm sewing on that richard railing tovira That is for the orials leaning on a map in safety Actually funny enough that you say that richard. That's many machado at my other when I worked at my old job To my company party.  Now sam's of all though to 150 congrats on that one at the Uh padres calling how best hit from stadium club. We hit a uh redemption for a j-rod going yard uh orange autograph That is two 99 of joey vato and a uh stadium club in 3d j-rod as well A lot of j-rods today Yeah, j can pose with an ice cream and a microwave with some glasses on and Okay, well he also puts ice cubes in milk. Is that weird? kind of I mean what why To keep it just just like a glass two 99 is uh matt mclean like he regularly puts milk on ice Not less like a regular glass of milk like you can't have this glass of milk without no, I mean ryan cliford refractor Nice one there for the astros. Sometimes it's okay. Okay I mean, thank you russ Get back to work jumbo rando maia Riley green. Yeah, it's gonna be fun, man. It's gonna be a lot of fun. Shane Blue auto marion void the 150 for the fillies congrats philadelphia This one there you have shane picking up a couple of spots in that very nice christopher with a dawn rusk spot and danie selling us out on dawn rusk Let's talk to your shita. Thank you very much guys sold out on dawn rusk. We got three to go and we got eight to go Shita gold lava to 75 yordana alvarez And gabby marion doubted. Yes, you do fmj. Yes, you do You can come back with brain damage mini diamonds eerie perez Base and gray rod that is a green miggy 299 for the targets base auto colby thomas going to the a's And show hey uh james outman Wave and jordan walker Yeah, as i've gotten older i've drank less and less milk for sure. I'm not a huge milk fan I used to drink a lot when i was younger Like a full school and stuff. I need a milk with warm cookies if they're fresh baked cookies need milk 250 Going to the guardian solar freeman Nice one there. Yes, I agree with that now. Now. We're just uh Now we're just ripping on jake for putting ice cubes in his milk On our brown and tommy trod What freaks me out more people who pour milk into the bowl before the suit. Yeah, that's weird That's that's that's chaotic That's enough to call like orange riley green call the police the 25 Congrats on that targets and madeline It's white laneford on the shellac as well riley green base auto louise laura going to the uh brew crew Nice one there and raffi. There was thank you. Shames picking up the uh raptors in a break Well, how is the eye now? It wasn't that game was not his fault, dude. I'm tired of people putting the blame on him Yeah, that was not his fault cunha Yeah, peanut butter jelly and milk is really good. That's a very good like lunch. Yes. I love that like a summer lunch Just had this yes, let's go rust fire though green auto chase bow into 99. There you go pirates And kasey schmidt see that's valid boost Ripple to 250 of brando maya. We're allergic to milk peanut butter and eggs when you were a kid RPC come on allergic to the whole yeah What do you what you eat the food bread? Yeah, the food pyramid you're allergic to Red has eggs in it.  Grass and water man. There we go hard water. I'm 99 base auto manual bonilla Was that okay fair? I was a born too much early valid That changes things That is max clerk Nolan gorman that is oscar colas base and base photo synthesis.  It's a big photosynthesis guy anthony big on the photosynthesis refractor davis and goody eras and jaco He's in on different oreos in the milk. Absolutely brooksley and jr. Kamenado Drop guys this link really quick to phoenix divisions phoenix divvies everybody It's so many carrots. I've been orange tint to my skin.  We eat berries and twigs hunter gatherer. What are you hunting? Goldfish That's base auto randed a hasus Yeah, milk's got to be cold if it's if it's like room anywhere near room temperature Like trout and kodaisenga But i'm also not like nursing a glass of milk for hours on end to where it's going to get warm No, exactly. It turns out the cookie milk for me. I think you should just not be a baby and drink your milk quicker That is two two ninety nine match shop base auto you entry Vargas Jake's punching the air, right?  Drink your milk faster. Sorry my god Blue milk, wait, whoa, whoa, blue milk at disney isn't that like the star wars thing? Blue milk, isn't it? Yeah, isn't there was a thing in star wars where like blue skywalker was drinking blue milk blue milk, really? Yeah, cheese, dude A bit messed up as well. Yeah, two percent. Let's go two percent The soy milk fans are hiding right now. Oh, yeah Thin oreos are 150 roger gargoyles double stuff, bro Yeah, that luke that's kind of a hot take thin oreos. I mean, I don't dislike the thin oreos Los romantes base auto for the Mariners their double stuff is now what the regular stuffing was a decade ago There's iri Perez. Not iri Perez estuary ruiz estuary ruiz Bantamilk Lugano hoppy and demis and de la saunas. Yeah, bantamilk. Oh my god Not a big bantamilk guy. I I feel like green auto of brando. Mya tonight, you know Brando milk let's say de pala I think milk should just be one color and one color only I mean chocolate milk is nice, but two at 199 is christian becaro and it could be chocolate or white. That's it You only strawberry milk, right to 99 and under we've got miguel cabrera Uh colson montgomery anthony vulpe max clark to at 75 is in yordan Spencer jumps to 50 to 25 is riley green Uh first before we continue. Thank you dean for a spot in random, uh hit style and also richard with a spot in random hit style of leaf Tonight we've got lazaro montez randy de hasus Emmanuel bonilla louise lara colby thomas braylon tavera ambiora stavarez michael royo Javier osario candor delgado at samsevala Milo Diaz Ludwig espinoza joss casavich Willi fannas gaberan cones jacob melton davison gudierrez dean horre juan chivargas ryan clifford kim collier to 250 tyler freeman the 150 sam zavala amaryan boyd carlos horre to 99 riley green randon maia jace bowen 275 carlos horre We add the hymn of gunner henderson That is a red louise guanipa two of ten And then a corbin carol to 50 32 of 50 That'll do it for the break everybody. Thanks again for the fill up next is uh


I am Alan Cecil. With me here is Dan. Dan Petro. And we're going to present on you're doing IoT. Well, you're doing it wrong. You say it. You're doing it wrong. This is great. Thanks again to Defcon for having us. We're really happy to be here. This is a pretty big issue. One of the things that's sort of unique in the IoT landscape is that there's not very much tying IoT devices together. They're very heterogenous as a group. It's hardly a group at all. They share almost nothing together in terms of hardware or software. So it's pretty unlikely. It's pretty rare for a single-like issue, a single vulnerability, per se, to affect nearly all IoT devices. And I've got bad news for you. So before we talk much about random number generators, let's talk first about random numbers. So random numbers are kind of a stand-in for just about anything that a computer might need. Obviously, numbers are kind of how computers work. And so a random number generator doesn't necessarily have to make numbers exactly, right? We're talking about encryption keys, authentication tokens, and lots and lots of other things that could be covered under business logic. Yeah. So there's some interesting things about random numbers. So computers are notoriously bad at making random numbers. For the task stuff I do, I rely on them being bad at this.  So if you need to actually get a random number out of a computer, you need some kind of hardware RNG to make entropy. Which should solve the problem of getting random numbers, right? Yeah. So that's how we started all this obviously was sort of thinking that hardware random number generators surely are the gold standard for generating entropy, for generating random numbers. Oh, also as a quick aside, we'll be using the word entropy around, throwing that word around. That's not actually going to be important that you understand the technical meaning of it. Just sort of assume that it means some quantifiable amount of random numbers. So like you have 121, like 128 bits of entropy, that means a thing. But it's not really important for the purposes of this talk to dig into that much further. So there are two kind of broad categories of random number generators that we're going to care about here. There's the pseudo random number generators, which are software and true random number generators, TRNGs. In particular, I hate the term true random number generators because it sort of implies a certain quality to them, right? You might think of this, you might look at this and say, well, if I want a secure device, surely I want a true random number generator.  I just don't want pseudo random numbers. I want true random numbers, right? The names are leading you astray here. Don't get too caught up in the names. Really the only distinction is one is made in hardware and one is made in software. Really they should be calling these hardware random number generators. That's a much better name.  So within pseudo random number generators, there's two kind of basic kinds. There's like the regular ones, so like the Mersenne twister, Lib Z Rand, all kinds of algorithms basically that start with a seed, an initial number, and then from that produce an endless stream of bits there forward. The regular ones are not meant to be secure. They just like make good random numbers for ordinary, non-secure usages, things like the Mersenne twister. And then there's a cryptographically secure pseudo random number generators, or SISPINGERGA for short, that are like cryptographically secure, meaning that there's no way to take the output from it and derive any of the other inputs from it, any of the previous values or the seed from it. Alright, so hardware design.  It turns out it's actually really hard to do this properly, and lots of times you might think that you're doing it and doing it wrong. One of the other troubles about this though is that trying to get any information on how any particular device works, a particular like IoT sock actually does its random numbers, is really hard.  So the one exception to this is the STM32 from ST Microelectronics. They actually really great information upon some of which this is based on how to generate random numbers. So we'll go into two really good examples here. The first is using like an analog circuit method. But this isn't an exact logic diagram that you see on the screen here, but basically the way that it works is that you have like a loop in a circuit that has like a knot gate in it. And it's just going to kind of loop back and forth over and over again. It's not clocked by anything. And so the rate at which it's going to spin going 0 and 1, 0 and 1 or depending how you implement it, it could actually be intermediate values in the middle actually. The rate at which it's spinning is not clocked. So it's just going to kind of be an arbitrary, not necessarily random, but arbitrary and sometimes fluctuating frequency. And so if you just kind of pull, if you just ask it, hey, at any given moment in time, hey, what are you, a 0 or 1? It's going to be kind of arbitrary, not necessarily random, right? But certainly arbitrary. And then there's other methods too. Yeah, the other way to do it is with two clocks that are running at different speeds. So yeah, two independent clock sources, say two quartz crystals. And you're measuring the difference between the two because those two clocks are not tied together. There will always be a little bit of a difference in time between the two. So you can measure the delta, the difference between the two and get a normalized distribution effectively. This is oddly something that I struggle with on the tool assisted speedrun side when I'm trying to run stuff on a Super Nintendo because they happen to use two different clocks, one for the main CPU, one for the audio processor. They run at different speeds and it makes it very hard to get deterministic results. So unintentionally in a video game console of all things. But most of the time you'll see these implemented in just a black box. You can't really see what's going on inside of it. You just get the result out of it. Yeah, so there's a number of issues that can run up from some of these designs, right? So you imagine the first method, the analog clock method. So you're just like you have this circuit that's looping, right? Well, what if it's the case that your CPU is much faster than that loop is? You might ask it, hey, what is the value of it? Okay, it's a one right now. And then really quickly afterward you ask it a second time and they didn't get a chance to loop around. It's going to give you a one and then a one again and then a one again. Or what if you're like the clock that you're asking it at just happens to be at a multiple of the frequency of the thing, right? You're always going to get the same answer every single time you ask it. So this isn't actually a really hard problem.  And lava lamps and lava lamps, you know, but in the IoT space, things have to be really cheap obviously. So these are two really common designs that you're likely to see. Sure. So in the year 2021, most devices have a most IoT socks that you'll buy us a system on a chip, how you start an IoT device will have a hardware RNG peripheral on them. That is a like a piece of device, a piece of hardware as a device that's made to do this one and only one thing. Surely you must think that like that's fine, right? That's super secure like what's the worst that could happen. The problem starts running in that there's no operating systems in IoT devices typically that really what you're doing is just writing C code that runs on bare metal. When you talk to a peripheral like any other kind of peripheral, you the operating, you the user of this device has to write code to talk to that individual peripheral at the hardware level. C is hardly even, C is usually derided as a portable assembly, but it's not even portable in the IoT world that like trying to run it against a different device is not even a really a thing that works. So this is not any different in the world of hardware RNG peripherals. You basically are given something like this. There's a how function, a hardware abstraction layer function that says here's how you call the RNG. It's a C function that looks something like this. So if you're not familiar with C, there's a function name that's how get random number. This is just pseudo code. Obviously every device is going to call it something slightly different. It'll have an output variable which is a bit weird if you're not familiar with C. This will come up later, so keep in mind this, but you give it a pointer to a region in memory that says here's where you should put the results. So it's an output variable. It's usually a 32 bit number. And then it will return an error code. Whatever that happens to be a boolean, unsigned, 8 bit integer, whatever that happens to be. So I'm going to let him take the rest of this part. Oh, you're sure. So of those two parts, like there's really two things we care about here, right? That output variable and return code. So you can ask the natural question, which is the first question that we thought of upon doing those research was, there's an error code, huh? Like, there's a lot of things that can go wrong when you're calling a piece of hardware. Like, what if the hardware just isn't ready yet? Or what if, like, it was busy? Or what if, like, the relative positions of Jupiter and Saturn aren't aligned for all we know? Like, the hardware could just have an error. And like, what happens if it errors? And what happened? How many people out there are actually checking this code? It turns out almost nobody checks the return code of the hardware RNG function. These are just two examples. Yeah. So what we're finding is that code online repeatedly just doesn't even bother to check the error code. So what happens when you don't check the error code? Well, undefined behavior happens. It's not written down what will happen. So usually undefined behavior is undesirable. So there's a couple of interesting things that can go wrong. Usually you are going to get either partial entropy. Like, some of it is random. Some of it isn't. But one of the interesting results is the number zero. A lot. There's an example down here where the first call got random numbers. The second call, still random numbers. The third and fourth call, they're just zero. And you might not necessarily notice this when you're looking at a file. It might not necessarily pop out at you right away. You can get things that look like they're pretty random, but they're full of zeros. This is the classic XKCD joke of dice roll. Okay, we're just going to return a static value. Just a constant. Here you go. Oh, actually here. Oh, you have one thing. So there's, the XKCD joke is sort of a joke, but it's also kind of an actual real problem. If you see a number like we have to deal with here, right? How do you know that those zeros are, like, not random? The sort of left with this, like, deeply existential problem, which if you try thinking about it too deeply, it will cause you to go into a deep existential panic, and I recommend you not do it. But you're actually left with a real logistical problem of, like, how do you know random data versus not random data? It's not labeled for you, right? Like, how do you know that? And so much of the brunt of our talk at the end here is going to dig into exactly that. So this is also anecdotally how this whole thing sort of got started. We, pentesters, obviously, at Bishop Fox, and we do a number of what we call product security reviews, where if it's like, usually we say, like, if it drops when you break it, then it's a product security review. And so we, on one occasion, we're on an engagement testing, an IoT device. It was like a security device, right? And it involved a lot of cryptography and a lot of random number generation as part of the cryptography. And so on a lark, we just sort of asked, like, hey, like, what are you guys doing for random number generation? Like, on the device, they answered, oh, our SOC has a hardware random number generator embedded on it, so we just sort of used that. And so then on a lark, I just asked, like, hey, could you just, like, give me, like, a gigabyte file, like, just like a big dump of numbers from the RNG, just so I can look at it, right? Just to, like, see how good or bad it is.  And immediately, it just failed all the statistical randomness tasks we threw at it, which was concerning. And then you just look at the file, you kind of scroll through it, and just large swads of it are zero. This was very concerning. And to the point, I was thinking, like, surely this must be, like, some crazy buggy code or, like, what is going on here? And that sort of leads us down to basically everything that came out of this presentation. So this is where I'd like to segue into what I want to call Petro's Law. That's me. If I could have one law, one eponymous law, help me out here, Defcon. It would be this. It can always be worse. No matter how bad you think it is, it can always be worse. So you might think, wow, entire device making crypto keys of zero, how could it be worse than that? Let me show you. Uninitialized memory. So you might recall that the howRNG function uses an output variable from the call to store the output from, because it has to use the return code for the error code. Well, lots of the howRNG functions will operate in a way that they don't actually set the value to anything in case of an error. Sometimes they'll set it to zero, and that's how you get all those zeros. But not every device works that way. Sometimes if there's an error code, it just won't touch your variable. So if you do something like this where you declare but not instantiate a variable as your random number, it'll just be something in RAM. And then you send it to the howRNG function and then send that on its way out onto the internet or to a potential adversary. You're basically heart bleeding yourself through the random.  Basically like heart bleed style like on your device. And what's so bad is like it'll actually sort of kind of look like random numbers some of the time too. And this is very likely to occur in the real world where you might be doing something like a Diffie-Hellman key exchange where like the very first thing you do is generate a random number and send it to the other guy. So yeah, that's terrifying. So one of the things we want to touch on is that there was a talk, a given, let's see, was this 2019 I believe, that they looked through a large number of certificates and like millions of them. And they found that one in 172 of the certificates were vulnerable to attack. Now I'm not saying that what we're presenting today is what they found, but I'm saying that what we're talking about is what they found. Yeah, in this research in particular, they called out low entropy IoT devices as like one of the like reasons that all these certificates were failing. So one of the things that we immediately realized upon doing this research is that one of the things you should expect to see if this were the case was that lots of IoT devices out in the wild should be like generating bad certificates. And lo and behold, that's like exactly what we see. And so this might actually very well be like the cause of this prior research. So this sounds all nice and tidy, right?    Like let's actually walk through this a little bit more, right? So the very top here is some code from the MediaTek documentation from the 7697.  And it has this comment in here that says error handle. It turns out that that comment handle error is doing a lot of heavy lifting. The problem is that when you're doing some security critical piece of logic, right? And you need a random number. You really need it. You can't just continue on without it and handle the error, right? It's sort of critical to the core thing you're trying to do at that time. So you kind of just have two options. There's one which is to spin loop. This is actually recommended in some of the documentation and it's terrible. You're just going to keep on calling it using 100% CPU, spin looping, maybe forever in case of like a broken peripheral, like that's a terrible solution. Why would you ever suggest people to do that? Why would you ever expect developers to do that? The second is just to quit out and kill the process entirely. Like can you imagine if like you're in the networking and you're making TLS connections to somebody else and like the RNG function generates an error. Which happens all the time. Like if you're trying to make a 4096 bit RSA key and you call the RNG function really, really quickly, you're literally going to be incapable of creating an RSA key basically using the RNG functions this way. It'll just quit out of the entire network.  So what do you give developers? Well, if they can't handle the errors, YOLO. You just go for option number three, which is to ignore the error. And doing that has some consequences. It's not that... How do I say this? Really, it's not the fault of the user. You can't blame them here. They're stuck in a no-win situation. RNG and IoT are fundamentally broken. And now there are some right ways to do RNG. What we're going to probably be talking about for the majority of the talk from here on out is why you need a cryptographically secure pseudo-random member generator. So let's talk about the right way to RNG. As a cryptographically secure pseudo-random generator has some distinct advantages. Namely, it never blocks execution. So we don't have to worry about that here. API calls don't fail. It specifically, and this is the big thing that you should hear, it pulls entropy from multiple sources. So if you have an IoT device, it's probably got one hardware RNG. Now, the problem with seeding anything, even if it's a pseudo-random generator with only a single source, is that if you're an attacker and you control that source, that's what the end number is going to be. This is how tool-assisted speedruns work. As an attacker playing a physical Nintendo, I'm controlling the only source of entropy, which is player input, thereby guaranteeing that I always get exactly results I want and the deterministic behavior out of the pseudo-random generator. So you don't want to rely on a single source that an attacker can take advantage of. Instead, you should be pulling your hand and you would have to have full control of every single source at the same time in order to manipulate that. There's also the big advantage of cryptographic, we secure pseudo-random number generators and I'm not even going to try to get into the same, I'm going to say CSP RNGs always return cryptographic quality results. We're going to come back to that in a second. And this is how every major operating system works, Linux, Mac, Windows, BSD, Solaris, they all have some method of doing CSP RNG and IoT should as well. Anything else you want to say on that topic? Yeah, so the CSP RNG subsystem basically works like the diagram here, right? You're going to stretch out that initial entropy pool indefinitely going forward. In practice, the algorithms for doing this can actually be rather simple. The easiest way to construct one of these is just by using a hashing function. So you just take the entropy pool and you just like it. You can pick whatever hashing function that you prefer. I like md5. You md5 your entropy pool, right? And then in order to produce the next batch, you can md5 the entropy pool and also the last thing you just made. And you can keep on chaining that out indefinitely going forward until that way you can kind of keep on making bits infinitely into the future. What's nice about that is because it's a cryptographically serial hashing function, you can't go backwards. There's no way to take the bits output from it and then derive the seed or any of the other values therein because you also need to know the entropy pool itself. You can switch around those seeds and like I was saying, that's basically how this works everywhere else just not in the world of IoT. So yeah, basically we're going to go into why you really do need a CSP RNG subsystem and know you really can't just make do without it. So nobody codes from scratch, right? If you have a library that does what you want, you're going to use that library and in IoT there's lots of reference and example code for how to call hardware random number generators, even libraries to do it for you. But sometimes the reference code has vulnerabilities that propagate. There are some really unfortunate examples of this. You want to actually talk about these ones? You know this the best. So yeah, the way that some libraries work such as Contiki NG specifically when using the Nordic NRF 840 SOC as well as the link at 7697 Arduino code when you call it for a hardware you call the how function, you might think like I'm going to be getting a number from the hardware RNG, right? That's not actually how it works. All it does is take the hardware RNG, generate a seed and then use libc random there forward. So you're actually not even using the hardware RNG really, you're just using libc random. That is as broken as it sounds and for that we actually have a demo video. Let's see if the audio works on this, there's an audio on this, so if there's anybody that needs to turn that on in the audio section then they should be good. So for this demo we built an IoT security camera device.  So the only thing keeping an attacker from being able to view your photos is that each file is named to this long random file name here, chosen by the camera. Now before you go through. It's not playing. No, the video is not changing. Oh yeah, there it is. So for this demo we built an IoT security camera device.  So the only thing keeping an attacker from being able to view your photos is that each file is named to this long random file name here chosen by the camera. Now before you go thinking that this is unrealistically vulnerable, this is how Discord works and lots of other applications like it. Anytime you take a photo and send it to a friend over Discord, it's publicly accessible. The long random file name is the only thing that's keeping people from seeing your photos. Our device, however, is built using Contiki NG, a popular IoT operating system. When you call the operating system to get a random number it will use the hardware RNG on board, but only to seed the insecure libc random function. So we don't know what the seed is, but we don't have to because we can derive it. So suppose one day you take a photo with your camera and post it on social media. Well what a cool camera you bought. How fun. But what you didn't know is that the attacker can use this file name to derive what the original seed was that generated it. That's because that's how the libc random function works. Our attacker here uses untwister to find the seed. Once they have they can use that seed to determine every past and future value from the RNG. So our attacker can just plug in some of those numbers back into the camera website and view every photo that the camera has ever taken even if they've never been shared before. Cool. Alt tab. Excellent. We're back. That was demo. Cool. So anytime we talk about random number generation and security, the conversation goes to exploitability. How exploitable is this really? Sometimes they turn out to be cryptographic vulnerabilities where crypto is often considered broken if you're able to break it even just a little bit. If you're able to derive RNG, like 128 bit ASP and 127 bits, that's considered broken in cryptographic terms and sometimes RNG vulnerabilities are the same way. Turns out this is very real and very exploitable but just not in a very canned way. So the very often what you're going to find here is that it's going to depend on the particulars of the individual device that you're looking at. Again, like I was saying earlier the IoT devices tend to be really heterogenous and very different. There's a single piece of logic that all IoT devices like use. Not all of them talk over HTTP. Not all of them even use TCPIP. So you're likely to need to look into the individual business logic of the individual device that you're looking at. So it's very unlikely to be like you're doing it wrong .py that kind of like you can just point at devices and like exploit. So keep that in mind. One exception to this, one perhaps notable exception to this is actually asymmetric cryptography. So one of the things that will cause these RNG health functions to fail is calling them very quickly over and over again. And if you're trying to make a 248 or 4096 bit like RSA key, you're going to need a lot of bits and you're going to have to call the RNG function in a loop over and over again and it's a very reliable way of causing the RNG functions to fail. And so if you were a pen tester and you're looking at like trying to use this in your upcoming IoT engagement, I would definitely keep an eye at any asymmetric cryptography that your IoT device is trying to do. Especially if you're able to look at the code, then see if it's checking the error codes coming out of the health function and if not, then there's a very good chance that virtually all of the asymmetric keys that it's creating have just big chunks of zeros in them. There's also a separate talk at this DEF CON called the mechanics of compromising low entropy RSA keys. We did not plan this, that's just kind of a thing that happened. So there's also, I haven't seen that talk yet, there might be some interesting new exploits coming out of that that might be practical for pen testers to use on like actually using some RSA keys that have a bunch of zeros in them. So if you are a pen tester again and you have like an IoT engagement coming up, whether you have source code or not, is going to dictate a lot of like how you're going to try to like attack this going forward. Try to tax the RNG as much as possible. So if there's like a remote request that you can perform to the device that you know will cause it to use the RNG, basically spam those requests and see if they start coming back zero. And that would be like a really good way of trying to see that. Or if you know that there's going to be a lot of photography then keep an eye out for that. Okay. So he had his law. Mine is somewhat different. Alan's law is going to be never write your own RNG code. It's as dangerous as writing your own crypto code. Everyone knows now, or everyone really ought to know by now, that you don't write your own crypto code. It's very difficult to do right. Well, this is at the same level. It's no different than crypto code because there are so many ways to do it wrong. We already talked about the user is already in a bad situation as it is. But even if you manage to do all of the other things right, there are still some interesting gotchas. Sometimes the hardware itself has things you would never expect to encounter. One of the examples we ran into was the LPC 54628. There is a warning on page 1106 that says if you want actually random numbers you have to ask for a random number then ask for 32 more random numbers but throw those away then ask for another random number then throw another 32 away and repeat until you have the length key you want. If you saw this code written down you might think it was a mistake unless someone heavily commented why they were doing this. You would read this code and think what you have to do in order to talk to this particular hardware RNG correctly. And this isn't the only bad example. Even if you are you this is pivoting the talk a little bit but I'm going to get passionate about this one even if you do everything right you read that 1,000 page manual you find the weirdness you're still not out of the weeds you can call the hardware RNG spin loop do everything right you get random numbers that are not random so this is an example of taking a large multi-gig file of randomness and then doing a statistical analysis of how often each byte happens to appear. So 0 through 255 which bytes are happening or which values are happening more often in this byte and what we find on the media text 7697 is this very interesting sawtooth pattern what you would expect to see is that it happens with an even distribution they're all happening at the same frequency and that's not what's happening at all and if you zoom in a little bit you can actually see that the sawtooth pattern of unequal frequency repeats which is even more mysterious and we would really like to understand what exactly was happening there I'm going to let you take this one So you think surely it can't get worse so basically what a lot of this involved was we had to write code for all of these digital dev boards, these socks bought up like every internet of things dev board that we could get, write code for them that would generate random numbers and store them out to a big file, like a couple of gigs worth of files and then analyze them offline. One that's actually really hard to do logistically that very often dev boards are just not set up to do this and had to have some janky setups of uploading it to a web server that we were connecting to and we had to get the I.O. out of the thing it was Wi-Fi and sometimes you can output it over a serial line sometimes they can only read and not write over serial, it was a complete mess so that's kind of the analysis that you're looking at here is us finally like okay no more shenanigans, no more like weird API gotchas no more like silly randomness stuff that can happen through the APIs like what is the actual randomness like coming out of these things is it good is it bad? So this is obviously not great obviously repeating patterns and cryptography don't go along very well together. This is from the Nordic NRF 52-840 it has a curious repeating 12-bit pattern it has three zeros, the 12 bits that repeat every exactly in hex 50 bytes except sometimes it jumps it'll go like 80 or so and so weird that we thought for sure it must have been a mistake in our instrumentation for the longest time and no matter what we did it wouldn't go away so we think it's not that for a few reasons, one of which being that we looked at the code for a really long time and had no idea how there could possibly be a 12-bit repeating pattern if it was one bit you think like maybe it's like a null terminated string you're copying it around stupidly I'm completely capable of writing that code but 12 bits like how does that so that was very concerning it also sort of says something as well that like even if it turns out that so also we're going to be releasing all I think it's up now, releasing all of our code on our github so it's on github.com slash bishop fox there's a repo called you're doing it wrong that has all the source code to the devices that we did and also some of our results things like that so it's entirely possible that maybe this was the result of some buggy code so you have to consider what it means that you know two computer security experts spent months analyzing rngs and spent months trying to find bugs in the rng code that they wrote and still failed at it like what does that say about the state of IOT security having to do with rngs going forward that's very true for the STM32 in this case I took an L432 KC try to say that fast tried to get good random numbers out of this and I failed and I tried again and I got some help and I failed and we tried again and tried blocking in a different way and we still failed the initial numbers we got out of this looked terrible because we weren't actually doing the blocking correctly once we got all of the blocking fully worked out we were spin looping every time we needed to wait for the air condition to clear only getting random numbers from the device and we still had a problem it was mostly clean but then when we ran it through dieharder which is a suite of statistical analysis tests it still managed to fail a particular minimum distance test with a high degree of certainty I'm going to let him try to explain this part of it yeah so there's a well known tool called dieharder excellent name that is a statistical randomness test that's used outside of security people use this for other things like the number of tests one of which being the RGB minimum distance test RGB is just the initials of the dude who made it so the minimum distance test is basically you take a bunch of numbers and you interpret them as integers in n dimensional space and then calculate using a very simple algorithm what the smallest distance between any of the numbers is so really what this is doing is trying to find repeated values or nearly repeated values so you can imagine a world where like you have a generator where some value depends on some other value that's earlier on in the sequence right and maybe it's not always exactly the same maybe it's like it jumps a little bit or something like that those kinds of correlations between data can be really hard to pick out and there's lots of ways of trying to find them this isn't the only way obviously but the way that this works is that like if there's any repeats there's going to be two points in that space that are going to be close to each other and so when you do the subtraction they turn out to be zero or near zero so you can run this test and it'll kind of tell you whether things fall into expected parameters or not so this one was curious only because we never quite figured out exactly what was going wrong with the STM32 like exactly what that repeated value was or like how concerned we should be exactly about it but the whole point of hopefully this presentation is that you should need to care about this sort of thing like the output of the hardware random number generator raw like this is a thing that you should be using as an input to a cryptographically secure pseudo random number generator and not like on its own so we have some conclusions for you and some of them are going to be bigger than any one person because this affects the entire IOT industry it's not a single vendor it's not a single device it's not a single library it's not that the only viable solution is a CSP RNG subsystem that is implemented in software so this can't be fixed by changing documentation and blaming users I'm going to steal a line from him the good news is that this can be fixed through software the bad news is it has to be fixed through software and we're talking about IOT devices here RNG, I said this earlier RNG code should be considered dangerous to write on your own it's just like crypto code you really need to have a viable subsystem that you can call that's already been vetted and last but not least if we haven't shown you this at least enough examples of it there are more out there we didn't include here that you can find in an article we wrote we have more examples of bad stuff but you really don't know how strong or weak your hardware is actually going to be it's really hard to tell what can you actually do about this it depends on where you are first is device owners this is going to have to be a software fix IOT devices not known for their ability to be patched out in the wild but at least it's possible for some devices that have the ability to do over-the-air updates some of the ones that are particularly sensitive to a random number like if it's a security device then hopefully this will be a thing for IOT developers there's this emerging space of IOT operating systems things like Riot OS there's Kintiki NG FreeRT OS we don't have a particular one that we would recommend over the other but the CSPRNG subsystem there since the individual device SDKs are you're unlikely to see the sort of feature here those are mostly about hardware enablement and not so much about trying to implement features that make like a full device so if you're developing a whole new device from scratch I'd really recommend using one of those one because they're actually pretty handy and there's other benefits to them aside from security but that's certainly where the security is there's servers and laptops and desktops because there's an operating system you can just ask Linux like hey please give me a random number just ask devu random for a random number and it will just give it to you and it's perfect but this is not true in the IOT world and it really needs one so IOT device manufacturers and those OS developers you really do need to implement the CSPRNG subsystem like there's not a way around it you can't just change the documentation or try to change the documentation to get people to check the error codes none of that stuff is sufficient it can't just change by blaming users disallowing users certainly deprecating the HAL functions for the RNGs would be a nice way and perhaps even just disabling users from being able to use them entirely would be a good way to go forward and for pentesters this is likely to be a perennial finding for years to come if you have an IOT engagement coming up then keep an eye out for this especially if you have access to source code then trying to find it could be pretty simple and then building out an exploit is going to be highly contextual dependent but entirely possible and that's about it thanks a lot I'm Dan Petro I'm Alan Cecil and I think we're going to do Q&A how much time do we have? if somebody wants to shout we can repeat a question there I see a hand do your very best to shout I didn't hear much of that I'm afraid perhaps you could come a little bit closer yeah so I guess the question is like maybe how hard is it to implement your own CSPRNG subsystem maybe a decent way to do that there's a couple ways you can do that by like the hashing function RNG like code that you can do those are if you're in a real pinch then those are probably better than just like making the call to the howl function yourself there's definitely a lot that can go wrong in the development of a CSPRNG subsystem that's non-trivial and in the abstract I'd love to say to not try to make that yourself but it's not the end of the world it's certainly a lot better than the alternative of just like yoloing the howl function yes there's a lot of implementations of that that are like simpler than others and the simpler you can get probably the better as long as you're sure that it's actually a CSPRNG sub-RNG and not just like linear congruential generator or something like that yeah and again if you can source from multiple entropy pools I don't really care what you use just try to find more than one entropy source so an attacker would have to control all the variables at once time is not a variable that you should use cool do we have yes right in front so there is a FIPS standards like FIPS 140-2 or something like that I want to say around a specifically random number generation and the question is like did we notice any difference between like ones that were like FIPS certified and ones that aren't I think the short answer is no that like the certification so the best that we found was specifically around ST micro electronics had like a proof of randomness they had their own code you could look at so huge shouts to ST for that but otherwise I don't think we really notice the difference between like FIPS certified devices and otherwise yeah and one thing I should note is we were trying to replicate STM32s or the results that ST micro released we were trying to validate and replicate those results and we couldn't do it so it might be that it's from a vendor might work differently than the next one same exact model family just a different device and somehow is less random we don't know so I think that we're going to be doing is there a room for Q&A that we could follow the mob of people that is invariably going to be following us to a Q&A session afterwards we'll keep answering questions for a while but otherwise yeah thanks a lot thank you very much


Everybody felt emotional because he gave so much of time and he talked about our Gurus and then he wore a turban. So that was very very emotional for the Sikh community I think.


From our studios, in the heart of Silicon Valley, Palo Alto, California, this is a CUBE Conversation. Hello, and welcome to the CUBE studios here in Palo Alto, California. We have a special around the CUBE segment, Unpacking AI. This is a Get Smart series. We have three great guests, Roger Sheth, VP of AI and Product Management at Google. He has all the AI development for Google Cloud. Dr. Kate Darling, research specialist at MIT Media Lab and Professor Barry O'Sullivan, Director, SFI Center for Training AI, University of College, Cork and Ireland. Thanks for coming on, everyone. Let's get right to it. Ethics in AI, as AI becomes mainstream, moves out of the labs and computer science world to mainstream impact. The conversations are about ethics. And this is a huge conversation, but first thing people want to know is what is AI? What is the definition of AI? How should people look at AI? What is the definition? Let's start there. Roger. So I think the way I would define AI is any way that you can make a computer intelligent to be able to do tasks that typically people use to do. And what's interesting is that AI is something, of course, that's been around for a very long time in many different forms. Everything's from expert systems in the 90s all the way through to neural networks now. And things like machine learning, for example, people often get confused between AI and machine learning. I would think of it almost the way you would think of physics and calculus. Machine learning is the current best way to use AI in the industry. Kate, your definition of AI, do you have one? Well, I find it interesting that there's no really good universal definition. And also I would agree with Rajan that right now we're using kind of a narrow definition when we talk about AI, but the proper definition is probably much more broad than that. Probably something like a computer system that can make decisions independent of human input. Professor Barry, your take on the definition of AI, is there one? What's a good definition? Well, yeah, so I think AI has been around for 70 years and we still haven't agreed the definition for it, as Kate said. And I think that's one of the very interesting things. I suppose it's really about making machines act and behave rationally in the world. Ideally autonomously, so without human intervention. But I suppose these days AI is really focused on achieving human level performance in very narrowly defined tasks. So game playing, recommender systems, planning. So we do those in isolation. We don't tend to put them together to create this sort of fabled artificial general intelligence. I think that's something that we don't tend to focus on at all actually, Okay, so the question is, is that AI is kind of elusive, it's changing, it's evolving. It's been around for a while, as you guys pointed out. But now that's on everyone's mind. We see the news every day from Facebook, being a technology platform with billions of people. AI was supposed to solve the problem there. We see new workloads being developed with cloud computing where AI is a critical software component of all this. But that's a geeky world. But the real world, there's an ethical conversation, is not a lot of computer scientists have taken ethics classes. So who decides what's ethical with AI? Professor Barry, let's start with you. What are we going to start with ethics? Yeah, sure. So one of the things I do is I'm the vice chair of the European Commission's high level expert group on artificial intelligence. And this year we published the ethics guidelines which were the AI in Europe, which is all about, you know, setting an ethical standard for what AI is. You're right, you know, computer scientists don't take ethical standards. I suppose what we are faced with here is a technology that's so pervasive in our lives that we really do need to think carefully about the impact of that technology on human agency and human wellbeing, on societal wellbeing. So I think it's right and proper that we're talking about ethics at this moment in time. But of course we do need to realize that ethics is not a panacea, right? So it's certainly something we need to talk about, but it's not going to rid us of all of the detrimental applications or usages of AI that we might see today. Kate, your take on ethics. Start all over, throw out everything, build on it. What do we do? Well, what we do is we get more interdisciplinary, right? I mean, because you asked who decides until now it has been the people building the technology who have had to make some calls on ethics. And it's not necessarily the way of thinking that they are trained in. And so it makes a lot of sense to have projects like the one that Barry is involved in where you bring together people from different areas. I think we lost Kate there. Roger, why don't you jump in, talk about it. You decide issues of responsibility for harm. We have to look at algorithmic bias. We have to look at supplementing versus replacing human labor. We have to look at privacy and data security. We have to look at the things that I'm interested in, like the ways that people anthropomorphize the technology and use it in a way that's perhaps different than intended. So depending on what issue we're looking at, we need to draw from a variety of disciplines. And fortunately, we're seeing more support for this within companies and within universities as well. Roger, your take on this. So I think one thing that's interesting is to step back and understand why this moment is so compelling and why it's so important for us to understand this right now. And the reason for that is that this is the moment where AI is starting to have an impact on the everyday person. Any time I present, I put up a slide of the Mosaic browser for 24. And my point is that that's where AI is today. It's at the very beginning stages of how we can impact people even though it's been around 70 years. And what's interesting about ethics is we have an opportunity to do that right from the beginning right now. I think that there's a lot that you can bring in from the way that we think about ethics overall. And so for example, in our company, you all hear me? Yep. Okay, we've hired an ethicist within our company from a university to actually bring in the general principles of ethics and bring that into AI. But I also do think that things are different. So for example, bias is an ethical problem. However, bias can be encoded and actually given more legitimacy when it can be encoded in algorithms. So those are things that we really need to watch out for where I don't think it is a little bit different than a little bit more interesting. This is a great point. Okay, go ahead. What's your point of, yeah, just one interesting thing bear in mind that I think Kate said this and I just want to echo it is that AI is becoming extremely multidisciplinary. And I think it's no longer a technical issue. Obviously there are massive technical challenges but it's now become as much an opportunity for people in the social sciences, the humanities, ethics people, legal people, I think need to understand AI. And in fact, I gave a talk recently at a legal symposium and the idea of this sort of parallel track of people who have legal expertise and AI expertise. I think that's a really fantastic opportunity that we need to bear in mind. So unfortunately, us nerds, we don't own AI anymore. It's something we need to interact with real world on a significant basis. You know, I want to ask a question because the algorithms, everyone talks about the algorithms and the bias and all that stuff. It's totally relevant. Great points on interdisciplinary. But there's a human component here. As AI starts to infiltrate the culture and hit everyday life, the reaction to AI sometimes can be, whoa, my job's going to get automated away. So I got to ask you guys, as we deal with AI and is that a reflection on how we deal with it to our own humanity? Because how we deal with AI from an ethics standpoint, ultimately is a reflection on our own humanity. Your thoughts on this. Roger, we'll start. I mean, it is. Oh, sorry. Roger? So I think it is. And I think that there are three big issues that I see that I think are reflective of ethics in general, but then also really are particular to AI. So there's bias. And bias is an overall ethical issue that I think is particular here. There's what you mentioned, the future of work. What does the workforce look like 10 years from now? And that changes quite a bit over time. And if you look at the workforce now versus 30 years ago, it's quite a bit different. And AI will change that radically over the next 10 years. The other thing is what is good use of AI and what's bad use of AI? And I think one thing we've discovered is that there's probably 10% of things that are clearly bad and 10% of things that are clearly good and 80% of things that are in that gray area in between where it's up to kind of your personal view. And that's the really, really tough part about all of this. Kate, you have been a way in. Well, I think that I'm actually going to push back a little because not on Rajan, but on the question because I think that one of the fallacies that we are constantly engaging in is we are comparing artificial intelligence to human intelligence and robots to people. And we're failing to acknowledge sufficiently that AI has a very different skill set than a person. So I think it makes more sense to look at different analogies, for example, how have we used and integrated animals in the past to help us with work? And that lets us see that the answer to questions like, will AI disrupt labor market? Will it change infrastructures and efficiencies? The answer is yes, but will it be a one-to-one replacement of people? No, that said, I do think that AI is a really interesting mirror that we're holding up for ourselves to answer certain questions like, what is our definition of fairness, for example? We want algorithms to be fair. We want to program ethics into machines. And what it's really showing us is that we don't have good definitions of what these things are, even though we thought we did. All right, Professor Barry, your thoughts. Yeah, I think many points one could make here. I suppose the first thing is that we should be seeing AI not as a replacement technology, but as an assistive technology.  I think absolutely, the labor force will be transformed in the future, but there isn't going to be massive job loss. The technology has always changed how we work and play and interact with each other. Look at the smartphone. The smartphone is 12 years old. We never imagined in 2007 that our world would be the way it is today. So technology transforms very subtly over long periods of time, and that's how it should be. I think we shouldn't fear AI. I think the thing we should fear most, in fact, is not artificial intelligence, but it's actual stupidity. So I think we need to... So I would encourage people not to think... It's very easy to talk negatively and think negatively about AI because it is such an impactful and promising technology, but I think we need to keep it real a little bit, right? So there's a lot of hype around AI that we need to sort of see through and understand what's real and what's not, and that's really some of the challenges we have to face. And I suppose one of the big challenges we have is how do we educate the ordinary person on the street, understand what AI is, what is capable of, when it can be trusted and when it cannot be trusted, and ethics gets us some of the way there, but it does not get us all of the way there. We need good old-fashioned, good engineering to make people trust in the system. As a great point, ethics is kind of a reflection on that mirror, I love that. Kate, I want to get to that animal comment about domesticating technology, but I want to stay in this culture question for a minute. If you look at some of the major tech companies, like Microsoft and others, the employees are revolting around their use of AI in certain use cases. It's a knee-jerk reaction around, oh my God, we're using AI, we're harming the world. So we live in a culture now where it's becoming more mission-driven, there's a cultural impact, and to your point about not fearing AI, are people having a certain knee-jerk reaction to AI because you're seeing cultures inside tech companies and society taking an opinion on AI? Oh my God, it's definitely bad, our company's doing it, we should not service those contracts or maybe I shouldn't buy that product because it's listening to me. So there's a general fear. Does this impact the ethical conversation? How do you guys see this? Because this is an interplay that we see that's a personal, it's a human reaction. Yeah, so if I may start off, I suppose absolutely there are, the ethics debate is a critical one and people are certainly fearful. There is this sort of polarization in debate about good AI and bad AI, but AI is good technology. It's one of these dual use technologies. It can be applied in bad situation in ways that we would prefer it wasn't and it can also, it's a force for tremendous good. So we need to think about the regulation of AI, so what we wanted to do from a legal point of view, who is responsible, where does liability lie? We also think about what our ethical framework is and of course there is no international agreement on what is, there is no universal code of ethics. So this is something that's very, very heavily contextualized, but I think we certainly, I think we generally agree that we want to promote human well-being, we want to have a prosperous society, we want to protect the well-being of society. We don't want technology to impact the society in any negative way. It's actually very funny, if you look back about 25, 30 years ago, there was a technology where people were concerned that privacy was going to be a thing of the past, that computer systems were going to be tremendously biased because data was going to be incomplete and not representative and there was a huge concern that good old fashioned databases were going to be the technology that would destroy the fabric of society. That didn't happen. And I don't think we're going to have AI do that either. Kate? Yeah, we've seen a lot of technology panic that may or may not be warranted in the past. I think that AI and robotics suffers from a, the civic problem that people are influenced by science fiction and prop culture when they're thinking about the technology and I feel like that can cause people to be worried about some things that maybe perhaps aren't the thing we should be worrying about currently, like robots and jobs or artificial superintelligence taking over and killing us all aren't maybe the main concerns we should have right now, but algorithmic bias, for example, is a real thing, right? We see systems using data sets that disadvantage women or people of color and yet the use of AI is seen as neutral, even though it's entrenching existing biases or privacy and data security, right? You have technologies that are collecting massive amounts of data because the way that machine learning works is you use lots of data. And so there's a lot of incentive to collect data. As a consumer, there's not a lot of incentive for me to want to curb that because I want the device to listen to me and to be able to perform better. And so the question is who is thinking about consumer protection in this space all of the incentives that we're collecting and using as much data as possible. And so I do think there is a certain amount of concern that is warranted and where there are problems. Like I endorse people revolting, right? But I do think that we are sometimes a little bit skewed in our, you know, understanding where we currently are at with the technology with the actual problems are right now. Rajan, I want to give thoughts on this is education is key. As you guys are talking about this and things to pay attention to, how do you educate people about the, how to shape AI for good at the same time, calm the fears of people at the same time from revolting around misinformation or bad data around what could be? Well, I think that the key thing here is to organize kind of how you evaluate this. And back to one thing I was saying a little bit earlier, it's very tough to judge kind of what is good and what is bad. It's really up to the personal perception. But then the more you that you organize how to evaluate this and then figure out ways to govern this, the easier it gets to evaluate what's in or out. So one thing that we did was that we created a set of AI principles and we kind of codified what we think AI should do. And then we codified areas that we would not go into as a company. The thing is it's very high level. It's kind of like the constitution. And when you have something like the constitution, you have to get down to actual laws of what you would do. It's very hard to bucket and say, these are good use cases, these are bad use cases. But what we now have is a process around how do we actually take things that are coming in and figure out how do we evaluate them? A last thing that I'll mention is that I think it's very important to have many, many different viewpoints on it. Have viewpoints of people that are taking it from a business perspective, have people that are taking it from kind of a research and an ethics perspective and all evaluate that together. And that's really kind of what we've tried to create to be able to evaluate things as they can. Well, I love that constitution angle. We'll have that as our last final question in a minute but do we do a reset or not? But I want to get to the point that Kate mentioned. Kate, you're doing research around robotics. And I think robotics is a, you're seeing robotics surge in popularity from high schools, have varsity teams now. You're seeing robotics with software advances and technology advances really become kind of a playful illustration of computer technology and software where AI is playing a role and you're doing a lot of work there. But as intelligence comes in to say robotics or software or AI, there's a human reaction to all this. So there's a psychology interaction to either AI and robotics. Can you guys share your thoughts on the humanization interaction between technology? As people stare at their phones today there could be relationships in the future. And I think robotics might be a signal. You mentioned domesticating animals as an example back in the early days of when we were, as a society, that happened. Now we all have pets. Are we going to have robots as pets? We're going to have AI pets. Is this kind of the human relationship? Okay, go ahead. Share your thoughts. So okay, the thing that I love about robots and in some applications to AI as well as that people will treat these technologies like they're alive even though they know that they're just machine. And part of that is again the influence of scientific culture that kind of primes us to do this. Part of it is the novelty of the technology moving into shared spaces. But then there's this actual biological element to it where we have this inherent tendency to anthropomorphize, project human-like traits, behaviors, qualities onto non-humans. And robots lend themselves really well to that because our brains are constantly scanning our environments and trying to separate things into objects and agents. And robots move like agents. We are evolutionarily hard-bired to project in tents onto the autonomous movement in our physical space. And this is why I love robots in particular as an AI use case because people end up treating robots totally differently. Like people will name their Roomba vacuum cleaner and feel bad for it when it gets stuck. Which they would never do with their normal vacuum cleaner, right? So this anthropomorphization, I think, makes a huge difference when you're trying to integrate the technology because it can have negative effects. It can lead to inefficiencies or even dangerous situations. For example, if you're using robots in the military as tools and they're treating them like pets instead of devices. But then there are also some really fantastic use cases in health and education that rely specifically on this socialization of the robot. You can use a robot as a replacement for animal therapy where you can't use real animals. We're seeing great results in therapy with autistic children, engaging them in ways that we haven't seen previously. So there are a lot of really cool ways that we can make this work for us as well. Barry, your thoughts, have you ever thought that we'd be adopting AI as pets someday? Oh, yeah, absolutely. Like Kate, I'm very excited about all of this too. And I think there's a few, I agree with everything Kate has said. Of course, coming back to the mark you made at the beginning about people putting their faces in their smartphones all the time. We can crowdsource our sense of dignity and that we can't have social media as the currency for how we value our lives or how we compare ourselves with others. So we do have to be careful here. Like one of the really nice things about, one of the really nice examples of an AI system that was given some significant personality was quite recently, Thomas Santom and others at Carnegie Mellon had this, produced this libratus poker playing bot. And this AI system was playing against these top-class Texas Holden players and all of these Texas Holden players were attributing a level of cunning and sophistication and mischief on this AI system that clearly it didn't have because it was essentially trying to just behave rationally. But we do like to project human characteristics onto AI systems. And I think what would be very, very nice and something we need to be very, very careful of is that when AI systems are around us and particularly robots, we do need to treat them with respect. And what I mean is, we do make sure that we treat those things that are serving society in as positive and nice a way as possible. I do judge people on how they interact with the least advantaged people in society. And by golly, I will judge you on how you interact with a robot. Roger. We've actually done some research on that where we've shown that if you're low empathy, you're more willing to hit a robot, especially if it has a name. I love all my equipment here. Oh, yeah. I've got to tell you, it's all beautiful. Roger, I mean, computer science and now AI is having this kind of humanization impact. This is an interesting shift. I mean, this is not what we study computer science. We were writing code. We were going to automate things. Now there's notions of math and not just math, cognition, human relations. What are your thoughts on this? Yeah. You know, what's interesting is that I think ultimately it boils down to the user experience. And I think there is this part of this which is around humanization, but then ultimately it boils down to what are you trying to do and how well are you doing it with this technology? And I think kind of, I think that example around the Roomba is very interesting where I think people kind of feel like this is more kind of a, almost like a person. But also I think we should focus as well on what the technology is doing and what the practice have. My best example of this is Google Photos. And so my whole family uses Google Photos and they don't know that underlying into some of the most powerful AI in the world, all they know is that they can find pictures of our kids and their grandkids on the beach anytime that they want. And so ultimately I think it boils down what is the AI doing for the people and how is it? Yeah, expectations become the new user experience. I love that. Okay guys, final question. Now it's a humanization. We talked about the robotics, but also the ethics here. Ethics reminds me of the security debate and security in the old days. Do you increase the security or you throw it all away and start over? So with this idea of how do you figure out ethics in today's modern society with it being a mirror? Do we throw it all away and do a do over? Can we recast this? Can we start over? Do we augment? What's the approach that you guys see that we might need to go through right now to really not hold back AI, but let it continue to grow, accelerate, educate people, bring value and user experience to the table. What is the path? We'll start with Barry and Kate and then Ron. I guess I can kick off. I think ethics gets us some of the way there, right? So obviously we need to have a set of principles that we sort of sign up to and we agree upon. And there are literally hundreds of documents on AI ethics. There are, I think in Europe, for example, there are 128 different documents around AI ethics, I mean policy documents. But we have to think about how are we going to actually make this happen in the real world? And I think if you take the aviation industry, we trust in airplanes because we understand that they're built to the highest standards, that they're tested rigorously, and that the organizations that are building these things are held account when things go wrong. And I think we need to do something similar in AI. We need good, strong engineering. And ethics is fantastic. And like I'm a strong believer in ethical codes, well, we do need to make it practical. And we do need to figure out a way of having the public trust in AI systems that comes back to education. So I think we need a general public and indeed ourselves to be a little more cynical and questioning when we hear stories in the media about AI, because a lot of it is hyped. And that's because researchers want to describe their research in an exciting way, but also newspaper people and media people want to have a sticky subject. But I think we do need to have a society that can look at these technologies and really critique them and understand what's been said. And I think a healthy dose of cynicism is not going to do us any harm. So modernization, do you change the ethical definition? Kate, what's your thoughts on all this? Well, I love that Barry brought up the aviation industry because I think that right now we're kind of an industry in its infancy, but if we look at how other industries have evolved to deal with some thorny ethical issues, like for example, medicine. Medicine had to develop a whole code of ethics and develop a bunch of standards. If you look at aviation or other transportation industries, they've had to deal with a lot of things like public perception of what the technology can and can't do. And so you look at the growing pains that those industries have gone through and then you add in some modern insight about interdisciplinary, about diversity and tech development generally, getting people together who have different experiences, different life experiences when you're developing the technology. And I think we don't have to rebuild the wheel here. Yeah. Rajan, your thoughts on the path forward, throw it all away, rebuild. What do we do? Yeah, so I think this is a really interesting one because of all the technologies I've worked in within my career, everything from the internet to mobile to virtualization, this is probably the most powerful potential for human good out there. And AI is, you know, the potential of what it can do is greater than almost anything else that's out there. However, I do think that people's perception of what it's going to do is a little bit skewed. So when people think of AI, they think of self-driving cars and robots and things like that. And that's not the reality what AI is today. And so I think two things are important. One is to actually look at the reality of what AI is doing today and where it impacts people's lives. Like, how does it personalize customer interactions? How does it make things more efficient? How do we spot things that we never were able to spot before and start there and then apply the ethics that we've already known for years and years and years, but adapt it to a way that actually makes sense for this technology. Okay, looks like that's it for the around the cube. Looks like Talia, looks like Professor Barry, 11, third place, Kate, second place with 13, Rajan with 16 points, you're the winner. So you get the last word on the segment here. Share your final thoughts on this panel. Well, I think it's really important that we're having this conversation right now. I think about back to 1994 when the internet first started, people did not have that conversation nearly as much at that point and the internet has done some amazing things and there have been some bad side effects. I think with this, if we have this conversation now, we have this opportunity to shape this technology in a very, very positive way as we go forward. Thank you so much. And thanks to everyone for taking the time to come in. All the way from Cork, Ireland, Professor Barry O'Sullivan, Dr. Kate Darling doing some amazing research at MIT on robotics and human psychology and a new book coming out. Kate, thanks for coming on and Rajan, thanks for winning and sharing your thoughts. Thanks everyone for coming. This is Around the Cube here unpacking AI segment around ethics and human interaction and societal impact. I'm John Furrier with theCUBE. Thanks for watching.