So I wanted to thank everyone for coming out to CF Summit 2018 here in Boston. I hope you've all had an excellent time so far, a lot of good talks, a lot of good sessions and hands-on labs and whatnot. Now I was told when my talk got accepted that I had to show this slide because the Boston City Fire Department Fire Code mandates it of all speakers at all venues within the city limits.  Please note the exit, the location of the surrounding emergency exits and locate the nearest lit exit sign to you. In the event of a fire alarm or other emergency, please calmly exit to the public concourse area, not the private concourse behind the VPN. Emergency exit stairwells leading to the outside of this facility are located along the public concourse for your safety in an emergency. Please follow the directions of the public safety staff. And fire slide. This is my talk, The Vault of Secrets. We're going to talk today about secure credentials handling and how we relate with security credentials, sensitive materials, keys and whatnot in CF applications on top of everyone's favorite paths. So who am I? Why am I up here? Who am I talking and why are you listening to me? My name is James Hunt. I'm on the tweeters at I am James Hunt. I am the author of Safe, which is an alternate vault CLI that attempts to bring a little bit of sanity and ease of use to what is otherwise an excellent software product. I also co-authored Spruce. I wrote The Vault Broker in the Call of Foundry community. I maintain the vault and Safe Bosch releases along with a rag tag group of open source hackers on the public internet. Vis-a-vis Mr. Sarabian and Mr. Daly over there. Go Safe. Secrets are everywhere. We cannot escape secrets in the modern world. We have SSH keys to deal with. We have API keys for all the services we integrate with. We have passwords for accounts. We have RSA keys, client secrets, auth tokens, signing keys, X509 CAs inserts. Where do we store these? We've got some options. We can hardcode them into the applications that we push. Whenever we need a secret, we just pop open the code, import DB, set up a const, use the const, commit, push to GitHub. Everything's good. Long as you trust GitHub. As long as your repo is private, and Dr. Nick doesn't make it public accidentally. We can put them in environment variables. We have the ability in Cloud Foundry to just set n and say this is my database secret. This is my username for my admin account. And then you're only one CFN of a way from your creds being leaked. I do believe this is a problem that the CF Core team is working on. I'm very excited to see if they're going to integrate CredHub into the actual storage of secrets and make sure that spaces and orgs are enforced from a visibility and a maintenance standpoint. But until then, we have that problem. We have a third option. We can put them in the database. All of your apps have some persistent storage usually. Whether that's a MySQL database, a Postgres database, Microsoft SQL, Redis, you've got some persistent storage that your application is hooked up to that you can pull secrets out of. So we make a secrets table. No problem. Code no longer has secrets. Environment variables no longer have secrets outside of the service. All the secrets are safely stored in the database. No one can see them except your DBA. DBA can see your secrets. As long as you trust your DBA, you're fine. You trust the security of the database host. And I guess if no one steals the disks and mounts an offline attack against them, that's not going to work. We can encrypt them in the database. RSA, AES, some sort of symmetric cipher. Our secrets table now looks like this. We no longer have to trust the DBA. We no longer stay awake at night worrying that someone is stealing the disks out from under the database systems. Everything's good to go. We are in the clear. Now we have an encryption key. All right. Where do we store the encryption key? We have some options. We can hard code it. Put it in the app. No, that's not going to work. We can put it in an environment. No. We already ruled that one out. We can put it in the database next to the things that's encrypting. So that's not going to work. But we can encrypt it in the database. We have a dilemma. I like to call it secret zero and it's how do you handle the initial credential to unlock the next round of locks, to unlock the round of locks after that to get to the secrets in an automated fashion that doesn't involve a human operator to go in and set a password on reboot. I have a solution to the problem that secrets are tough. Handling them is difficult. Dealing with them is difficult, but it is a necessary part of our life as application developers, systems architects, and cloud engineers. The solution I'd like to talk about today, unsurprisingly, is Hashicorp's Vault product. It's an open source, secrets, credential storage system. It's been around for several years. It's really nice. I really like the Vault software. It's easy to use. It has some serious thought put into how it does security, both in resisting offline attacks, resisting online attacks, access control. They've got a ton of logical back ends. You can do PKI in vault. You can do straight up key value store. You can do Amazon integration. It's got everything. But what about Cloud Foundry? How do we take this really awesome piece of software and integrate it into our CF app workflows? There's a broker for that. It's out in the Cloud Foundry community. It's called Vault Broker. I want to go through a quick demo. So I have in my possession a Cloud Foundry instance set up just for you fine folks today for this talk. In it, I've got an organ of space, and we are going to take a look at a service broker. So I've got the Vault service broker deployed. It's registered into... Oh, well, that didn't work. That's going to be a lot of fun. Let me see if I can... Is that better? There we go. All right. As I was saying, I have a Cloud Foundry instance in my possession for you fine folks today. It has a service broker. The Vault Broker is currently registered to it, and it's available in the marketplace and providing the Vault shared plan. Now, the way the Vault Broker works is you hook up as an operations team a Vault to the Vault Broker. And you say this Vault Broker fronts this highly available console backed, file backed, database backed, however you want to deploy it, Vault. The broker responds to the open source service broker API requests for provisioning by going out and carving up a chunk of the secret KV store hierarchy. Names it after the instance ID. Binds will create a new token, an initial token that has access to that subtree. So basically, every service will have its own view into the Vault, and they're all completely isolated and segregated. So we're going to create a service. We're going to call it Vault, a demo Vault. It's the Vault service shared plan. There's our service, whoopee. And I have an app. This is just CFN. Nothing terribly exciting here. We're going to bind the service. And at this point, the application now has service credentials to actually connect to the Vault, either via its HA address, singleton address, using the token. And its prefix is included. So here, our route is our path right here, secret C3FF. This root token is basically akin to a user name API key, for lack of a better word, for getting into the Vault and managing it. It's running over here on this 10.0 1629 address. And that's the Vault broker. Go back to my slides. We also have a UI. And this was important to me to have a UI for this demo because what I just showed you isn't terribly exciting. We've all created services. We've all bound services. The interesting bits are in actually using the services. And it's really hard to show plumbing like this without an actual UI. So I wrote a UI. This is my personal GitHub org. I'll probably move it into Cloud Foundry Community after this talk is over. And we have another demo. I should warn you there's actually three demos in this talk. Lots of switching, lots of fun stuff. So back to Cloud Foundry. We have our API. I'm going to go into the app source. I'm going to push it. And I'm going to not start the application because the Vault UI does not like coming up with that Vault for obvious reasons. It doesn't have a whole lot to do. And then it falls over. So once we're done deploying all this, we're actually going to take, create another service, bind that service to the UI, and then we're going to see where the magic is. So here we are creating our service. This one's called Vault of Secrets. Binding. And then we'll go ahead and we'll start that Vault app. So the Vault UI actually approximates safe. I don't know. Has anyone here used safe? Couple people? Have you used safe, Dr. Nick? Couple times, once or twice? So we approximate a lot of the functionality that safe has on the command line, and we bring it to a browser. We've got searching. We've got the ability to generate random types of credentials, different types of keys, SSL certificates, which is one of my favorite parts. And this is what it looks like. So I go up here to plus. I can create arbitrary secrets, which are just objects with keys and values. So username is admin, password is secret. I can generate an SSH key pair. If I need to increase that. Of varying strengths, I can test James SSH. I'll make myself a 2048 bit key. There's my key. I won't use this for anything because it's on the screen. I can also create RSA key pairs in varying strengths. I can create X509 certificate authorities, and then I can use those certificate authorities to issue actual X509 certs and in so doing build a whole tree of CAs for an organization or a set of applications. We're going to skip that though. We're going to go back here and we're going to look at searching. So I can come in here. Take a look at the app or the secret that I generated. I can copy and paste these into various places if I need to. And we also have the not at all ripped off from one password, revelation of secrets that are currently in password fields. Totally original design, not stolen from agile bit software in any form or fashion. So now we have a vault UI. I also want to talk about an example app and a fun little strategy you can use with the Vault Broker plus an application is to bind a service or create a service and have one vault that you share across multiple applications. One of those applications might be the Vault UI and you can go into the Vault UI and you can generate secrets that the other applications can go in and consume. They can pull in keys based on path and you can say use this for your basic auth. I will put it there for you manually. You don't have to generate anything and the operators when they push apps don't have to do it. And then they can kind of live and persist beyond. So I wrote an example application. This one will not be going into Cloud Foundry Community because it is utterly pointless, but it is a bit of fun with API Gen. As promised, here's the other demo. We will go through all the fun bits. Again, we push it without starting because it definitely needs a vault to be deployed. And what API Gen does is very simple. It connects to its vault, which we're binding that same vault of secrets. We didn't create a new service. This will be the exact same one and then we'll start it. So API Gen talks to the vault on the web interface, which we'll see in just a minute. It asks for an email address, which it purports that it will email an API key to you. It will go into the vault and generate that API key, store it in the vault, ostensibly for retrieval later by people like me who forget what email address they use for random things. What we're going to do is we're going to go into API Gen, we're going to generate a key, then we're going to go to the CF Vault UI and we're going to verify that everything's wired up properly. Of course, it looks terrible in this browser. Let's go into the other browser. So API Gen, before we can get started, we will need to get you an API key. This is from my really cool new service. We're going to click give me a key, give it an email address. Boom! It has given us an API key. Don't write this down. This is secret. Please don't disperse or disseminate this information. Zero, two, F3, et cetera, et cetera. Now, if I flip back, close that, if I flip back into my vault and I look up tokens, I have this which I didn't have before. And we can double, triple check that by going and getting another one. And I'll do jhunt, startinway.com, give me a key. This one's 2613. If we flip back over to the Vault UI and rerun the token search, I now have two tokens, both stored securely and secretly in the vault. And if we go in here, 2613, e1017. 2613, e1017. So this is an interesting, to me, illustration of how you can use something like the Vault UI and any other applications that need to communicate secrets, as long as you can bind them in the same organ space to the same service, you can actually securely share credentials without having to put them in environment variables. Great. There are those. So Cloud Foundry gives us the tools. Vault and the Vault Broker give us the rest of the implementation for secrets. Also, Apigen loves you. We will go back to my final slide, which is the end of the talk, running a bit light on content, apparently. So I'm James Hunt. I work at Stark and Wayne. We build lots of things for Cloud. We love Bosch. We love Cloud Foundry. Please come see us at our booth. I think we still have t-shirts to give away. If we don't meet our quota, we won't get to come back next year. So please stop by, have a chat with some of the fine people manning the booths. And are there any questions, thoughts, interests, demands? He's got the mic. Hi. So what would you do if you don't own the Vault and you don't have access to create backends? That's a really good question. I might know the answer. You already know the answer. Yeah. What would you do if you couldn't deploy a Vault and didn't have the ability to create backends? So yeah, we just went through this at Garmin. We are using Vault, and we started with this broker and realized quickly that we didn't want to have Vault dropping backends as new orgs and spaces came up. And so what we did is we enhanced the broker to the point where it now makes API calls back into Cloud Foundry, gets a hold of the org and the space where an app is binding, and then drops or creates a token with a custom role that we have predefined knowing that we know who the orgs and spaces are. So the only back end now that comes up on the fly as the broker runs is the one where it's actually keeping track of the bindings themselves, and everything else we're prewriting all our secrets for the applications as we need them. That's interesting. Yeah, so it's a different paradigm than, hey, my app needs to write secrets, my app needs to use secrets, and I don't have control over where those secrets are or even to create backends in Vault as a Cloud Foundry broker. Maybe you would need them also to make it. And at the end of the day, that's one of the things I absolutely love about Vault. It is not very opinionated about a lot of things, and it's incredibly flexible in what it allows you to do and just compose. It's crypto primitives for however you need to use them. So you had a question? Yeah. How do you actually deal with the chicken egg problem about the passwords whenever you deal with that UI, for example? Like, now we've got another UI that you need to log into because the whole secrets are there. Right, so I will point out that the CF Vault UI, if the keen observers in the audience will notice, we did not have to specify any authentication, glaring security hole in the CF Vault UI, one that I hope to remedy by actually having credentials in either via authentication via UAA and space-scoping things, or allowing the Vault operators to set up passwords in the Vault for the broker to use and to pass back to the UI. So it's an unsolved problem. It's definitely a problem that we're investigating and trying to figure out solutions to. These are very much cutting-edge pieces of software. I believe they were both written within the last four months. It's fair to say it's amazing, but how do you actually, whenever you solve the problem, what you will be doing with those passwords? How do you let the user know? Like, where are you going to store those? Right, now you're back to the secret zero problem. We can't ever actually get rid of secret zero. That's the unfortunate problem that all the smoke and mirrors in the world can't get around. Unless Mr. Sarabian has a talk, I believe, when is your talk? Friday at 2. You want to be there? I believe you're using the cubbyhole backend? Yeah, the cubbyhole backend. Secret minus one. One way people have solved that issue is based on which IPs have access to that. That way, you need to come from that IP to get hold of the initial seed. And only those are authorized to get in. So you protect the front door to your building. So one of the things we've done at Sarkin Wayne with Vault is we've integrated it with our concourse platform automation tools. And one of the ways we've done that is via the, I believe it's actually now deprecated, the app roll app ID back ends. We're super cool at the outset because they allowed you to say, here's two parts of a username or two parts of a password. And they'll be known by different competing systems or different isolated systems so that they can't easily be put back together until deployment time. And it's only valid from this subnet cider range or this specific slash 32 so that the concourse workers can be told via pipeline and some other thing, either pulling stuff out of a vault or a cred hub or any other integration that here's the secret key that you and only you can use to go out here to do this thing to pull creds back. And that's one of the, like I said, Vault really cool pieces of software. I encourage you all to go play with it. Like I said, we do have two Bosch releases out in Cloud Foundry Community, one for experimenting with Vault and one for running Vault in prod. We still have five minutes. Four questions? Thank you. So what are you doing now that cred hub is being released and used in the community? Using it? So are you not using Vault in those cases? So what we do, and I'll probably end up segueing into the Genesis side of one of the things that I work on, big project I work on, we use cred hub for things that we can auto generate and forget, specifically on the platform deployment side. I work mostly with Bosch, by the way, I do some Cloud Foundry stuff, but a lot of my day-to-day is in Bosch and deploying of Cloud Foundry on various clouds. So a lot of the things that are in a typical Cloud Foundry deployment are completely irrelevant to day-to-day operations. The CA certs for mutual TLS, the NATS TLS configuration, very little of that information is used by operators for troubleshooting things. I can't remember the first time I ever had to get the NATS client cert so that I could test the NATS connection to make sure that Bosch was able to see the agents come in. Like, it's not a thing that we have happened. So we segregate our platform cert or secrets into things that operators care about, which go in a centralized Vault that's managed out of band, and things that only the Bosch deployments care about. And those go in Crathub because Crathub has excellent facilities for just saying, I need a cert, and I need it valid for this, and I need it valid for this many years for this name. Go do the thing, and then later I'm going to ask you to rotate it, and I don't want to have to remember any of that state. You do that. The concourse story with Crathub is a bit different because I think they're trying to solve slightly different problems than the Bosch team was, but I think there's a place for both technologies. I think they complement each other quite nicely. Yeah, some people are even taught about putting Vault behind Crathub. We actually had a... I think Tom's still working on that, actually, to do a Vault-based config server implementation just to see where that goes. So, yeah, they're complementary. Cool. Oh, one more question. All right. Hey, what are your thoughts around integrating with other existing services like directory services, LDAP, or internal services? Integrating with, let's say, LDAP, like AD that we already have for credentials and other management and integrating with HashiCarp. Oh, is that an auth back-end for Vault? So, instead of Vault... Vault stores credentials, right? So, you can have credentials. What if we have credentials outside of the Vault? For example, in LDAP, like, you know, AD Active Directory services, we have passwords and such, can it integrate so that we can use Vault for kind of a broker? That sounds like a federated model for this. I haven't put any real thought into it. That's the best I got. It's an interesting space, but integration with credentials is a very sticky situation. It's very, very messy. I think if you go to the booth, they'll give you a free t-shirt, and you can chat about it, and maybe even build a business relationship. We'll talk closer and more intimately. All right. So, Ernest, there is an urgent question. This is it for the first part of extensions. You now can go eat, come back to 35, and either Jeff or myself will be here for the next set of talks. Thank you for coming.


Our next speaker here, Trey Foghetti is here to talk about the challenges of user privacy in a location sensitive system like 911. Uh, he's part of the 911 association. Uh, this talk is being recorded for, to show it online later on. So if anyone has any questions, please walk down the, uh, to the microphone at the center aisle and we can record the questions too. So without further ado, Trey. Thank you so much and thank all of you for being here this afternoon. I know your time at DEF CON is precious. Um, and, and I will say for my part, I very much want to make this talk, uh, a lot less formal and a lot more, um, interactive than some of the talks that you'll see in the other tracks. So if you have at some point a question about anything that I'm saying, please don't wait till the end to come up to the microphone. And as soon as I get to like a convenient pausing point, I will, uh, take your question because, uh, this is a really, um, interesting area that I'm very passionate about. I've been told I can't be heard, so let me, I'm just gonna hold this. How about that? Everybody hear me now? Good. So, um, where we start is, I, I get a question all the time. So you guys at 911, you're just tracking everybody all the time, right? That's how you know where I am. And the answer couldn't be further from the truth. Um, we are not tracking you at any time. In fact, you would almost be appalled at how little location data we do get when we want to find you, which is when you call 911. So my talk is gonna go over a little bit about what we do get when we get it, um, how we are actually part of the privacy solution by virtue of keeping people who want access to location data away from it when they shouldn't have it, and how we, uh, acquire and store and use location data when you make it available by calling 911. So, um, at the outset, I think I want to start with just a quick thing about how location works. Are folks here familiar with how GPS works? Like, generally? Okay, I'm gonna go into it just kind of at a very high level. So basically, a few thousand miles up, we have a constellation of 30 odd satellites orbiting the earth, uh, roughly every 90 minutes. And each one of them is singing a, a, a unique song, but they're singing it on the same frequency. And by singing the same song inside a receiver, essentially, you can find out how far off in time your song is from the one being sung by the satellite. And that tells you a time offset. So you get, we multiply that by the speed of light, correct for things like the, uh, diffraction of the ionosphere that delays these signals. And then once we have ranging from multiple satellites, then your device can calculate its location in, uh, 3D space. Now, it, it's pretty good horizontally. GPS in the best of cases can give us, you know, 5 to 10 meter fixes most of the time. If you have a really good device with a great antenna and, you know, a multi band receiver and you leave it stationary for a very long time, you can get that down to like a 10th of a meter reliably or better. But for this typical case of something that's cheap enough to put in a smartphone and for us to carry around with us every day and, you know, doesn't have a gigantic antenna sticking out the end of it, that's not the, really the reality. You're not going to get anything nearly that close. So historically we've had this problem of we need to find people when they call 911. So how are we going to do that? Well, in the wire line world, it was easy. We knew where the telephone company's wires went. They kept a billing database that said, you know, here's where our wires go. And when you established phone service, we could go to them and say, look, for every address that you put in your database, you have to validate that with somebody. It's something called the master street address guide for your locality, which is a table that contains all of the valid street names and all of the valid number ranges within those street names. And so they could submit that address, make sure that it was valid. If it wasn't, they had to go back to you and say, Hey, it looks like your street doesn't exist or potentially your numbers out of the range. Like what happened? Did somebody build something new? And if that's the case, okay, we got to go fix the database. But there was a process. We knew where everything was. Smartphones aren't like that. They move around. Cell phones generally move around. And so we had to come up with ways to figure out where mobile devices were. In the early days, there were two very general ways of doing this. There was called the network based model and the handset based model. In the network based model, devices in the network either transmitted or received signals to or from a device and used information like angle of arrival, time difference of arrival or absolute timing offsets to calculate ranging and bearing data and figure out where the device was. And that worked. It's actually very fast. It's one of the fastest ways to find someone generally. The problem is it's not terribly precise. With those sorts of methods, you can get within two to 300 meters very quickly, but you'll never get down to some tens of meters. It just doesn't work that way. So that was the typical technology in CDMA or excuse me, in GSM networks, because in GSM networks, that was what was convenient. They had the right sort of signal structure to do that kind of ranging. CDMA had at the time a really cool advantage which is when I talked about the way GPS satellites sing little songs, they're actually doing that in a kind of a version of CDMA. All the satellites transmit on the same frequency, but they do it with different pseudo random number sequences imposed over that. And you can pick individual satellites out by knowing which one of those are, looking up in an Almanac, okay, what satellites should I be able to see? Getting that from the constellation itself even, and using that data to figure out, okay, what should I listen for? Since CDMA handsets already had correlators inside that could do that for the networks they were attached to, someone had the bright idea, well, let's just reuse that as a GPS receiver. And it worked. It worked brilliantly. Now the trouble was early on there was only one because they only expected you to be carrying on one voice conversation at a time. So when in the early days they wanted to figure out where you were for 911 purposes, they actually had to say hold on a second, you're gonna hear silence while I determine your location. And then they would trigger, the 911 center would trigger something in the carrier network to actually start that process with your device. The device would listen for the satellites, collect the data, correlate it, ship it off to the network, the network would calculate your position and relay it via a database to the 911 center. Really clue G and you don't want to tell somebody in the midst of an emergency, please hold, right? That's not a great thing to tell someone when they're like screaming to death. So so over time, we got better, we got better chips with more correlators, we got to the point that we could actually put GPS chips in almost every phone. And these days, that's what everybody does. We have GPS chips. Most of the network technologies have sort of gone away. We still have two out there called advanced forward link trilateration, which is a ranging technology used mostly in CDMA networks. And then OTDOA, which stands for observed time difference of arrival. How many navigation nerds are there in the room who have heard of Loran? Okay, we got a few people. Good. So this is this is the really cool thing. OTDOA is just Loran with cell towers instead of Loran chains. It's you just listen for the time differences between transmissions of synchronized towers. And then you can use that to do hyperbolic ranging, which is really fun. It's kind of a cool sort of thing. Again, it's one of these network technologies that are fast, but not necessarily super precise. All well and good. Okay, fine. You've got a met a measurement system. What do we do with that now? In the 911 case, and I said this before, we're not tracking you. We don't want that stuff running all the time, because frankly, we don't have anything to do with the data, and it's really not of any interest to us. We need to know where you are when you have an emergency. You don't want that going on, obviously for privacy reasons, otherwise you wouldn't be in the crypto privacy village. And you also don't want it going on because it's a very battery intensive activity to keep track of yourself precisely all the time. In fact, most of the things that you see, you know, this question, why is it that Uber can find me but 911 can't that we hear so often? I really hate this question, because like most of the time, Uber is not Uber doesn't hasn't measured where you are to any degree of certainty. They have software in your plat in your device platform and in their platform that's making a very educated guess. People tend to focus on the little blue dot that tells you where the thing thinks you are. What they often don't realize is that there's also a blue circle around that there's some uncertainty associated with that location fix. And often in the case of commercial apps on smartphones, that uncertainty is actually relatively large, it can be on the order of hundreds or even thousands of meters. That's calculated at about a 90% confidence level most of the time, which we want we want about a only about a 10% margin that will find you outside the circle. But if you think about a place like Caesar's Palace, an uncertainty of 100 meters or 200 meters makes it really difficult to find someone who makes it really difficult to go search for them. 200 meters might put me in any one of three different hotel towers, the casino, the conference center, and a bunch of restaurants. So you really have to deal with constraining that uncertainty somehow before you can use that for emergency response. Uber can make those guesses because it has that ability to then ask you, okay, where are you? In fact, does this match up? And if not, drag the map and pick the little dot, we don't have that luxury in 901. So I've said that we're not tracking you. Well, what are we doing? When you call 911, a few things happen. And I've talked about some of the different methods that we've used. We're starting to add things from the handset now. Historically, only stuff that was baked into the baseband or the network was really available for 911. But these days, we're actually doing a lot more than that. This is bleeding edge stuff that's just starting to come into the devices. But we're starting to be able to use Wi-Fi and Bluetooth, both to do sort of associative location to say, look, we know that Trey's Wi-Fi access point is at this particular address in this town, because he's there all the time and he's associated with that and so on. So if the phone can see Trey's Wi-Fi access point, that's probably a location of association that we can use as a starting point for a search. It's kind of a probabilistic sort of thing. That's a very uncomfortable proposition for 911. We want things to be deterministic. We want to be able to say to the cops and firefighters in the field who really couldn't find their way out of the building without our help, that this is how you, this is exactly where to go. But that's not realistic. That's just not the way the physics of measurement work. And that's why the other thing that we're looking at Bluetooth beacons, you put Bluetooth low energy in a light bulb for lots of really great reasons. One of the things you can do with that is associate that UUID, that Bluetooth beacon ID with that location in a database. We're working on something called the national emergency address database, which will actually associate Wi-Fi Max and Bluetooth UUIDs with physical locations in the world. So that if, for example, there's a beacon in this particular ballroom, I can be localized if I'm within 10 to 30 meters of that, I can be localized very, very precisely with that. But of course, that takes getting a lot of beacons out in the world. Thank you, Internet of Things. That's going to be easy. But then it also, it means we've got to track those things and make sure that if they move, we find out about it, that, you know, there's some aging in the database to make sure as things get older, we trust them less to provide location and so on. So at call time, when you make a 911 call, a couple of things happen. First off, the, your platform, your, your phone platform, whether it's Android iOS, BlackBerry, BlackBerry, whatever, completely overrides your location privacy settings. It says, look, you've made a 911 call, this is really important, turn everything on, we need to find you. Historically, that's only meant like the GPS chip, and you know, not some of the other cool things that you have in there, like the accelerometers, gyroscopes, magnetometers, barometers, Wi-Fi, Bluetooth, etc. But we're working to change that. We're actually working to get to the point where we can ingest data from all of those different things to create something called device based hybrid. And this is where we, you know, use those, use things opportunistically where they make sense. So GNSS, GPS, Galileo, GLONASS, Compass Baidu, all of those things work phenomenally well outdoors in urban and suburban environments, and even to some extent in sort of urban clusters, not necessarily in Manhattan and San Francisco and downtown Atlanta and Chicago, in those places GPS is really, really awful. But Wi-Fi and Bluetooth things have much higher densities in those environments and consequently they are of more use. And so this is, the concept behind device based hybrid is, we're going to opportunistically use what is best in the environment that it's in. So if you're in Manhattan, and the best thing we have to like figure out where you are is Wi-Fi and Bluetooth, we use that. And if not, if you're out someplace where GPS is working great, we'll use that. All of this is just, sort of starting to happen. We've had one company called Rapid SOS that built some really cool stuff to kind of get location data around the database limitations of the legacy 911 network. And in doing that, they've kind of proven out that you can get a lot more calls clustered down in that, you know, zero to 20 meter range with device based hybrid. And I don't mean to like endorse rapid or something like that. There are tons of folks working on this. Google just announced a few months back that they're going to be doing a lot more in their fused location provider for emergency services purposes over the next three or four years, including, and this, I thought this was really cool, they're putting dual frequency GPS receivers in their devices, which allows you to directly measure and cancel the ionospheric diffraction delay. And that is the primary source of error in a GPS fix. So in three to four years, your handset GPS locations indoors and out are going to be phenomenally better. They really will just get meaningfully better. We're also saying barometers go into devices, which is really cool. These allow us to implicitly pull out a vertical location. If you have, oops, I've gone dead. If you happen to have, I think so, there we go, we're back. If you happen to have a barometer and you have a good external reference for what sea level pressure is, you can use those measurements, you can compare them and say, okay, I must be this much up or down from that reference datum. Now, in order to do this, you've got to have measurements that are fairly localized. And so, you know, another example, this company, Nexnav, their idea is to put sort of pseudo lights or fake GPS satellites on cell towers and buildings and broadcast basically their own GPS signal down at 900 megahertz where it penetrates deep into buildings. And to broadcast as part of that hyper local barometric pressure corrections that let them pull up exactly the height that you are at with a great deal of reliability. Now, what do we know about 900? 900 is the junk band, right? Everything awful is in there. It's completely useless other than wireless meter readers who just sit on top of everything. Except that the one thing that is actually privileged in 900 megahertz is multi lateration and location monitoring services. And there are licenses and these guys happen to have a lot of them. There are other folks who have them for other markets, but this is like a cool thing that you can do in the 900 megahertz band and you get, you actually get interference protection if you do this. So another cool thing. When you make that 911 call, all of this stuff kicks on and starts saying, okay, now where are you? Initially, when your 911 call arrives at a 911 center, most likely we will not have good location data at the start of the call. What typically happens, and this is something called phase one and phase two, phase one location is what we get at the beginning of basically every wireless call. And it just tells us the ad street address of the cell tower that you're attached to. And it's the GPS coordinates of the tower. Again, this is not used, this is the tower. And it tells us the central bearing of the cell sector that's serving you. So if you look up at a cell tower, it's usually kind of triangular. It's got antennas on the vertices of the triangle. Each of those has a particular central bearing. It's serving about, you know, 60 to 120 degrees. And so by knowing that kind of general bearing and then asking you questions, hopefully we can figure out if you don't know where you are. The way all of that works is super-clugey. It actually happens because of a database process that got started in the 1980s built for the wire line world. And that's why before we can get your more precise location, we have to wait a period of time for a couple of processes to happen. So first of all, the device has to actually calculate a fix that is better than whatever it had from, you know, the tower and so forth. And then it has to communicate that back to the carrier network. The carrier network has to pass that to a database management system provider. They have to further process it and implant it into the database that ultimately your local telephone company typically maintains because it was the one that they maintained for that wire line process oh so long ago. Then at that point, the 911 center system either automatically or manually can re-query the database and say okay, is there anything better? And typically by the 15 to 25 second mark is when we'll start to see an improvement. Something like a GPS fix that's within the within the allowable error range. The carriers have location targets that they have to hit. Ultimately those are going to get down to 50 meters horizontally and some number vertically that we don't know exactly what it'll be yet but we're working on that. Or a dispatchable address. Dispatchable address is the process I talked about before with with the database piece of it. Now the one sort of worrisome part about this is those values, those location values, we know they're very sensitive, those are actually transmitted in the clear. But they're transmitted in the clear only within sort of trusted carrier networks and then only on the network's control plane. So the location information never touches the user data plane until it actually gets down to the 9-1-1 center and it is received at the start or you know some number of seconds into your call. And then at that point we kind of we look at this as like a hot potato. We'll keep that store as long as we have to to provide you with service and to keep the sort of records that 9-1-1 centers are legally obliged to keep for whether it's for investigatory purposes or just because the legislature said we want everything to be kept for 90 days or 120 days. Some things may get archived if they look like they're going to be important down the line but not necessarily everything. I mean we typically want to get rid of non-critical 9-1-1 data that's not subject to a litigation hold as quickly as possible. We really just want to get that out of the system. This is where it gets a little interesting because obviously being in public safety 9-1-1 is part of a lot of law enforcement responses. A lot of law enforcement responses start as a result of a 9-1-1 call. In fact almost all of them do. And 9-1-1 is known for having location data. People in the law enforcement and fire and EMS communities know that when you call 9-1-1 we get your location data. And so people not unreasonably often make the assumption that well if I need that for some other purpose the way I should be able to get that is by just going to my friendly neighborhood 9-1-1 outfit and saying hey help me out with some of that good good location data right? And we get those requests all the time. Our members actually get location data requests from law enforcement and fire and EMS often when there's no actual 9-1-1 call going. So I said earlier 9-1-1 really is on the front line of protecting user privacy because often the thing we have to do is tell sometimes even the boss no because we get 9-1-1 data for a purpose we use it for a purpose and that's part of user trust in the 9-1-1 system is that we're not going to misuse that 9-1-1 location data. The other reason that these things happen from time to time is we also have access to a process for exigent circumstances. So if you've ever seen the movie The Call with Halle Berry so small aside I got to meet Halle Berry she is amazing in person she is even more beautiful and like even more friendly and awesome and wonderful and she really like took a lot of time and effort to get in the trenches with real 9-1-1 call takers and learn how they do what they do which was really awesome and cool hopefully there will be a sequel but in that movie there was this occasion where a young woman was kidnapped and thrown in a trunk and she made a very she was only able to make like a quick discrete 9-1-1 call to say you know help I'm in the trunk right when that happens we've got to have some pro some process to get some location data when you're not on a 9-1-1 call now the challenge for us is as I talked about before all of the location privacy settings and all of those sensors and measurement systems are not on unless you're in a 9-1-1 call so the moment you hang up we lose the ability to query for updated location information and at that point the best we can do and I realize this sounds insane in 2017 is facts yes facts a form to the carrier that was serving that 9-1-1 call to say we need the best location data you've got for this particular phone number the turnaround time for that can be 15 to 45 minutes they generally they do take it very seriously they try to get the exit circumstances request process as quickly as possible but it is it is still a process like it's not an automated kind of thing it is a very manual process and then what happens once the carriers lawyers and engineers have looked at the form and said okay yes this is a valid exit circumstances request there really is an ongoing emergency with a threat to life or property then then and only then can they start to do a little bit to help us but again you're not in a 9-1-1 call they can't turn on all the sensors typically the best we can do is we fall back to that sort of phase one information we can typically get okay what tower and sector is serving the device so we get you know kind of a point and a bearing and if we're really really lucky in some cases they will also be able to give us a range estimate so I talked before about you know GPS and the way the ranges kind of crossover and you can get a location out of that another way you can develop a location fix is with a bearing and a range so that's a line and arc and wherever they cross that's sort of generally where you are because again our measurements are kind of fuzzy that arc can be really wide that line can be really thick you get kind of an annual or sector but hey that's still useful data but again it's not data that we're getting easily it's something that's very very protected and in the 9-1 case when we do get that again that's a manual process so there's no database to put that in that's actually something that's done on a you know we're on the phone with AT&T Verizon Sprint T-Mobile somebody saying okay where are they now okay has the location changed where are they now and it's this very sort of iterative manual thing and eventually hopefully you can work out okay it's a car that's going down you know I-40 or something like that and say let's go try to intercept whatever that is and and find somebody and then of course immediately after that exigent circumstance passes that location data that was used for that purpose basically has to go away because it wasn't necessarily obtained with an investigative purpose for law enforcement purpose so then you've got to start thinking about okay do we get a subpoena do we get a warrant how do we actually acquire that location data in a way that can be used for some criminal prosecute prosecutorial purpose that's not you know related to an immediate emergency response so that's kind of the multiple phasic approach that you have within the 9-1 center we have the sort of baseline case I weirdly I often hear people say one one two which is the European emergency number is is really part of the global emergency calling standard and you should use that and in a lot of places using one one two might actually get you to a 9-1-1 center like it might it won't always but it might and if it does the challenge though is it won't turn on usually any of that cool location stuff so don't dial one one two unless you're in Europe once that location process is over with there is sort of a residual question of what do we do with the data and I don't think we have a good answer right now because that your average 9-1-1 center is not technically sophisticated enough to be encrypting or hashing or doing things like that with those sorts of data they kind of have to rely on whatever their system vendor makes available and oftentimes that doesn't include robust confidentiality and integrity mechanisms but that is something we're working on so the next piece of this that I want to talk about is how we will acquire and use location information in a next generation 9-1-1 environment and basically the process still stays heavily dependent on what the device and the network knows but now instead of dealing with legacy wireline databases from the late 70s early 80s we're going to take this into a more IP based realm so in next generation 9-1-1 when you dial 9-1-1 from a phone whether that's a voice call to 9-1-1 or a video call or even a text session we're going to have real-time text very soon or total conversation if you have you know different accessibility needs you'll have the ability to communicate in all of those different ways simultaneously which is really awesome if you have a hearing or speech disability at that point you've got a couple of things that have to happen first the device needs to figure out what 9-1-1 system it's call should go to so it's going to query something in the access network providers network and here I want to stop and define a couple of terms because historically when you bought phone service from the phone company you were really buying two very different things but it looked like the same thing because it was all you knew when you bought voice service from the phone company you were buying first off a transport network so that's an access network that that is capable of moving things around but you were also buying an originating service the ability to connect from one end point to another end point using some you know particular address today networks don't have to do all of those things in an integrated way I can buy my access network service from for example AT&T and I can buy my voice originating service from Vonage for example we have to accommodate that in the 9-1-1 system we have to be able to deal with the fact that your carrier is no longer just the phone company it may be somebody else historically we have not done a good job of that because we've just bolted stuff on to the 9-1-1 systems over the years and we haven't really integrated a lot of these new technologies into the way the system works so your device goes to the access network provider and well okay why will the infrastructure provider they know where their wires go they know where their cell towers are they're the ones who have the sort of intimate real world physical connection with you that can be used to start that process of localizing the call so your device goes and queries their location information server it gets back currently actually location by value it gets sort of your here generally although we're trying to be very careful about giving that to devices because you know we can see a state where you might have malware on the device and you don't necessarily want to disclose location to that so we're working on some cryptographic stuff to try and keep that private as well to basically tokenize it so that then the device can pass either that location value or that token to something called an emergency services routing proxy and the purpose there is for the proxy to say to a big thing called a forest guide which is basically a hierarchical database structure that tells you here are where all the countries in the world what are their boundaries within those what are the states within those what are the counties within those what are the 911 center service areas and oh by the way you're there great you go to this 911 center that's where your call should go so this involves a lot of GIS processing that's geospatial information system I think I have a slide here that actually shows some of that so we actually have a standard for geospatial information systems remember I talked about that that old school way of doing the database of addresses where you have a list of street names a table of street names and a table of valid number ranges that's all well and good if you only need to locate people in like row houses but what if you have an apartment building how do we know that a floor number is valid how do we know that a room number is valid and moreover how do we decide where do all of those 911 calls within that thing go for example if I'm Caesar's Palace I might have my own I might be a limited municipality I might have my own response center that can get you help faster we might want to route all of the calls within Caesar's Palace to them instead Disney World's a great example of that they actually have the Reedy Creek Improvement District there are like six Disney executives that have houses like on the property they're the entire voting block for the whole city of Reedy Creek and and they actually have their own 911 center it's like a best practice place everybody loves to hold it up is like shiny and new good place to go but you have to be able to track those things so that you can do change management so that you can know when somebody puts up a new building how do we add that to our addressing system and then once you've consulted the forest guide it'll give you an IP address for something called the border control function of the ng911 system next-generation 911 system that actually serves that particular location at that point the routing proxy whether it's in your access network providers network or if it's disaggregated in the originating service providers network will actually send signaling and media to that border control function and then my to me the coolest part of all this is if at some point during the call things change and there's some reason your location changes or whatnot and there's some reason to redirect the call either before the setup is complete or during the call there's actually an intelligent mechanism to say hey there's this other 911 center that might need to be involved in this response do you telecommunicator handling the call want to loop them in even if it's on like a listen-only basis and you can do that and you also get another kind of cool capability is you can have now specialized 911 centers that deal just with particular populations that have very specific needs so for example in Washington DC we have one of the largest universities for the deaf and hard of hearing Gallaudet great school they actually do a lot of cool engineering work related to 911 stuff but if you have a 911 call that's coming from just the area of the Gallaudet campus you might want to put that into a different queue for someone who has a hot connection with an ASL interpreter or knows ASL themselves and can handle that video call that way or who's trained especially to handle real-time text calls and that's something in today's environment we absolutely can't do because remember back we're routing everything based on that cell tower and the sector at the beginning of the call that's all we know in the future the idea is we're going to route based on the actual X Y and hopefully Z someday from the device and that makes it a lot easier to do much more granular things like that that can really help it has a huge impact if you're in a population that has a specific need like that it's a really big deal to be able to get to the get to something like that on the first try because today one of the one of the terrible things that happens is if you need to connect with somebody like a video interpreter that can delay your 911 call on the order of 15 to 30 minutes and that is not an amount of time you want to be waiting in an emergency I've used about 40 minutes here I haven't had anybody volunteer questions so I just want a level set are there things that I've covered in insufficient depth or not covered or other things you want to hear about so that I can kind of tune the the rest of the talk yep oh it's funny you think there's IP switching involved in this that's that's very quaint yeah no so the question was when I said that location data is transmitted in the clear what part of the sort of transactional structure was I talking about so from the device to the network it is encrypted at the level of whatever the the bearer established for that so in LTE it would be encrypted in GSM it would be encrypted probably weekly same thing in CDMA within the carrier network it will be segregated but not necessarily encrypted and then the way it's and I know this is why I laughed about the whole IP switching thing the way it's actually transmitted within the telephone company networks is as a series of either medium frequency tones or maybe DTMF in some cases like it's really kind of hilarious how some of this stuff still actually works the data block we get with a 911 call I think I said this in my talk the other day but it bears repeating is 512 maximum 512 non extended ASCII character set letters and digits and that's all so when you want to represent something like a physical location it that's all you've got to work with and so we really had to like figure out okay how do we dumb this stuff down one of the big things that we're looking forward to is once we have these next gen systems where we can protect it properly encrypted and it is an IP system is actually starting to be able to represent things graphically I think I showed here's an example of like an uncertainty around a floor for example today we couldn't do that we couldn't pull up a visualization of a building and put circles or ellipsoids or spheres or other things in there and say look in this general area because we have no way of representing that in the 911 systems that we have now the cool thing is the carrier standard that makes all of this work at kind of the numerical level is something called J standard 036 I think it's an attis slash TIA standard it actually defines the ability to send different geometric shapes so I can send an ellipsoid point with uncertainty I can send a point with spherical uncertainty I can start to do you know annual or segment things like that nobody does that because again we don't have any way to represent it today so the question was what percentage when it comes to dispatch is sent in the clear versus encrypted or perhaps digital I assume you mean and I assume you mean the radio part okay so the radio part of dispatch how much of it is encrypted how much is in the clear overwhelmingly it's in the clear there are places that are starting to do more encrypted communications badly in most cases if you look Matt blaze Travis Goodspeed and some folks did an analysis a few years back of one like the dominant public safety radio standard in the US and found that it was basically hopelessly insecure they used a really weak kind of even more broken version of RC for as the as the stream cipher in some commercial implementations that were very cheap and so everybody used those even though they weren't terribly interoperable and they're not at all secure so we're in the process of trying to fix that earlier this year the Department of Homeland Security and NIST work together to get a requirement into the standard so that I think starting in 2018 going forward all radios built to that standard will now have to have a S 256 as a mandatory part of their cipher suite they can still have broken versions of RC for but a S 256 has to be in there as well and part of the reason for that is in the public safety community interoperability is a huge deal disasters may start local but they usually don't stay local and when they go beyond the you know disasters don't obey jurisdictional lines no matter how much we admonish them and so eventually you've got to have help from your neighbor and when that happens being able to talk with them as a big deal the other thing I appreciate that question about encryption the other thing that I want to raise a serious point about though is authentication and integrity checking because even though we are fixing the encryption problem sort of we're getting to a standard we are still not doing anything about protecting networks from radio networks from bad devices so devices that aren't that don't belong there so most of the radio networks today if you know like an eight digit code that is really easy to pull off the air you can get a radio attached to that network and start receiving traffic from it it's not authenticated in any way and then there are also vulnerabilities to replay attacks in these networks so you don't have a good integrity mechanisms either so we we know that's a problem it's something we're working on the trouble is these are tens of million dollar networks that turn over once every 15 to 20 years and the next iteration may just not happen because they're actually building an LTE network right now for for first responders that's going to be nationwide so there's a big debate about well do we even like keep working on these standards because at some point we're just going to use that because it'll be better not necessarily soon but someday other questions so the question was when you make a 911 call location stuff turns on does the data that's developed as a result of that gets stored on the phone in a way that it could be recovered by forensic analysis the cops down the line or something I don't know a really good answer to that question the challenge is there are so many parties involved because there are legal obligations and interests that the carriers have there are very different legal obligations and interests that the handset manufacturer has and even the chipset manufacturer the folks who are putting those measurement devices how those interface with the 911 aspect of the device can vary so I don't know the answer to that unfortunately I my hope would be that once the 911 call goes away that that location data goes away as well in part because in in the 911 business one of our big concerns is always well can we adequately protect the safety of someone who calls 911 but doesn't want a third party to know that they have one tactic that we see particularly like in an abused spouse situation all the time is somebody calling 911 and pretending to order pizza and when they do that most clever telecommunicators and dispatchers will say you're telling me that you need to order a pizza because you wanted to call 911 and you don't want someone else to know is that right yes pepperoni which is you know so and that's the kind of thing where we don't want somebody to be able to look at the device and say oh you called 911 from the house where you know you're not supposed to call anybody like that that would be a potential information disclosure that would not be good there's a question right there yeah could you come up to the mic so that we can get this on the recording yeah okay okay the question was can the location information that is collected be used to prevent things like for example swatting I promise I did not pay him to answer that to ask that question but that is like the best possible question you could ask it is something that we have been looking at for a very long time the answer is yes absolutely it should be used that way every time it it I go through no end of amazement that people don't don't always check mismatches like that when there is a call I do think it one of the benefits of frankly having humans in the loop for 911 calls is that rather cleverly a lot of the time they do say that and you know they'll they'll say to somebody look before you go full swat on Brian Krebs again maybe let's make sure that this call is actually coming from there and the other thing that we've been able to train on is when you get a call from a non-service initialized device so a cell phone that doesn't have service which is required to still be able to make a 911 call treat that as suspect until you have really really good information linking it to to that person's thing now part of the challenge is if you're too far away from the target of like a swatting attack in the first place your calls not going to go to the right 911 center anyway and then you're going to raise a whole host of other questions like well if you're in but if your call ended up in Boca Raton but you're calling from Denver then there's something clearly something has happened that we we don't expect and that we ought to be asking more questions about so great great question really appreciate that and in NG 911 one of the things we're going to do is a lot more data analytics capabilities to make sure we're authenticating things like that in fact as I talked about like the device based hybrid in the national emergency address database and all that when we were negotiating those new rules with the wireless carriers in the FCC that was one of our key concerns was if they get a Wi-Fi hotspot for example out of the database and it says this hotspot is here let's say Caesars Palace in Las Vegas but the call came in from a cell tower that was in Knoxville Tennessee okay you know Houston we have a problem don't use that location data fall back use the tower use the GPS whatever else you've got so location data that comes from the at the database process has to be corroborated going forward that that was a big deal for us for like from an integrity standpoint to make sure that we we had corroboration yep in the back the question was is there a 911 app let me be really clear no there is not a 911 app if I could get Apple and Google to do one thing it would be to just grep the entire app store the entire play store and delete anything that involves the words 911 they don't work they're totally bogus there's all kinds of things and I I hate to be this like passionate and upset about it because I love that there are developers out there trying to do helpful things for people in an emergency but the thing that like none of the developers with maybe one exception who have ever attacked this problem have understood is just how much legacy cruft there is that stands between anything you can do with an app and the 911 system it just doesn't work the only there's been one startup that sort of kind of halfway cracked the problem and they literally had to go and build millions of dollars of infrastructure they built their own NG 911 system to just do an in run around the whole thing and then ultimately find some carrier partner to trunk the calls out to peace apps it no there is no app that works with 911 and there won't be there there are very good reasons that you should not do that again I appreciate people's passion but like it just it doesn't work don't as unfortunately there are dozens of apps on both of both of the major platforms that purport to do something like connect you directly to the nearest police officer well okay a the app provider doesn't know where the nearest police officer is and be the only way to connect to that nearest police officer is over a radio platform that I guarantee you that app developer knows nothing about and more over that police officer does not want your 911 call like that's not the process you know they've got other stuff going on you know for the third of 911 calls that are like cat up a tree or mcdonald's got my order wrong or or you know what at whatever the thing was that went wrong the nearest cop the nearest firefighter doesn't want that we have these things centralized for a very good reason the people that take these calls are important part of the process and you can't circumvent that so sorry to be passionate about it but yeah no no app and won't be different clearly you missed my talk yesterday so that actually happened it was so the question was if we have all this legacy stuff that's very susceptible and you know minimum traffic carrying capacity what if somebody has some malware that gets on a major platform and then starts calling 911 repeatedly what I will do is point you to the DEF CON media server because once once that talk goes up you can get 45 minutes of a great answer to that question yeah in the back yep the question is what about the integration with SS7 the regrettable reality is SS7 is going to be with us for some period of time for most 911 systems SS7 would be an upgrade because they're still using MF and KAMA yes yes you can hey that is exactly the look you should have when I say they're still using MF and KAMA but they are that's it that's how they're signaling things they're not using SS7 the ultimate goal is to bypass that completely we'll still have to have SS7 interfaces for carrier networks that hang around because signaling system 7 is going to be with us for some period of time but next generation 911 is designed to start getting rid of that get it out of the system as quickly as we can I think we've got time for maybe one or two more questions okay I'll be hanging around in the hallway then thank you all very much for coming to the talk and great great getting to chat you take care


All right, first up, I've got some more keyboard cases. We have the 60% keyboard shells, but let's say you like arrow keys, you want a bigger shell, you know, like these are called the K68s or JKDK68s. I don't have CAD files, so I'm going to try to get them, but you can just measure the mounting holes and use these for designing a PCB foot keyboard. I always like to have the enclosure first before I design stuff, so that's why we're carrying a lot of keyboard shells before we make any keyboard PCBs. I just think it's better to get all the cases nailed down, because that's something that's very hard to do after the fact. It's not easier to adapt a PCB for a case than the other way around. Okay, next up, this is fun. This is like the best alarm clock for an engineer. Yeah, you sent this to me. I think this was on the social media. I just thought this was hilarious. So it's a, and it's actually, it's one of those things that's like, haha, so funny, but then you use it, you're like, actually, this is kind of good. It's a multimeter that's also a Bluetooth speaker and also an alarm clock and also has built-in batteries that can be recharged, so it's portable. And it's a good Benchtop multimeter. Like it's actually kind of nice. I think, you know, I wouldn't replace my, you know, handheld multimeter, but I think if you're going to do, if you're going to have something that's a Benchtop speaker slash alarm, you might as well toss a multimeter in and it works great, comes with probes. And it's not too expensive. It kind of does everything you want. And I like the nice big display. So yeah, it does it all. It's a, you know, dessert topping and a floor wax and a multimeter and an alarm and a Bluetooth speaker and does, you know, continuity and current and frequency and all that good stuff. So great for dorm rooms. I, yeah, I think actually this would be really useful for, to be honest, would be really useful for is a, is a maker space or workshop. Yeah. Because it's portable. I do like that you don't have to have it plugged in. You can charge over USB, but then you can pick it up and move it anywhere. And that I think does make it useful.   What am I gonna do? What am I gonna do? Yeah. Just grab your alarm clock. All right. Next up. We've got two sets of YC8s. These are aeronautical style quick connects. You know, sometimes called limo connectors, which we covered. Those are usually quite expensive in their military specification. And you're actually using aeronautics. This is aeronautic style. I would not use this in an actual aeronautics project. They're not specified for that. Don't use this in your effort. However, if you are making projects that want that kind of connector, a quick release connector, that's very reliable and durable. These are really good quality connectors. Usually this stuff is out of the financial range for most projects, which is why you don't see them because usually the connectors are $50 apiece. These are like under 70 you know, they're $70 apiece. So they're a much more affordable version. I've seen these used a lot in keyboard DIY quick connect USB cables, which you can use them for. And you know, maybe we'll do a project on that, but they're good for other uses as well. You know, when we were doing wearables, we use these for quick connects to connect different elements of the warble system because you want to be able to quickly remove them, but you want them to stay durable. These are extremely durable if you use the right diameter cable, which is the thing about them. You really need to use the exact right diameter cable because the inside of the connectors has like a little metal spring piece.  So that grips onto the outer cover of your cable and it really needs to be like within a millimeter. So you know, there's some people who are like, well, you can use thinner cable and then put heat shrink on it. I just whatever you do, you really do. You know, this is the thing that's hardest to use about it. The soldering isn't too tough. The assembly isn't too tough. But having the right diameter cable is is what makes it challenging. So I'd say just for that. That's the one thing to watch out for. Otherwise, the strain relief won't work. You can use a thinner cable and then maybe stuff some hot glue in there. That's an alternative. I think it will work just fine. But if you want to use it the way it was meant to you, there is that there is this press fit part that needs to be used. Next step, we have a all in one sort of like Tablity HMI kit from Espressif. I do like the ESP32 S2 and I do like that they packed this full of hardware. I will say it doesn't have circuit python support. I don't think it has Arduino support. You're supposed to use this with the Espressif IDF. And even then it's probably not going to have full support for a little bit. They do take some time to get support out. So I think that's one of the things that's really important about ESP32 is that it doesn't have circuit python support. I think it has Arduino support. You're supposed to use this with the Espressif IDF. And even then it's probably not going to have some time to get support out. When you plug it in, it doesn't do anything. So it doesn't come with a demo even. But it does have a beautiful 4 inch capacitive touch screen with a 16 bit parallel port connection. There's all these at the bottom looks like JSTSH connectors for like Semicut, but also SPI and UART. It's got the ESP32 S2 ROVER. So it's got Wi-Fi and native USB. It's got another USB port for the debug console and programming. So it's kind of good for that. It's got capacitive touch. A couple of sensors. I think there's an infrared thing. There's battery management. There's this particular one. They don't ship it with a battery, but you can grab a iPhone 5 battery off the shelf and apparently just plugs in. They, to make shipping easy. It doesn't ship with a battery. MicroSD card slot. So it's one of those things where I got this, not because we support it, but because I think this is a good hardware setup that I would like to support one day in CircuitPython because it kind of covers everything. So maybe I'll go to the overhead real fast and I'll just point out because there is there's some depth here. So it comes with this like 3D printed or I don't know injection molded. I don't know exactly case. But this is kind of a divot here. So this is where that iPhone 5 battery would go all the different pieces. This case has slots here for all the connectors. Two USB C's. There's a speaker that's built into this kind of piece here. This is nice gold copper. You know, it's what all the pin connections are etched into copper. You can go here to download the software, but it's kind of got a little bit of everything which I thought was kind of neat. And particularly having something that has the capacitive touch screen clearly it's like from a phone or something that is being recycled. In a nice beveled case with light sensor and humidity sensor and SD card and everything. So a good collection of everything you want and it features our favorite new chip, the ESP32 S2. Alright, and start of the show tonight as you lead our community, our customers, our team is the IS31 FL3741 RGB Matrix driver. We've been working on this breakout for a bit and I was a little scared of putting this together because there's so many RGB LEDs. 117 of them. In fact, it's a 9 by 13 grid of RGB LEDs matrix through I2C using this cool chip, the IS31 FL3741 as mentioned. It can drive like 351 LEDs or 117 RGB LEDs. It's designed so you can tile them side by side and of course each LED, it's not neopixels or dot stars or just analog LEDs that are PWM that 8 bit per channel. So you get 24 bit color basically. And iSquirtC makes it really easy to use. You can use it with Python or CircuitPython or Arduino. If you want a lot of LEDs we recommend going up to our RGB 75 matrices. You know those of course you'll get 32 by 64 16 by 32 like tons and tons of LEDs but you do need a lot of pins whereas what's nice about this is it's all over iSquirtC. So it's really easy and it's low power and it looks really good and honestly it shows on the overhead but this kind of kind of looks better although you can see it but nice and bright. This looks good too. I'll say this, I wish that anyone who does electronics would show videos of things working or do even just do like a quick phone video because you never get to see this stuff actually working. Yeah, there's actually a surprise that the color came through. It looks right, yeah. But yeah, it's very easy to use and we designed it so you could tile them side by side. An earlier version of the prototype didn't have this bottom edge but we thought people would rather have mounting holes because again if you want a lot of LEDs you should just get our hub 75 matrices, honestly they're going to do a way better job than this. So it's cute, we're going to make more of these. We're going to go through a lot of LEDs. I actually order like a couple hundred thousands of LEDs because it adds up fast. We don't have anything else that has this many RGB LEDs in them. So you know what I normally would think of as a lot like a reel of 6000. Nope, that was gone in about like two minutes. Thankfully our pick and place has a lot of heads. Yeah, ten years ago I said it'll take a little while but we'll get there. We'll eventually be able to make a jacket that plays Blade Runner the movie. So we're getting there. It is on that long list of things. We can help pixel Blade Runner. Like scripting languages for microcontrollers also wearable TVs. Okay. Anyways, more on that later. And that's new products. Yay! Little colorful.