Hello, Mike Test. Thank you. Let's get started. Apologies for delay. Welcome to the session and I'm Binu Ramakrishnan from NVIDIA and I'm going to talk about Building Trust at the Edge, Lessons Learned. The work I'm presenting is broad in nature and there are many people who contributed to this effort so I would like to take the opportunity to acknowledge their contribution and grateful to be here because of the teamwork. Here is the agenda and outline what we are going to cover in today's session. We'll start with a brief introduction to Edge Computing and we'll get into the third modeling, attack models and motivations and we introduce the building plots which is used to orchestrate security at the Edge and we end the session calling out some of the challenges which we encountered as part of the journey. How many of you are security practitioners here? Okay, all right. Edge Computing, what is Edge Computing? Edge Computing brings compute and storage closer to the sources of data and it enables applications that require ultra low latency, high scalability and high throughput. In our case, our application is provided AI inference at the Edge so it enables applications like AI inference to process data in real time, conserving network bandwidth and saving the response time. So if you look into the overall architecture, the one we are describing from Edge Computing perspective, we have a cloud layer which is used to manage or a fleet of Edge devices which are deployed at the Edge and the primary reason that Edge nodes are deployed closer to sensors is it will allow the Edge nodes to do real-time processing. Examples of Edge Computing includes retail, shopping malls, airport, hospitals. The Edge Computing combined with AI expected to transform industries from retail to healthcare, manufacturing to smart cities and many of the Edge systems are mission critical and industry largely underestimates the critical need to provide highest level of security for the applications running in this Edge. Before we get dive deep, let me try to categorize the Edge Computing models to broadly into two. One is the Edge Appliance and the other one is Edge Platform and what we are covering today is Edge Platform and Edge Appliance provides end-to-end experience. It is mostly purpose-built and it is capable of doing predefined set of functions and the data collected from the sensors are processed by typically trusted applications. Whereas in the case of Edge Platform, it is a generic platform that allow or support wide range of applications such as manufacturing, retail and healthcare. It is relatively powerful machines, for example, can run applications such as AI in France. The fundamental difference here is the data is processed by the customer deployed applications rather than the trusted application directly from the provider itself. You can think the appliance more like a vertical built-up versus Edge Platform is more like a horizontal built-up. Here is a reference Edge node stack. We have hardware, firmware and on top of that we have Linux Kubernetes and all the applications which are running as a port in this Kubernetes cluster. I understand that each Edge stack differ. You may not be using Kubernetes. You might be using free BSD instead of Linux. In our case, we support both x86 and ARM. However, this talk is more focused on x86. We are still working on the ARM side of things. But overall, many of the things what I'm going to discuss will be applicable to any Edge systems. Here is a variation here. Here you have a system that is BYOD. It's basically bringing your own device. You buy the device, you hook it into a system and use it. As you can see here, it creates unique security challenges. For instance, who is responsible for keeping the firmware up to date? In the case of x86, it's the UEFI lockdown and so on and so forth. So it creates some challenges from a provider perspective. I'll get into those soon. Here are some assumptions and scope. The assumption here is we have a cloud managed Edge platform service. Essentially, a cloud-wide service that manage fleets of Edge nodes which are deployed in the customer location on the remote sites. The scope here is the Edge node deployed in the customer premises. We are approaching this from a platform or service provider perspective. What is not in scope is we are not going to talk about cloud security. That could be something we can. That will be a topic for some other time. Likewise, we are not really talking about the security of the IoT sensors. Edge security, what level of security is desired? Where do we draw the fine line? It largely depends upon your business model and your risk appetite. Like I mentioned, you can get into an appliance mode where it's probably relatively easy to secure, but you will provide a fixed set of functions, whereas in the case of a platform, you are essentially providing a general platform which have more security consequences. The other factor include whether it's a purpose-built hardware or you are entertaining, bring your own kind of device, so all those things can impact the security threat landscape. To secure Edge system, we first need to understand the threat landscape. Who are the threat actors and what are their motivation? How do they attack and what is the impact? The threat actors include nation-state actors. Those are the group either funded or supported by government agencies around the world. Organized group are, they provide, in short, cyber crime as a service or command and control as a service. The other group are very minor but still discontent of employees and activists. Activists, essentially they champion for certain costs and indulge in these kind of activities sometimes. Their motivations are steel data and intellectual property. For nation-state, stealing is primarily to achieve some security, strategic objectives versus organized group primarily do that to make some money. ransomware, they lock you out and demand ransom. Steel computer resources, once the machine is compromised, let's say, sometimes these computers are powerful enough and they will make it part of their command and control. They often use for crypto mining and things like that. Denial of service, that is another aspect here. For example, activists can really disrupt it. They don't really gain out of it but it can really create chaos in the system. Attack models, how do they attack? There are physical access because this is the edge system and since it is connected, over the network attacks are also possible. Vulnerabilities. Vulnerabilities are everywhere. Any non-trivial piece of software have defects and some of them have security consequences. So, attackers always look for opportunity to exploit these vulnerabilities in the system. Hardware and software supply chain, the attackers trying to do some backdoor either in hardware or software and trying to gain access to these edge systems. What is the impact? Depending upon the industry, for example, if it is a healthcare, there could be risk to life and disclosure PII and other sensitive or confidential data. Financial loss, litigation, losing market to a competitor, impacting a brand and it can disrupt services causing chaos in your system. Imagine Friday afternoon in LACT, all the traffic lights go off at the same time and you can imagine what can happen. Things like that. Here are some additional considerations here. One is the physical accessibility. This will lead to unauthorized physical access devices. The other scenarios include it is susceptible to side-channel attacks such as timing attacks, power consumption analysis, cold boot attacks, things like that. And since they have the physical proximity, it can also mount firmware. It can basically modify firmware and exploit any firmware whenever it is. The other factor is heterogeneous hardware. It is mostly the bring-your-own-device kind of scenario where there are so many vendors here and that significantly increases the attacks of us. Likewise, there is no real standardization in the pre-boot environment, especially in the firmware world. Different vendors support different kind of tools that is also something to consider. The third one again is related to the bring-your-own-device shared responsibility of the edge nodes. As a provider, we need to trust the customer and the customer, you are expecting customers to lock their UFI, lock their firmware, pass the system in time. That is another thing to consider. To build security, we need to identify the security building blocks. A security building block is a unit of technology, process, or a combination of both. In this side, I am introducing the building blocks, which is used to orchestrate security. I just call it out. One is the hardware root of trust. The trusted computing base, firmware security is very important here. Identity, authentication, authorization, secure edge, enrollment, encryption and data protection, remote attestation, runtime and boot time integrity, threat detection and failure reporting, supply chain security, vulnerability management and over-the-air update and defense in depth. We will cover this topic in next few minutes. Root of trust and trusted computing base. The diagram in the right shows how the chain of trust is established from boot to the runtime. And there are two complementary technologies here. One is the UFI secure boot and the other one is the TPM or trusted platform module and the measured boot. Here you can see the secure boot use the UFI as the root of trust and measured would use the TPM as the root of trust. What is secure boot? Secure boot ensure that the node boot only using the software trusted by the provider. And if any of the software that is part of the boot chain is compromised then the signature match will fail and the system will not boot. So it is more of a hard fail. Whereas in the case of measured boot, we have in the TPM we have something called platform configuration registers which basically is used to store a cryptography measurement or a hash of each of the software layer in the in the boot chain. Our integrity measurement architecture extend the measured boot to enforce file runtime file integrity. The TCB it's not from a strict sense but mainly from a broad sense. This is a set of this is a stack we need to defend against. We need to make sure that the integrity of the stack is protected. TPM is a separate topic but this is a brief 30 second intro to TPM in case if you are not familiar with the TPM. A TPM is a co-processor on a motherboard which is capable of doing or generating secure random numbers. It can generate keys and hashes and you can also use to store cryptographic keys. And it can also do cryptographic operations such as RSA and AES operations. And one important thing at least more relevant for us is it provides platform configuration registers essentially a set of registers around 25 or so 24 I think that is used basically to store some hashes which measured boot use. It is used for performing attestation. And one thing to note here is this device is passive and it is a very it is a pretty slow device actually. Secure boot and measured boot I already covered so I would skip here but the main thing here is in case of secure boot is more like a hard failure if the signature of any of the component in the boot chain fails then system will fail to boot whereas in the case of measured boot it will compute the hash and update or append the hash value in the TPM PCR registers and it is passive in nature but it requires an external service to attest these measurements. Edge node enrollment. You have a device now how do you securely introduce this device to the service provider. Imagine you as a customer you bought a device. What is the process there? There are multiple ways to do that. I'm just describing one way and there are a lot of variations possible in simple terms the way it works is you have an account with the service provider for example the same way you have an account with AWS GCP. So you have that account now you when you're ready to register a device what you have to do is you go to the the web console whatever the provider provide and say I want to register it so one method is they provide some kind of a token one time short short leave kind of token say yeah here's the thing you just go you take our software and deploy it in your edge node and you also pass this one-time token and internally it install the software as part of the initialize the TPM generate the TPM keys and it register the TPM key with the edge through bypassing both OTP and the keys associated with the TPM. In the end what happened is the edge system knows that okay oh this is the edge node I know that and this is how I can authenticate with that edge system and the TPM protects cryptography keys and operation and prevent attackers stealing keys and and key cloning essentially. When we talk about edge node it is not just one application there will be multiple applications running in edge node likewise if you talk about the cloud there are multiple applications running there as well so how do they communicate with each other and we need to have a strong authentication in place to communicate with these systems and how do we do that? Let's talk about node identity so you'll be one you'll be thinking yeah we have a TPM based identity so we can use that to authenticate with cloud services. Yes and no it works in some case but it is cannot be used there are some issues here it is easy for the edge agent to connect to the edge node manager because they already have the TPM's public key but that is not the case with many other services in the cloud where the edge wanted to communicate that means you cannot really communicate beyond or to any other service other than the service that has the TPM's public key. Imagine you are a provider which is managing hundreds of nodes or thousands of nodes managing these keys is also a little difficult. I mean we need to spread across or share these keys with many services so to remedy the issue we introduced a cloud authentication service or token service what it does is it actually do the authentication with the TPM and it issues a OIDC compliant JWT and any service and that is a short load of ephemeral and any service that is compliant with OIDC it can make it work. Again you don't have to use JWT you can also use ephemeral certificates and things like that. Here's one example here you can see the edge agent request TPM to sign you can see that edge agent is making a request to the TPM to sign sign the node token and it make a request to the OIDC service token service and the token service authenticate and oh this is coming from I know this edge and I know it is a legitimate request and it will issue a short-lived OIDC token and all the subsequent connection from say edge agent to edge management service they can use this OIDC and through OIDC discovery this service can authenticate the token and allow this edge node to access the services alternatively step one and two can be replaced with mutual TLS with the TPM backed key using pgcs 11 interface. Let's talk about remote attestation it is a complicated concept but let me try to explain the array is a mechanism to you array is a mechanism used by the service provider to verify that a system an edge system connecting to itself is is booted with trusted software. Let me discuss about one scenario here here the edge imagine the system the edge system booted up and the first thing they do is you try to connect to the service providers the cloud service essentially and the cloud service say yeah I know you are edge Bob I know you through the TPM key but I do not know whether you are booted with the trusted software let's do this I give a nouns you just you just go and get a PCR code so the edge system take the nouns and contact the local TPM and say edge generate a PCR code for me and what is a PCR code PCR code is nothing but a PCR measurement signed by the TPM attestation key and and and the PCR code and an event log sent back to the edge manager platform configuration registers yeah it's a TPM construct it's basically provide set of registers where you can only extend the value you cannot really modify it I I deliberately not going into those details because those all are kind of another topic so but I hope you understand the larger context so where did I stop so so the PCR the value upon receiving the PCR value we just compare with a reference value and see whether that matches if there is a mismatch you saying something okay if there's a mismatch then what happened is it trying to pass the event log and see where the mismatch occurred this diagram is not really good but please bear with me so how do we get the reference state that's important thing you can you can get it by different means one way here is remember I showed you the note enrollment process where the system is actually booted first time so as part of the process it's basically get the measurement at that stage and consider that as a reference quote it is more like a trust on first use kind of model we following but but there are other variations possible if you know your hardware well then you can also know what will be the value at that stage as a as a runtime extension to attest so we can we can actually extend it through IMA it'll support the runtime integrity check IMA what it provides is the file integrity check like you can basically hash the file and any of the files modifies and then we can enforce or at least we can detect that okay this file has changed in runtime can you be loud we use tls the the pcr values are signed by tpm at private key so we cannot manipulate that okay let's let's we have a q and a so we can discuss and if I can also sync up with you offline yeah so data security data protection is critical and whatever we data we are storing in the edge it has to be encrypted this will protect when the drive is stolen repurposed or reused data in transit we need to make sure that whatever data transmitted over network is protected tls is a recommended protofolder tls is very critical not just for securing the data but it is it is critical to protect against man the middle attacks it's very important secure deletion of data if the drive or nvme is retired or repurposed then we need to make sure that the data is cleaned up network punching holes into customer firewall to allow inbound connections to edge services is easier said than done the the problem here is the customer network they have their own it policies and and some organizations are pretty big and they have separate organization to manage those things and if you want to open let's say some firewall acl it may take ages to do that or it may not even happen at all the recommendation here in this kind of scenario is since generally firewall generally allow outbound connections particularly like ports like 443 so always initiate connection from the edge device that way you can really reduce the dependencies and it's also good practice not to run any services or listeners listeners specifically network-based listeners on the edge nodes just reduce the attacks of the area runtime integrity and threat detection for edge platform threat detection is an important function and such systems are required to detect and alert anomalies that occur in the system during runtime here are a few rule set examples reversal we can have a rule set basically to identify a reversal and reverse dose reverse dose is someone getting to the edge node and trying to mount a denial of service in the local network the another thing is you have a part which is a imagine it's a web server it's only if you see you can have a rule set to detect any kind of bash exit so that's that's i'm just giving some examples if you know the certain patterns you can update your rule set to detect certain CVEs for example like lock 4j lock shell kind of vulnerabilities again detect brute force SSH crypto mining we can we can track the outbound request and see where it is going and depending upon the IPs we can determine if it is a crypto mining happening or not then we can also track based on stratum protocols usage which crypto miners generally use we can also tag certain files and see if anyone touched that and we can get notification that will help to prevent data exfiltration supply chain software developed in recent times heavily rely on open source components besides the security of the software we built open source security of the open source also have direct impact on on the edge the fact is that any non-trivial software contain vulnerabilities and some of them has security consequences we have both hardware and software security risk when you talk about hardware we have what is that uh introduction and seeding introduction is essentially you're modifying uh in transit from manufacturing to the end user or the consumer whereas seeding is you're putting some backdoor as part of the manufacturing process itself and some the software side of things there are many ways to compromise third party packages or third party or open source libraries like compromising GitHub accounts updating making changes things like that so what are things we can do here you have to source these packages uh get to be careful and if possible yeah if there is an option not to use then don't use it every is now open source is not free it bring a lot of liabilities to your to your stack and you scan as a perspective you scan for vulnerabilities looking for CVEs and malware etc secure distribution of whatever artifacts we build that's important uh over TLS recommended published checksums um there are some new advancements here on image signing uh six store i don't know you're familiar with that but they offer uh tools uh for signing and verifying software uh they it is good uh for container signing at this moment but uh but there are a lot of other artifacts as well you need to consider build supply chain provenance or some kind of attestation of the build process uh adjusting the s-bomb and things like that uh in toto and salsa those are the two options in this space uh again those things are in these early early stages uh vulnerability management and over the air updates uh vulnerability management is a continuous function to identify and patch uh vulnerable packages in a timely manner and OTA or over the air update is a vehicle to uh to update or patch these edge systems and OTA happens uh in this uh following condition not necessarily a full list of things but in general like uh remote attestation failures whenever we do remote attestation remember like there is a pcr quote comparison if any such failure happens the uh it you get alerted and they may determine a there is some bad happen we need to remaze the system sometime we require uh physical presence of sometimes you can do over the air uh likewise we identify some vulnerability in some of the packages as part of our stack then we will do a OTA uh regular upgrades generally go through the OTA and other uh recensing routes some kind of security attacks or something we identified as part of our threat detection system sometime we use OTA uh to uh update note that the OTA may change the uh some of the pcr measurements so uh we need to make sure that we are updating the reference state accordingly in a uh in a reliable manner it's a little challenging but it's possible so all the things which I discussed the the building blocks those are the big big things but that is not the end but there are a lot of small but important steps we need to do to make the system secure uh I'm listing out a few but there are many more um from authentication side we generally use uh recommend uh use of ephemeral tokens and certificates um we should have a our back for customers uh remote console for device administrators if you want to manage the edge device from the cloud isolation uh we have a host and application level isolation um we can enforce uh that containers can run in an unprolaged mode um and between applications we separate using Kubernetes namespaces operating system we we have to harden the operating system it's very important we use a syscat as a reference uh in addition to that we also do other things uh for example we strip down the OIS to make it as small as possible uh that way it'll reduce attack surface and also reduce the the need to pass the system because some of the unused packages may have vulnerabilities and we end up patching the system we can avoid that uh read only file system for system partitions uh and encrypted uh data partitions uh some examples here and hardening of the Kubernetes cluster uh enabling our our back policies uh and provide admission controllers to enforce certain policies the policies include uh you cannot you can run only as a particular unprolaged user you can only open certain ports only certain inbound outbound are allowed uh you can also apply up on our policies sitcom things like that um uh you can also use uh the the uh the six store which I shared previously uh it also provides some uh can also use admission controller to basically uh verify the docker uh container images signed container images um we have dedicated namespace for customer workloads so things you can follow uh so if you are listening this probably be the most uh uh interesting uh slide I would say uh this is uh basically uh show uh all the things uh which we discussed and uh you just show how all those things are mapped and connected uh if I describe it we have we just convert it into three phases here one is post deployment uh what all steps we uh we have to do and then another is during the deployment phase then the third one is the pre-deployment a post deployment where some of the runtime controls kicks in uh in the in the pre-deployment you're talking about third party software and what all things we do there we talk about uh applications native applications which we build what all things we do there talk about Kubernetes what all things we do there and operating system very important component what all things we do there and there are many things uh some nuances uh I excluded that is for clarity but uh so all these software are built and put into some artifact factory and we can potentially provide uh some integrity like uh you can you can publish uh check some hash uh you can sign those artifacts those are some other things you can do once uh the system uh and and and you can see here the user basically pulled uh the software from the from the providers artifact repository and uh you basically install that in the edge and at that stage you will do a pre-boot protection like ufa lockdown and secure node enrollment those are some of the activities part of the deployment and when you talk about uh the the runtime we have the runtime protection we have runtime detection runtime detection include uh tpm attestation uh falco ebpf logs uh then secure boot and runtime prevention include immutable file systems uh k uh Kubernetes namespace isolation admission controllers uh and the trust we depend upon uh the tpm and uh and the ufi uh tpm backed identity uh tls ca bundle uh yep uh so i hope this will be helpful for you to visualize all the things which we discussed challenges uh building trust on disparate uh disparate pieces of hardware and software provided by variety of vendors is a uphill task uh here are some of my observation on issues uh came across uh spot of our journey uh secure boot uh it is cool but remember it is hard failure if something bad uh happened you just stop uh booting up uh you that is that may not be the thing you want because you have a cloud service which is monitoring or supporting ed device and if the ed device is not even coming up you you don't even get an opportunity to see what went wrong from that perspective it will be better to have measured boot than secure boot both are better but secure boot other other problems include uh we have you have custom kernel modules or unsigned you have to use some unsigned third party modules so how do you manage that uh you need to update uh the ufi database that require physical presence uh so those things uh can create more trouble uh and uh stability of the tpm pcr bank uh we come across uh the values are not consistent between uh hardware providers like sometimes we're just changing uh even the for example the bios pcr bank zero uh not supposed to change but sometimes we just reboot just change uh those things like that uh and lack of standardization in firmware uh reboot environment is another thing we have to deal with there are some uh redfish api all those things are coming up but uh i think it has to evolve i think uh supply chain attestation is it in early stages and i believe it's not ready for a wide adoption yet but that is something to watch out uh artifact signing package check some work it's easy simple um container signing is doable now uh but signing and verification of other artifacts uh currently it's not trivial uh so with that i is a teamwork and uh that's it yeah it's arm based it is arm based no no no the arm uh we are currently working on uh um a solution to work on jet sun we are we are considering a uh firmware tpm uh but still early you know maybe let's talk offline yeah yeah uh yeah sure i need to talk about uh yeah so if you are using secure boot you have to rely on ufi to trust but if you are not using secure boot you don't have to rely so secure boot it is a hard fail uh hard failure right you you have a boot chain now you are starting your application and imagine one of the component in your boot chain is compromised or uh something happened and the signature uh there's a signature mismatch at that time the system won't boot yeah yeah yeah let me conclude this uh yeah yeah i'm just yeah yeah yeah yeah yeah uh i i didn't mean to make it that way but it ended up that way it ended up that way yeah when i started chatting out but i still if you pay attention it's uh it's not very complicated uh yeah when you when you see first time maybe you get overwhelmed but if you just look through that probably it'll be not very hard uh yeah which which one this one uh you're talking about this this one or something else uh what i'm uh no what what we are presenting here is uh some other experience uh on building an edge system an edge platform and some of the patterns we can use across oh you're talking about the use case okay i think i described as part of the edge computing introduction uh one use case here is imagine uh one simple use case i can tell you like the air inference uh you have an edge system that can process live traffic feeds looking for traffic violations that's one example and you can use this edge system in the factory floor as a robot and it can do some AI inferencing the robot uh that's another use case and there are so many use case related to edge AI inferencing these are like two just uh two examples uh the customer at least in our case get a uh an approved kind of device from uh oems which uh which provide is approved device and they bring it in and they plug it in yeah uh yeah this is more of a generic platform that means we can fit into different kind of application different industries and uh we have verticals built on top of that for various verticals so i just only covered the the platform uh yeah we are on time uh let me uh any more questions before i can wrap uh okay uh yeah go ahead oh i didn't i i deliberately excluded that uh because that is another another uh bmc i i didn't really add it here but yeah if bmc is another thing we need to really uh protect that what we uh that's where the network side of things become very important because bmc security is primarily the network security if network is not secure then you are kind of compromised um all right i'll be around today and tomorrow so if you have any questions if you want to have a conversation feel free um thank you and have a good rest of the day cool so i'm gonna go ahead and get started in just a second here actually i think i'm gonna use the restroom since i had a ton of caffeine as i finished this presentation on the car right here so since i don't want to piss myself in front of you guys i'll be back in just a second all right so i'm gonna go ahead and get started uh my name's alex and today i'm gonna be telling you guys the story of how i accidentally made a hardware business at 18 so actually just last week i was at the hope conference in new york which is hackers on planet earth and i gave basically the same presentation with just about the same amount of preparation as well which was starting the presentation a day was supposed to give the talk a day before i was supposed to give the talk working on it all the way up until the last five minutes as things happened for me so my name is alex lind i'm an open source hardware developer and also a cyber security content creator so i'm usually um making education focused content on the hack five youtube show if you guys aren't familiar with hack five there are cyber security vendor that creates products for pen testing red teaming and that sort of thing i mostly create beginner focused content where i create walkthroughs tutorials that kind of thing but more recently i've been focusing on hardware and physical attack exploits and i'm going to be showing you guys one of the tools that i developed today that i've been showing off in some of my more recent demonstrations so when i'm not creating content um well i also create more maker oriented content on my personal blog where i have walkthroughs for different hardware stuff that i make i create various projects like the one i'm going to show you guys today but my primary interest is in low-cost microcontrollers in ways that you can use them um for low-cost hacking like wi-fi hacking signals intelligence and that sort of thing so just an overview of what this talk is going to be about i'm going to be telling you guys my back story sorry i'm a little out of breath so i'm going to be giving you guys my back story i'm telling you how i got to the point where i am today i'm going to be telling you a little bit about the business that i started completely by accident i'm going to talk to you guys about the ideation process of the product that we're now selling which is called the nugget it's a cat-shaped circuit board and i'm going to tell you guys about some of the mistakes and key things that i learned in the development process of this product so hopefully if any of you guys are product developers or makers um you won't make the same mistakes that i made or that we made um along this process so what is the nugget the nugget is the project that we've been working on over the course of the past year it's a cat-shaped circuit board that's designed to teach beginners about cybersecurity and hacking so we currently have two splits of this product one of them is focused on physical attack vectors like usb hacking keystroke injection and that kind of thing and it has a little gamified interface that walks you through the process of running some of these attacks so it's really great and beginner friendly and we also have another version of the nugget designed for wi-fi hacking so this one's more oriented towards things like network defense running wi-fi attacks and it's a great little walkthrough for beginners who are just getting started in these areas in addition to that it also comes in a pretty easy to put together kit that you can solder together yourself so this has been a centerpiece of some different soldering slash hacking workshops that i've been doing over the course of the past year and it also makes for a great beginner programming tool and it supports some languages like python and arduino and stuff like that so i'm basically just going to be telling you guys about how this product was developed and also some areas where we screwed up so first i'm going to start with a little background about myself so i first started getting into hacking and programming i would say sometime around five years ago in middle school when we were issued ti-84 calculators in math class and if there was anything i wanted to be doing in math class it sure as hell wasn't math so i taught myself how to program on this calculator actually with a built-in interpreted language called basic so i taught myself how to create like bitmaps and like games and stuff like that and i started selling it to other kids at school but once i got kind of bored of that um i looked into some more hardware stuff like how i could actually hack the calculator like overclock it and add wi-fi so i could create like a mini ioc channel to communicate with some of my friends in class and some crazy stuff like that but eventually when i got bored of that um i would say sometime around the summer between middle school and high school um i learned about the raspberry pi which was a low-cost micro controller or micro computer for $35 and that really piqued my interest because growing up i was kind of poor i didn't have a computer at this point but seeing that there was something so accessible and so cheap that i could learn hacking on um this is really interesting to me so i picked up a raspberry pi i started learning about linux and sure enough i discovered that there were all sorts of interesting things there were all these different distributions that you could use for specific applications like kali linux for hacking and all this cool stuff still catching my breath so i eventually taught myself how um to do stuff on the raspberry pi and one of the very first things that i taught myself how to do was actually hack wi-fi so as i said i grew up kind of poor we didn't have any internet growing up so when i had to do schoolwork i would basically have to skate over to starbucks use their wi-fi and then go back home um and then just sit in silence so i eventually taught myself how to hack wi-fi so i could have that um from home and be able to do my schoolwork and that kind of thing um and i eventually started teaching myself um how to do some other stuff like programming and that kind of thing but i found it kind of frustrating that there was just a lack of general documentation on the internet for some of the stuff that i was trying to teach myself so eventually i created a youtube channel where i started documenting various tools that i was learning i started documenting walkthroughs of how to set up various linux tools that kind of thing for other beginners that were out there and here you can also see one of the first um raspberry pi based projects that i worked on called the audit pi but um getting my start in wi-fi hacking really peaked my interest just about that general subject um with signals intelligence and that kind of thing so i started trying to teach myself a little bit more about that and that's where we get to the very first signals intelligence project that i worked on so the first um yep okay so the first project um the first independent project that i worked on that um actually had some contribution to the project that i'm going to be showing off today was called the creep detector so this actually used the little handheld device that i made here called the audit pi which was focused around a raspberry pi in order to basically what i wanted to do with this was showcase a way that an everyday person might find signals intelligence or wi-fi hacking or that kind of thing useful so the project that i ended up coming up with was what if i could create a device that allows you to determine if maybe a creep or a pervert is stalking you so that we did this was we used a signals intelligence technique called war driving which is basically where you can drive or walk around and map the physical location of wi-fi devices around you we used that technique to determine if basically the presence of a wi-fi device like a cell phone or laptop or something like that was spotted at multiple unique coordinates along the route that you are traveling so this is one of the very first projects i came up with um on the raspberry pi it was a combination of um some different linux tools and also also python and the demonstration ended up working so that was really cool and rewarding and this is the project that i worked on with my friend kody who i'm now running um this business with today so while i found um working with the raspberry pi very interesting i wanted to get into things that were a little bit easier to get started with and also more low cost so this is where my interest for microcontrollers developed microcontrollers were really cool um they were really cheap these are things like um arduino nanos the expressive line of chips um at tiny's at megas that kind of thing i thought they were really cool and i saw some research out there on how these could be used for more um hacking oriented stuff besides just like developing so i thought this was really cool and i wanted to find ways to integrate this into the signals intelligence research that i was doing i didn't really know where to start there wasn't a lot of great beginner documentation on using um some of these microcontroller platforms for hacking so i didn't really get into that until um i would say somewhere around like my sophomore or junior year i got this crazy ass commission from this guy who wanted me to create a device for him for tracking down a cell phone that he lost in the forest so i had no experience with creating anything of that sort but i was poor i wanted money this sounded cool so i said great i'll take three hundred dollars and uh give me maybe a month or two and i'll have something for you so i started looking into how it would be feasible to do this and i built off my previous research with the creep detector project um to also use signals intelligence in order to track down his lost cell phone so the way that this worked is i built this platform around a microcontroller that we'll look at in a second called the esp8266 it's about you can find them for like one dollar eighty on ollie express maybe five dollars on amazon but basically there is a library that allows you to do promiscuous wi-fi sniffing and you can look for a particular type of wi-fi packet called a probe request so even if you turn off your phone not if it's completely shut down but if it's just like turned off and left somewhere your phone or other wi-fi device is still going to be emitting a type of packet called a probe request where it's basically looking for networks that it's seen before and constantly emitting this out into the air so i thought hmm what if i could program this microcontroller to look for that specific packet have it hone in on the signal strength of this device and then use an antenna to be able to track down the physical location well it turns out that crazy ass idea worked he was able to not only find his phone but he was able to find multiple cell phones that people lost in a local river so he lived near like a resort or something like that and he would go rafting quite often but it turns out um lots of people brought their phones there and like waterproof bags and they ended up like dropping them um also at this point i was still poor i didn't have a cell phone so i was like hey do you mind sending me one maybe so that's how i got my very first phone um from this random guy on the internet that i made this crazy device for esp8266 i have a picture of it in an upcoming slide if you want to take a look but yeah all this is relevant because this is how i got my background in signals intelligence and also microcontrollers but i was really interested more in the low cost aspect of being able to do these kinds of attacks so anyways i was doing all of this um sometimes towards the end of high school i was getting like occasional commissions and that kind of thing but i found myself at a point where um in my senior year i discovered that i was one of my only friends who didn't apply to college so at the very last minute i was like scrambling to put together these applications and then i realized i don't really want to do any of this i don't understand why i'm applying to college this isn't really what i'm looking to do right now i'm more focused on like um it just didn't really speak to me but also i didn't really have an option to you know like stay home or something like that because my home life sucked so i tried to start a personal brand called lind labs where which is my last name where i tried to advertise basically the services that i would commission which was creating custom hardware software web apps and stuff like that that wasn't enough to sustain me or be able to move out from my crappy home situation um but as it happened the friend that i worked on that previous project with the creep detector he was creating cyber security videos for a company called hack five which is where i now create videos for and he also ran a live stream on a different channel called security forward but he basically said hey we basically are in a struggle right now to produce content so if you want to come out to montana for just two weeks that's where they were they moved out there during the pandemic just because it was like kind of hectic here he said hey if you want to come out to montana for two weeks create a bunch of content for us maybe you can stack up a couple thousand dollars move back and figure out what the hell to do with your life so i was like oh great that sounds cool i'll make a little bit of money but i still don't know what i'm doing in terms of college or if i'm going to move out or what i think i lied to my parents and said that i got into uc santa cruise or something and that after i was going to come back that i would go there spoiler alert um those two weeks ended up becoming a whole year that i ended up staying there and then also um just another side thing my friend was previously working on a little project with a friend that didn't end up working out so well so he had a ton of extra um just like random hardware components on hand so he was like hey uh while you're here if you also just want to help me come up with a design so i can get rid of all this extra hardware and maybe sell it on the internet or something that'd be cool so we ended up starting out as a fun little project also became the high intensity startup that we're now running today and dedicating most of our waking active hours towards so it's quite grueling so this was the initial thing that we came up with so my friend had um he had a bunch of leftover screens he had some buttons and then he also had a wi-fi microcontroller that we're going to look at um in upcoming slides so what we ended up coming up with was this design which you can see is sort of akin to the probe hunter that i showed in a previous slide so i said hey why not just slap this all on a board and then we'll sell it as maybe a soldering kit we can teach classes or something like that put it on the internet i don't know it was just kind of a random idea we just wanted to get rid of this hardware we didn't really have a focus for this it wasn't really a product that we wanted to bring to market so just as a joke we decided to slap some random graphics on there we decided to call it the hack cat um we put a little cat boy on the back we have a little cat emblem so that's where this all started out so now this brings me to my next point which is the first lesson that we learned pretty hard and this is why all in one hardware platform suck or more specifically why hardware multi tools suck so we didn't know this going into developing this product because we weren't even looking to design something that we wanted to bring to market we just wanted to basically dump all of our components into this design sell it all on the internet and then be done so the first reason um in a second i'll get more into the actual storyline of how this product was developed also let me check the time so the first reason why i think this sucks and this is also a pretty common mistake i think that a lot of makers and product developers end up making is initially it seems like a great idea to create like an all in one multi tool to try to appeal to basically everybody this is a tool that does everything who wouldn't want this i think this sucks for a variety of reasons but the first one is you're the only one who actually truly understands how to use this product um like as you can see here i think i found this design on tindi it's supposed to be like a wi-fi multi tool with like a screen and temperature sensor and like a bunch of other random crap but who knows what the hell this is it's kind of crazy so this brings me to the first case study which is my own personal project the probe hunter so let's analyze why this kind of sucked so initially it started out as a single purpose design for tracking down wi-fi signals that was really cool but once people on the internet after i started publishing this design um started gaining some interest in this i tried to make it do too many things i started slapping on an sd card reader so maybe if you wanted to save wi-fi packets to it you could do that a gps module so if you wanted to do war driving it could do that and then eventually it just became this general purpose wi-fi reconnaissance multi tool iot embedded project that does everything and no one was attracted to that it was a piece of crap nobody bought it that being said um while i do think multi tools suck it's important to know who you're developing the product for so if you are creating a design and it's for yourself as a product developer or a maker or something like that i do think there is some value to adding extensibility and modularity so for example one of the other products i worked on this was also a pretty crazy ass idea i seem to attract that sort of attention it seems but some guy who worked at a gas station wanted me to create a mesh network a cryptocurrency mesh network that detects when truckers pull up to the gas station and then automatically incur a charge through bitcoin wallet so that way they don't have to physically get out of the truck so it was a pretty crazy idea um but i was like hey if you're gonna give me money i'll do it he ended up sending me two thousand dollars i was like that sounds sick so um i wish i had a demonstration of what these actually did but i thought it was so crazy i never bothered to document it but these were some of the nodes that i ended up developing um and the underlying circuit board actually reuses my probe hunter design so in this case um a multi tool is kind of useful if it's just um behind the scenes and it's just for you as the developer so in my case since i knew what i wanted to do with it and since the customer just wanted a polished product it never had to see what was under the hood that was an okay application for it if i was trying to sell this as a general purpose multi tool not so much so this was the microcontroller i was talking about earlier which serendipitously turned out to be a great design choice so this is the d1 mini it's a particular form factor of the esp8266 so you can see up here um there's this little wi-fi chip on this board that's the esp8266 it's a really cheap low-cost wi-fi microcontroller that you can use to do things like develop embedded products that do wi-fi stuff but what's also really cool about this is it has some great applications for wi-fi security you can do things like wi-fi attacks such as the authentication so you can kick people off their wi-fi networks you can also use it for gathering reconnaissance to see when your i don't know your pervert neighbor is home by based off the presence of their cell phone but when i was developing the probe hunter platform i chose to go with this particular form factor since now that i really wanted extensibility in mind and i thought i was going to create a multi tool for everybody turns out they had a bunch of these different modules that you could just stack directly on top so that way if you wanted to add a temperature sensor or another sd card reader or maybe a battery module it was really easy to do that and i thought this was something that consumers really wanted so i ended up um designing around this platform and that's also the chip that we happen to be using in our current design today so that worked out one example um is this d1 mini war driving rig that i created so if you guys aren't familiar with war driving i think i explained it kind of briefly but basically what it is is it's a technique where you combine wi-fi and gps so you can drive around and map the physical location of um wi-fi devices and pin them down to a gps coordinate so this little creation here is basically just the d1 mini with the sd card module stacked on top um in order to log wi-fi data hooked up to a gps but in this particular case um i forgot where i was going to go with the slide but in this particular case we were able to use this for some cool demonstrations um such as war flying i ended up actually making a video on the hack 5 show if any of you guys are interested where we wanted to see if we could physically pinpoint the location of one of our friends in like a five mile radius by just flying a drone over for like 20 seconds around a huge park and we were able to actually use this to triangulate like their exact location but anyways on to the back onto the point of um why hardware multi-tools suck let's take a look at a case study from the company that i'm now sort of working for which is hack 5 so hack 5 creates very specific products that have well hack 5 creates um pentesting products with a very specific purpose and they generally have a single use case so i think this is great because um with a clear cut purpose it's kind of hard to get wrong it's kind of hard to get the idea wrong of what these products are supposed to do and although some of them might have alternate uses um that isn't the main selling point so it's kind of hard to confuse beginners i'll get more into that in a second but for example um these are some of the devices that hack 5 cells and each of them has a very clear cut purpose depending on what kind of engagement you're getting in for example if you want to hack devices over usb you'll probably go with a usb rubber ducky which is a keystroke injection tool if you wanted to do man in the middle attacks you have the plunder bug you can plug this into your phone to read packets plug into ethernet and you can sniff people's traffic um if you see an open ethernet port in the wall and you wonder what it's connected to you have the shark jack you can just plug it in it runs nmap or some stuff like that and then is able to give you that data back if you want to get into wi-fi hacking they also have a product for that called the wi-fi pineapple so it's all very clear cut and it's sort of a i would call it like a portfolio business model where each of these has a very um specific purpose and they're not trying to appeal to everyone so it's really easy as a beginner or even as an advanced user to determine which of these products you want to pick up and it's also not convoluted and crazy like my other ideas so this brings me to my second point which is beginners want to feel smart so how many of you guys have seen this product or heard of the flipper zero cool so actually for some of you guys that don't know what that is i'll explain it in a second but this is actually an example of a multi tool that i think fell a little flat unfortunately so the flipper zero is basically a it's a multi tool for sniffing a variety of different wireless protocols it can also do stuff like physical hacking so it has like these little breakout sensors that you can hook up to custom modules you can do sub gigahertz sniffing you can do stuff like open up garage doors you can maybe clone someone's badge that they tag in to work with you can open up tesla charging ports you can do all sorts of cool stuff it's meant to be a general purpose multi tool for all sorts of hacking um actually i'm gonna go more into this point but basically this thing was marketed as a multi tool for everyone and i think where this design actually fell a little flat was the fact well one thing that they did get right actually was that it's supposed to be a friendly intuitive design that's meant to attract beginners you can see it's a gamified interface that has like a cute little dolphin that guides you through like how to hack different things it has like a little d-pad that's in a gameboy-esque style um but they really um like ramped up the advertising on this one and when they actually delivered it to the audience that they were trying to sell to i feel like it fell a little flat in terms of the actual documentation for um beginner use cases some feedback that i've been getting um on this device i we do a weekly live stream where we just basically ask people about like um any hacking questions or what tools i think is cool or not one of the biggest pieces of feedback that we've been getting on this particular device from people that bought into this hype was that they pick up this product it looks cool and all but they don't really know how to use it there's not beginner documentation on the actual hacks that they can do with it and it's more of the advanced audience that actually knows how to use this tool which is really unfortunate um but this brings me to the point that you really have to know the target skill level that you're trying to appeal to if you're gonna create a device if you're trying to sell to beginners you don't want them to feel let down when they buy this product i feel like a lot of people bought this expecting that it would be easy to figure out like a really cool hacking project to do with project to do with but then they were kind of let down when they didn't really know how to get started and result only they put it on the shelves and felt kind of stupid of course it's not the case for everybody um some feedback that we got from some advanced users who have this product is that they felt like it was packing too many things in one product and they typically advanced users don't really want that because it's kind of limiting it's kind of limiting to have like an all in one um thing that isn't particularly good at everything if that makes sense so that was another key mistake that we made we didn't know any of this going into designing our product because as far as we were concerned we just wanted to get all this crap out the door sell these extra components and that was that so here you can see the first design that we worked on that was called the hack hat that's basically the board we just slapped all these components on to um but the thing was we didn't really have a big idea we didn't know who we were selling to we didn't know who the audience was we didn't know our target skill level or any of that but what we decided the best thing to do was was basically widen the pool and make this a thing for everybody this is a tool for everybody you want to learn how to solder you can do that it has a microcontroller that supports arduino and circuit python so you can use existing community stuff that's out there in order to interact with that board and that was kind of a lazy approach um just leaching off existing resources but I'll talk about more of that in a second so in addition to that in order to widen the pool that we were trying to get rid of all this hardware to we also came up with this crazy paperweight of an idea which was called the long cat so it was basically a longer form factor and it had a battery on the bottom so that way you could now carry around your portable paperweight but then as a joke I came up with the idea for the nugget so since we were sticking to a cat theme just because my friend really liked cats I said hey what if we came up with a conference badge kind of thing where it's actually shaped like a cat and um there's some existing community projects out there that we'll talk about in a second but one of them was called the wifi deauthor so this was actually the project that my friend was initially working on that ended up flopping but the software was still out there so I said hey why don't we run this on this nugget board and we'll see if any people are interested in this design sure enough this one ended up becoming a hit in the other ones not so much this is also the initial oh this is also the uh the author project as it is today so to clarify we don't develop this product but the deauthor is basically a software tool for wifi attacks that runs on that microcontroller I was talking about the esp8266 so it serendipitously worked on our platform but you can see that developer that things didn't end up working out with is now creating some of these other crazy designs like the deauthor watch and also that thing in the corner yeah so this was the initial design this is where it all started so we drew that pretty crazy whiteboard drawing that is pretty akin to the actual design we have today and then we have this crazy ugly board design that I literally pulled out of my ass in like five minutes and then for a rejected design for obvious reasons we have the wifi penis which we also called by some other pretty nasty names that I won't repeat right now but if you want to ask me at the end of the talk what that is maybe I can tell you guys so at this point it was more of a design experiment to see which of these three designs would sell better um so we had the wifi nugget the hack at the long cat but the wifi nugget is the one that ended up sticking at some point so the first month into developing this product fortunately we made the decision that developing all three of these platforms at the same time was a terrible idea so we decided to focus specifically on the nugget instead still we had no clear idea what this product was actually for it was just sort of a beginner kit you could solder it together it sort of ran a wifi hacking tool that's kind of cool but it wasn't our project um it was a general purpose beginner hardware developer platform it had like breakout pins on the back that you could connect sensors to if you wanted to and it was a beginner programming tool so it was an all-in-one it was your one stop shop for doing whatever the hell you want provided you actually knew how to use the tool so that was the first big fallacy of this design before we actually gave up um designing the other boards you can see we went through a little evolution process here i wish i had bigger pictures of these but we ended up designing some other crappy prototypes of the hack cat and the long cat that didn't exactly pan out oh yeah and then down here is the evolution process of the nugget which the circuit board hasn't changed much but the enclosure has a little bit it's become more of like a gameboy type thing okay so this brings me on to the next key thing that we learned which was that if you're trying to sell a product you really need a big unifying idea so this wasn't something that we realized for a while but basically the point that i'm trying to drive here is um obviously creating a multi-tool sucks because if you can't really explain in a single line maybe one or two sentences what this product does and if you barely understand what it does yourself you're going to have a hard time explaining or selling this to someone or explaining to them why the hell they would actually want to use this tool um this slide is poorly made but i'll elucidate in a second so one common mistake um that i think a lot of designers get wrong and actually where we went wrong is that a lot a lot of them essentially try to do what i call um is just reinventing the arduino so what i think arduino got right with their platform is their target audience is developers if you're trying to sell a product your target audience isn't developers you're creating something that's supposed to have a single purpose use for the person that you're trying to sell to and you should really know who that um who these target people are so i think arduino got that right and that their target audience is um basically developers and people that are um trying to develop products but i think if you're doing what's basically recreating your own kind of multi-tool you're basically acting as a shitty proxy for devices that already exist out there um there are some exceptions to that for example adafruit i'll talk about this more at the end but um what i think adafruit got right is they identified the need for um well basically what they did was they also created their own versions of these multi-tool hardware platforms but they repackaged it in different form factors and that's where they have a one-up over something like arduino you can see they have i think this is called like the lily pad or something it's supposed to be extra beginner friendly instead of breadboard wires you can clip on alligator clips you can plug in a battery so it's portable they have the trinket which is like a really tiny version that you can use if you're not doing a whole lot of hardware stuff then you have the feather which is also battery powered it has like a ton of expansion modules kind of like the d1 mini um but i think adafruit is one exception to this case um that got this right so um i'm actually going to go ahead and skip through some of these since i'm coming short on time but somewhere around the second or third month of our development process we decided to do something bold so we weren't we didn't exactly have a product that we could just market and sell as like a finished thing but we did something kind of crazy where we decided to just advertise it on hack 5 through the videos that we were creating and see if anybody wanted to buy this so we were kind of working backwards we were just experimenting to see what kind of people wanted to buy this design what they were most interested in so the very first video that i did was how to use the nugget in order to detect wi-fi attacks damn i wish i had a video but basically the demonstration that i did was i had one nugget that was set up to do wi-fi sniffing and another one that was running deauthentication attacks to kick clients off a wi-fi network and i basically showed an interesting proof of concept that allowed beginners to visualize what the actual process of running one of these attacks looked like so that really hit people liked that demonstration and they also liked that it was a really cute form factor so we actually ended up selling out of our entire stock within i would say the first two or three weeks of this video going out so that's when i decided that i wanted to focus more on these beginner wi-fi hacking demos since people really seem to connect with that and since most of our target audience that was watching these videos were beginners at this point though we still wanted to try to appeal to everybody so this was still a general purpose multi-tool for hacking for programming and hardware expansion so my friend kody decided to start experimenting with micropython which is a programming language that runs on the nugget so this is where things got a little bit confusing but after selling out of our initial stock we decided to see how long we could just run this thing out let's produce another round of these boards order more components see how long we could ride this project out and continue selling it to people still really with no clear idea of what our product actually did except that people thought it was kind of cute which was actually kind of disappointing because lots of people would just buy our product thinking hey it's cat-shaped that's kind of cool but then they got kind of pissed when they realized it didn't actually like do a whole lot at this point so we basically were just experimenting with this design creating random videos every so often I was creating Wi-Fi hacking videos my friend was doing programming videos and then I think sometime around October or so of that year we flew to Los Angeles and we hosted our very first workshop featuring the nugget just randomly so this was quite a mess since we still wanted to appeal to everybody we advertised this workshop as the general workshop to learn programming with Python soldering and also Wi-Fi hacking so I was in charge of teaching soldering and then right after I taught them how to hack Wi-Fi and then right after my friend taught them how to program with MicroPython so this was all a complete mess people were really confused about this product and it was kind of honestly embarrassing you can see though so the workshop that we did in October we decided to go with we decided to create like Halloween edition nuggets this is something that ended up actually sticking and this is something I wish we committed to earlier on but there's a point I'm going to get to later on in this presentation if I don't run out of time which is telling a story through your product so one of one of the ways to do that is by essentially creating like a character or an icon that people can associate with in our case the cat theme really hit with people and I really wish we leaned into gamifying that more towards the beginning but that wasn't something we realized until the very end but yeah these are some really cute designs that we advertised and ended up selling a bunch of these products so anyways about four months into this project we were getting some pretty consistent sales people were buying our product via our web store which is Redia which we were advertising through our youtube show so we were getting some pretty consistent sales also some pretty consistently confused customers but I was creating more wi-fi hacking focused tutorials so I was showing people how you could do stuff like detect when a particular phone or device enters a room with the wi-fi nugget and I was also showing how you could do some other basic networking stuff like create a like how you could legally practice hacking by turning your nugget into a vulnerable access point so that was all really cool and people really like this idea my friend Cody was still focusing more on the programming aspect my friend Cody was still focusing more on the my friend Cody was focusing more on the damn it it's like I'm going through puberty again my friend Cody was focusing more on the programming aspect of things and at the time it just so happened that the developer of the D1 mini platform that I talked about earlier came out with this new design called the S2 mini so this is where things became really confusing so my friend Cody started experimenting more with this S2 mini since it was based around a hot new microcontroller that could do a bunch more cool things it had more pins it was dual core it was faster and it also had a really hot feature called native USB which could basically be used to run USB attacks and some of that other cool stuff which I'll talk about more in a second but most importantly it supported a language called circuit python which in my opinion is better than micro python it's easier to get started with so my friend started focusing more on the programming aspect while I started focusing more on wi-fi hacking so this is where we created sort of a dichotomy with this product because now we were really confused about what our product actually did we did something kind of stupid where instead of actually developing something new with this new hardware thing we decided to just go ahead and start also selling this on our store alongside the old microcontroller so now we had a split of products where we were selling a wi-fi nugget that was called the D1 nugget with the old microcontroller and the new one which was called the S2 so this is all really confusing because as an average person coming to our store and seeing these two very arbitrary nebulous things how the hell do you know which one to pick and more importantly more people were interested in the wi-fi hacking aspect which the S2 didn't support so we got a lot of confusion where people ended up buying the wrong product they got pissed at us we got a lot of customer support emails and that was a real big nightmare so around the fourth or fifth month this is when we really decided that we needed to figure out what the hell to do with our product so we had some pretty good sales at this point I think we were selling somewhere in the hundreds which was pretty good because we only put out maybe four or five videos and basically the return of investment on that was pretty good for how little effort we were putting into marketing these things so at this point the prospect of actually working with the cybersecurity vendor hack five came up where they said hey maybe if you want to start selling this through our store and you come up with a cool use case for it then maybe we could collaborate on something problem is people were already confused enough about what the hell this product was supposed to do so we would have a hard time selling this to the hack five customer base so we really needed so we at that point we realized that we really needed to come up with our own big idea and something that we could actually sell this product as but since we were also confused about what the hell our actual product did we decided to really double down on the idea that the nugget was truly a multi-tool for everybody and that we were the only ones that got this idea right it supports two different hardware platforms you can program on it you can do wifi hacking basically everything terrible idea though so what we ended up coming up with was that we needed to come up with our own customized software so what was initially lazy about what we tried to do and also is a mistake that I think I've seen I'm not going to call out any specific projects even though I have a few in mind but one of the biggest mistakes that we made was we essentially designed this product and we said well since this is a multi-tool we're just going to dump this into hands of people and we're going to let them figure out what to do with this product terrible mistake and that's also a mistake that I see lots of other makers sort of kind of doing I'll talk more about that at the end actually but basically we were just relying on the fact that there were other community projects out there like the de-author like circuit python and Arduino and we said hey we're going to give this to the users they're going to figure out how to use it themselves that didn't end up handing out so well so we decided to create our own customized software for it called the invader so this one was kind of a spin-off of the de-author project not entirely but we wanted to pack more features into this and make it into a wifi hacking suite so you could use this thing to learn about network defense you could detect if there's a threat on your system by creating a honeypot with this you could also use it to detect the authentication attacks disassociation beacon floods that kind of thing it was supposed to be a general tool for learning about wifi hacking and basically everything you can sort of see where this was going this also became a mess quickly because my friend just kept suggesting like more features to out of the product without me ever actually finishing any of the like initial suggested ones so this also sort of started to become a general purpose like multi tool for hacking and that was kind of a really big mess I'm actually gonna skip this because we're low on time but this was a whoa shut up so that was supposed to be a demonstration of the nugget invader but I'll save that for maybe another talk so at this point we were still selling the d1 nugget the s2 nugget people were really confused about these very abstract naming conventions but we decided hey we finally have a direction for the d1 nugget which is the one that supports Wi-Fi attacks it's going to run this product called the invader or this software thing called the invader and that's what we're going to start selling it with as for the s2 nugget it was kind of lame we decided to sell it with circuit python on our store and inevitably people just bought this product and they were like what the hell it doesn't really do anything but yeah that's where we were at at this point so I think this brings me to my last point which is about picking a niche and then I'll wrap up real quick but sort of in the ideation process or I guess like the creation process of this design since we initially started out as I said just wanting to get rid of all these components and we didn't really view ourselves as actually developing a product we worked backwards to learn a lot of these pain points that we now realize today and one of the very first one of the most important things that I think any product designer should take into consideration when creating something besides knowing your target skill level which I talked about earlier you should also know really who it is you're selling to specifically that's not something we really knew we were trying to appeal to programmers and hackers and hardware developers and everybody but we eventually narrowed that down by just putting our product out there and figuring out who was buying it and then eventually honing in on that so anyways sometime around the fifth month we were in a rush to produce the D1 Nuggets we were going to sell it with the Invader project on Hack 5 since that seemed really promising I would say we were somewhere around like 600 or 7 modules that we sold on our personal store with really poor advertisement but our partner Darren who runs Hack 5 promised that we would make numbers way above that if we started selling with him so we started creating we started batch processing all of these Nuggets in like two days I managed to assemble I think like 150 to 200 by hand and then I spent the course of a month really going ham on developing this Invader project and making it all specced out so we could sell it as a finalized thing on the Hack 5 store and in the meantime while my friend Cody was waiting for me to finish developing that he started working on more circuit python proof of concepts so this is where shit really hit the fan so oh also this is so I don't know if any of you guys noticed but we were developing like through whole modules but we eventually pivoted towards like an SMD design somewhere in the middle of this it's a bunch of nuance I'll actually skip that okay but anyways somewhere around the fifth or sixth month when we were set to launch this product on Hack 5 my friend Cody discovered that the S2 Nugget that he'd been experimenting with could run something really novel it could do USB attacks which was really cool so if you guys aren't familiar with HID attacks which is human interface device attacks this is essentially where a device like the Nugget emulates a keyboard and when it's plugged into a computer the computer is going to inherently trust this since it thinks it's just a regular keyboard that's been plugged in however it can be pre-programmed to run like a bunch of keystrokes super fast and type them out into a victim computer such as this demonstration that I'm doing here so we found out the S2 Nugget had that capability my friend was like hey this seems a lot more clear cut what if we focus on developing the USB attack capability of the S2 platform and sell this to Hack 5 instead it should be a simple proof of concept it's more in line with their tool since they also have USB attack products and it's a lot less convoluted than this dumb invader product and project that we started going started working on that's also another point that I want to bring up kind of disconnected is if you're developing a multi-tool everything is a feature so it's hard to focus in on which things need to be improved or added upon since there's just too many there's too much shit to focus on so anyways we decided to make this crazy pivot at the last moment and I spent another month or so just going hardcore focusing on developing this attack platform instead which we ended up calling the rubber nugget and then I ended up also focusing or completely switching over to this new platform so if you guys are confused about the name the rubber nugget is basically a port manto of the USB rubber ducky and our nugget project so that's the hack 5 product that does keystroke injection it looks like a normal USB but when you plug it into someone's computer starts typing out a bunch of crap and it hacks them so at this point we were kind of having a branding slash identity crisis people were really confused about our product they didn't know where to buy them since we were selling on our store it was called the nugget invader we had the rubber nugget we had all sorts of crazy crap going on so we really needed to basically figure that all out so we ended up going with the name rubber nugget so after I finished developing the USB attack software we decided to do the soft launch on hack 5 in over the course of two weeks we hand assembled over 300 of these devices another key thing is you can see we also damn it oh here we go you can see we also made a little design improvement where we put an enclosed faceplate over it and like a little button so it felt more comfortable than the exposed circuit boards that you might have seen in the previous slides but yeah anyways this was really crazy this was a hard pivot for us and of course lots of our personal customers were a little let down that we stopped developing the invader they were still confused as hell because the D1 and the S2 were floating out there they still didn't understand that naming convention but then all of a sudden out of nowhere this new character appears on the hack 5 store called the rubber nugget turns out we sold out all 300 of these modules on the very first day people thought this was really cool and we ended up turning it into something that was a little bit of a more complete and polished product at this point we had our own packaging that I designed we had our own little insert cards with a QR code on the back I think I had my own sticker that I included but basically this was a USB attack suite for beginners and basically what this offered over some of the other hack 5 tools was it was a cute gamified version of some of the existing hacking tools that made it really easy for some of the more beginner audience in their cybersecurity customer base to get started with this kind of thing because it's a lot less intimidating than like the rubber ducky that I showed you earlier okay so I'm gonna go ahead and wrap up by telling you guys basically where we are today I did not update any of these slides cool but basically some key takeaways that I think are really important having cohesive branding I feel like I wonder if I will slide on that having cohesive branding and being able to tell a story with your product allows the customer to really connect to it I think what I'm trying to say with that one is if you yourself don't understand the product or you don't understand who you're selling to then customers or people that come to this product they're not gonna really understand if it's for them they're not gonna know if it's right for them and they won't want to pick up this thing I also think having more on that idea of telling a story with your product I think it's also important to have I guess I would call it like well like one thing that we leaned heavily into and that we're starting to focus on more is more of like the graphics and the branding and that kind of thing and we're actually allocating like more of a budget to graphic design but one thing that people really connected with with our product is the fact that of course it was cat themed it was really gamified and that kind of thing I think that aspect is particularly useful like when you look at some other companies out there like maybe GitHub for example with their OctoCat I think that's really cute people connect with that that kind of thing that's something that I think is really important to be leaned into besides that I think also the community aspect of this is really important if you are designing a general multi-tool for everyone and trying to leech off existing things that are out there kind of like we did at the beginning which was creating a platform saying you know you can use Arduino circuit python all these beginner all these community stuff that's already out there I think that's a terrible idea and it's going to be hard to build a community around your product that will want to reciprocate and also develop stuff for you so also another key thing about this design is all of it is open source so anyone that wants to create any of our projects or inspector hardware it's completely open source and that's also been a really big thing for us I think so I'm sort of going off on a tangent here except I've prepared these last slides but the last thing is I think also generally a thing that I personally learned from this is well actually I would say that working with and targeting this towards like more of an open source community and like makers and hackers and that kind of thing they've been generally more forgiving about our product and also have helped us along the development process with like this experimentation and that kind of thing and also people have helped contribute to the actual like software and everything like that so really giving people something that they can believe in and understand and giving it like a clear focus I think is really important so stuff that we learned you should choose a niche in the end we learned that that was our beginner cybersecurity audience and not programmers, makers, hackers, developers oh another thing that we learned is identifying who your competition is I think that's one of the first steps if you're actually trying to create a product that you're bringing to market identify who your competition is how you can achieve basically feature parity basically how you can get to the steps where they are identify who's above them and then eventually work up towards that so we worked backwards from there after all of these months we're finally identifying who our target competition is and we're sort of working on the next features to build from there tell a story I just sort of talked about that one yeah that's basically it and now where we are today we're still developing hardware we're still trying to sort out this sort out this branding shit show and we're currently about to hand assemble a whole batch of 500 modules for DEF CON so actually we were just we just did our first experiment without sourcing our assembly to China for the first time previous to this we were just hiring random makers in the community but we made a huge screw-up that I think is a really important takeaway where one of the modules particularly the S2 module we decided to go with a cheaper vendor on AliExpress without actually testing the design but it turns out they forgot to connect a single capacitor to ground which caused our design to take an arbitrary amount of time anywhere from like 30 seconds to a minute to start up but we have 500 of these modules that I now have to fully fix by hand over the course of the next two weeks so if you guys are also designing hardware just a random takeaway make sure you test every little component before you actually slap that shit on a board anyways that was my talk if you guys have any questions anything you would like to hear elucidated let me know thank you guys so much for coming this was a very last minute talk I also want to give a shout out to scale for letting me submit my talk actually last night and then approving it for the schedule today but yeah it was really cool to be able to share this story with you guys thank you guys for coming if you want to learn more you can follow me on twitter at alex linda see what i'm up to you can also follow our project on twitter at hack hat tech and if you guys want to pick up a nugget I have some up here at the end so thank you also I'm glad I didn't piss myself yeah it's actually crappy hello is everyone able to hear me all right I think we'll go ahead and get started unless anybody knows anybody who's waiting for the waiting to come into the room got some stragglers coming in welcome so welcome welcome my name is Nathaniel McCullum and I am the CTO and co-founder of profion I'm also the co-founder of the nrx project which you're going to hear about today and the title of this talk is confidential computing why it has to be open source now what are we actually talking about here when we're talking about confidential computing well there are three different types of runtime isolation and you're very familiar with two of them at least the first is the standard virtualization model and basically we want to with virtualization we want to have isolation because it's really important but what is it so there's basically three properties of isolation the first one is confidentiality this means that people can't look in and see what your workload is doing the second one is integrity which means that people can't tamper with your workload while it's running and the third one is availability which means that your workload actually gets the resources that you are expecting to get now generally the the last one availability is easily observed so if your service is down or if it's you know if it's not getting the CPU cycles you expect it to it's performing poorly then you should be able to observe that pretty easily the other two however are much more difficult to observe and there are three different modes in which we can talk about isolation so type one is workload from workload isolation so this is where I think of its typical cloud system where you have a host and you have multiple workloads running on the same host and you want to make sure that one of the workloads can't reach into one of the other workloads and fiddle fiddle around with stuff and again we're trying to get those three properties confidentiality integrity and availability so the the workload one workload should not be able to for example hog all the CPU resources now we can do this pretty well today right we have C groups this is this is a solved problem but confidentiality and integrity are a little bit trickier and we're going to see why that's because the second type of workload isolation type two is host from workload isolation and this is basically we want to make sure that the workload cannot attack the host itself and this should be pretty straightforward you don't want workloads that are running on a system to be able to attack the host because why well then they'll be able to attack the other workloads so it's really important to notice that with type two isolation the entire of our security model is predicated on type two right so even though we want to type one we actually have to have type two in order to be able to get it now we can do this pretty well today and web assembly for example was designed is a new technology with using new new primitives to be able to provide type one and type two isolation so you've probably heard a lot about web assembly outside the browser and how it's going to hit the hit the cloud space and that's because it provides a lot of great tools for being able to do these two types of isolation in a way that is very reliable very lightweight and so forth however we have a third type of isolation and that third type of isolation is workload from host isolation that is we what if the host becomes malicious and begins to attack the workloads well there's really nothing that handles this today at least not in our traditional cloud paradigms so VMs don't handle this a host can definitely reach in and tamper with whatever it wants to in a virtual machine and containers don't handle this for exactly the same reasons type three isolation is very important for many types of workloads if you have sensitive data for example this could be medical data just to give one example could be financial data it could be regulatory data where you have to comply with laws for example the GDPR however there's also sensitive applications as well that is where the application that you want to run may contain algorithms that are themselves a trade secret and there's number of places that do this lots of AIML for example comes to mind and so we want to be able to protect both sensitive data and sensitive applications but how do we do this protecting from the host now the host could be vulnerable for a variety of reasons it could be because the hosting provider that you that you are running on are themselves malicious right but that's probably not the most common case more common case would be something like a system administrator who goes rogue and decides that they want to cause as much damage as they can on the way out the door or it could be just simply that there is a kernel compromise there's a number of kernel compromises for example that give you ring zero access from outside of the server and so these are really big problems so the host can actually be malicious either based upon the fact that there's an entity on the other side that's malicious or just simply due to a compromise and privacy enhancing technologies are top of mind for these types of situations where you want sensitive data and sensitive applications these are things like homomorphic encryption for example but we'd like to introduce confidential computing now hopefully many of you have heard about confidential computing confidential computing is the protection of data in use by performing computation in a hardware based trusted execution environment and this definition was crafted by the confidential computing consortium the confidential computing consortium is a linux foundation project and it is based on open source software so like other linux foundation projects you can basically start an open source project there and collaborate and this is precisely what the confidential computing consortium was set up to do the ccc has broad industry adoption there are players like intel amd arm red hat microsoft facebook excenture ant and so forth the list is quite long in fact that's not even the full list not even by a long shot so everybody wants to play in this space why? well you see confidential computing uses trusted execution environments and this is based on cpu hardware so it allows you to actually craft a workload that runs in a way that is protected and encrypted by the hardware itself so they encrypt workloads and they predict integrity and confidentiality and this is true even if for example you have a tap on the the bus to the memory right you you literally cannot observe and cannot tamper tamper with the application even if you have direct access to the hardware so what's actually happening without a tee memory pages are are unencrypted so you have a typical workload you've got some memory running and everything is running in an unencrypted way this is what we normally do every day but with the tee the memory pages of that workload are actually encrypted and they're encrypted by the cpu itself so there's new hardware in the latest generations of chips this would be ice lake xeon from intel and the Milan generation from amd and they actually have integration with the memory controller and there's built-in circuitry for doing the the the encryption and so basically when when it comes time to set up one of these execution environments basically you tell the firmware I would like one of these environments please and it generates a new cryptographic key that's used for all of the memory associated with that workload and then all of the memory is encrypted transparently using that key the only way that you can get memory into the initial state is to ask the firmware to put it in which the firmware does a cryptographic measurement of and so now we know exactly what the contents of that trusted execution environment are and then finally you can begin execution by asking the firmware to essentially jump into that tee for you and it's really important that each tee instance has a separate cryptographic key right so even if you were able to tamper one of those keys it doesn't necessarily impact all of the other workloads that are running on the system and it's also important to note to this that this is happening as a hardware primitive so this is not something root has access to if you have a kernel compromise and you are on ring zero you do not have any access to this memory so the memory pages are only ever unencrypted within the cpu so as soon as you do a load or a store instruction the cpu is telling the memory controller transparently to do that encryption back and forth once the memory comes into the cpu and is stored in the registers it will only be allowed to use in those registers when you're executing in the context of the tee and then it clears the state of the cpu when it swaps out of the tee for other applications and there are two broad categories of tees so we basically have implementations from all of the major hardware vendors on the left hand side we have process based tees and the way that process based tees work is that you have a normal linux user space process and when you set up the tee it is a sub region of that memory so you don't actually get a different set of page tables to manage it's all just the normal user application but a certain region of that memory is encrypted there's two primary examples of this technology the first one is intel sgx this is one of the older ones trust zone from arm came earlier but we do not believe it meets the definition of a tee which is why it's not on here but intel sgx is about eight years old but really just saw limited adoption in different places until they brought it now it's now available on every ice lake xeon server as of this year the other one other major one in the process based tees is risk five sanctum so for those of you who are interested in the risk five community there is a design for a sanctum which provides process based tees there's also another one which is actually not on the slide because it's not been released yet which is arm is releasing you see realms on the right hand side they are also going to be releasing a process based te on arm in the coming months on the right hand side we have virtual machine based tees and the way that these work is different than the process based tees so in this case you bring up a virtual machine and then everything that's in the virtual machine is encrypted so every time you cross across the the virtual machine boundary and are out you're doing you're crossing into the te at that point and there are several examples of this technology as well amd sev intel tdx tdx has been announced but it is not yet available in silicon but they are working on patches for the linux kernel you can go google that for more information sev has had three generations now the first one came out on the naples series processors they improved it again in in the realm generation but really where it starts to get really fundamentally interesting is in the latest generation the malon generation because that's when they finally added integrity support for memory pages as well so that's the first one that now fully meets the criteria of the confidential computing consortium a couple other worth mentioning here I've already talked about realms pef is from power so IBM is working on this as well but let's back up here and take a look at the larger picture what are we actually trying to accomplish we're going to talk about what nrx is in a moment but I want to first start off by just giving our nrx foundational principles there's 10 principles that we have in the nrx project the first one is that we want a minimal trusted computing base so in other words there should be as little code as possible in order to get an application up and running and this is just simply for attack surface the smaller the trusted computing base is the harder it is to attack second is the minimum number of trust relationships so who do you have to actually trust in order to get all of these pieces up and working and we want to be able to minimize those as few as possible number three deployment time portability it's really important to that once we have tees that we'd be able to do what I call functional equivalents of workloads and that is when you're doing cryptographic measurements now of these tees to be able to guarantee what's actually inside them you need to be able to determine is this workload the same as another workload well if you've compiled the native application for example on every single tee they're all going to have wildly different measurements and so now you'll have to build a huge management framework on top to track which measurements are which applications and it's very very unwieldy and we want we want something that's simpler than that so think about for example a blockchain use case where you wanted to be able to execute a job and you wanted to be able to certify that this job ran in a tee through an attestation report and then you wanted to put the results of that attestation report on a blockchain for everyone for time immemorial to be able to validate well we want this to be simple and self-contained and so we need some kind of deployment time portability in order to be able to determine that this is in fact this workload no matter whether it ran on Intel or AMD or ARM etc there's another factor of deployment time portability as well which is that vulnerabilities happen and they happen in hardware and in fact every single tee I've mentioned has been compromised now it's not it's not we shouldn't be fatalistic about this the situation is in fact rapidly improving attackers are really having to put a lot of work in order to be able to to actually penetrate these but what happens if you have a hardware of vulnerability today and your application is running and it has all of the stuff that keeps your business running inside of that tee well you've now essentially lost all of that security that you've had because you've consolidated your trust onto a particular hardware vendor so what we want is we want the ability to deploy a workload to another host to another piece of hardware without having to recertify without having to do any sort of you know additional testing it should be just an instant switch if you're running on one piece of hardware to running on a different piece of hardware when it becomes vulnerable principle number four is the network stack outside the TCB and this comes from years of experience looking at kernel vulnerabilities and in fact all of the worst kernel vulnerabilities were all in the network stack and so what we want is we want for the network stack to actually be outside of the tee and the reason for this is because if there's a vulnerability in the network stack that's going to be really really bad and then we don't want that to be inside the tee we want it to be outside number five security at rest and transit and in use that that's pretty straightforward we want to enforce that we always have encryption on everything whether it's on the disk whether it's on the network or whether it's being used it's an entire envelope and once data and code enters the system it never leaves unencrypted number six auditability pretty straightforward that goes with the minimum minimal trust of computing base as well we want it to be small so it can be audited we want it to be seven open source so that it can be audited right many eyes make it much easier to find bugs eight we have an emphasis on open standards on memory safety number nine and we have a commitment to know backdoors and we make that as a public commitment to everyone who might use the project so what is nRx nRx takes web assembly it allows you to take an application written in whatever language you want to compile it to web assembly and you can deploy it using nRx onto any of these hardware platforms so why web assembly well importantly it gives us a bunch of these properties that we really care about so it gives us that minimal trust in computing base it has a minimum number of trust relationships we have deployment time portability we have auditability because we can actually see what is actually happening we know that web assembly is well defined and we also have open source and open standards the same thing with rust nRx is written in rust and rust gives us auditability because it's it's pretty easy to look through rust code to make sure it's doing the right thing it's much harder with c and c++ for example it's also open source all the code we do is open source and as well as our compiler right so we were open source all the way down and rust also provides this memory safety which is a really in critical aspect of the system because memory safety if you if you don't have memory safety then it's going to be a source of bugs for you and we don't want those kind of security bugs inside the TEE so why open source well with open source again we get this minimal trust in computing base we get to use the best libraries that are out there that are well reviewed and well understood and we don't have to write everything from the ground up we also can make sure that what we're doing is small and tight and anyone can audit that which is the next point it's also we obviously want to be open source we want to have open standards and no backdoors so if you look at our traditional virtualization stack let's let's talk about who you actually trust so in just if you want to deploy an application in the cloud these are basically all the different parts you have to deploy you have a CPU and a management engine usually the cloud takes care of that and then you have a bios in EFI usually the hardware manufacturer working in conjunction with the cloud you have firmware for that same thing but notice that they're different colors and that's because all of these are coming these components are coming from different places and this is a lot of trust relationships that can go wrong if you're deploying a common operating system usually you'll have like your bootloader kernel and hypervisor as well as user space and and any middleware you know from that sort of same provider but if you're deploying an application just on bare metal you still have to trust all that stuff if you're doing virtual machine you need to trust a whole bunch more stuff because inside the virtual machine we're going to reproduce all of the stuff that's on the outside including the bootloader and the kernel the user space middleware and multiple applications and this doesn't really change if for example you're doing a trusted execution environment using virtual machines and so when you when you actually have a TEE that's VM based what what actually happens here is that we exclude all of the stuff that is outside of the virtual machine because now the virtual machine does not have an ability to tamper or sorry the host does not have the ability to tamper with that virtual machine and so now we've just clawed back a huge amount of simplicity because we don't actually have to worry about vulnerabilities in all of that part of the stack but we still have a lot of stuff that's inside the VM as well things like the bios bootloader kernel user space in middleware etc one of the things that's really important by the way and it's one of the reasons why nRx has chosen to deploy web assembly rather than a traditional guest virtual machine with a full operating system is because the only way to maintain compatibility and this is how you're seeing everyone implementing TEE VMs in the cloud is that the host is providing the bios and so the cloud provider provides the bios that sits inside of the TEE and then you do everything else but I thought the whole point was to exclude the host from that relationship altogether right and so we have a in order to maintain this sort of compatibility with existing stuff we have to we end up compromising the integrity of the system so what we actually really want is we want to minimize that and reduce all of those layers and just have a single TEE runtime where we can run our application inside and so fundamentally what we end up here with is three components that you need to trust and this goes back to that question of how can we have the minimum number of trust relationships so the first thing that you have to trust in a TEE system that's well designed is you trust the CPU manufacturer well I hate to break it to you if you don't trust your CPU you probably shouldn't be doing computing you know there's yeah that's all I'm going to say about that there's really no way you can have any guarantee about the output of any sufficiently large program if you really don't trust your CPU manufacturer so besides the CPU manufacturer who else do you have to trust well you need to be able to trust that TEE runtime so the bit that's sitting below your application inside of the TEE and this is precisely what NRX provides NRX provides a common open source TEE runtime that is something that can be audited tested because it's open source and it gives you that minimal or relationship so CPU NRX and your own application now I hope you can trust your own application maybe you maybe there's reasons not to trust your own application but I suspect that most of us have a pretty good trust in the code that we're actually writing but there's actually some more that you need to trust because what good is actually bringing up a TEE inside of a cloud if you can't have any proof that it's actually a TEE right I can if you tell me you know please launch my program and let's bring it up in the cloud and I'm the cloud cloud provider and I say oh yeah I definitely set this up it's like super secure best it's the best of securities what kind of security do you have oh it's all of them all of the all of the ones that you like yeah all the securities yeah you've got them all right well this is a bit like asking a kid to grade their own homework so what we actually need to do is we actually need to be able to get some some sort of non-forgeable message out of the application to prove that what is actually running inside of the application is in fact tamper resistant and this is where an attestation service comes into play now again the way that the cloud providers today are building attestation services besides the fact that they bring up a virtual machine with their own bios on the inside of it inside of the TEE they're also then running the attestation service in their own cloud and they are managing in the case of I won't say any names here but they're actually managing the list of the CPUs that are that are good within their their own cloud so this is this is kind of a problem and so you need to be able to trust your attestation service and the way that attestation works is basically if you remember earlier in the talk the firmware brings up the environment and the only way to get pages in it is through the firmware well this is where all the we get a cryptographic measurement of everything that's put inside of that that TEE and the attestation is a cryptographically signed message from the firmware out indicating the state of that particular application and the attestation service is something that's running either in your own infrastructure or hopefully inside a TEE but not in a way that you can tamper with it and the attestation service actually checks the signature on that message to confirm that everything is correct now because this is the modern internet and we do things in a modern way you're probably also going to have an application registry that you're deploying applications from and this definitely also needs to be in your in your trust model although there is there's ways we can minimize that trust by using for example append-only logs and signatures there'll be something interesting some interesting news in the coming months from the bytecode alliance which is the web web assembly consortium regarding this but I will not steal their thunder so this is essentially the components you need to trust right besides the cpu and your own application there are there are three things that you need to trust the te run time the attestation service that validates that the application is actually running in a way that's tamper resistant and the application repository that's going to deploy your actual application into it this is what needs to be open source notice the three pillars are all there right if you can't have confidence about these three important pillars the application repository the attestation service and the te run time then you cannot have confidence in the system because all three of these are required in order to have an efficient system at work so we're going to we're going to give a little demo here now I will ask that I'm going to invite all of you to come to try.nrx.dev just not now hold I know you want to do it hold on yeah if we are going to be giving a demo and I just don't want to swamp it's with traffic while we're while we're doing this demo so I would like for someone to tell me which platform they would like to demonstrate we Intel I heard an Intel oh we've got both of them let's put it to a vote okay who wants to see Intel first okay who wants to see AMD first oh that's really close you know what it's really it's really close close because that's exactly the point you need to be able to move your workloads from one to the other without any friction whatsoever so I'm going to I'm going to do the smart thing and I'm going to click the start now button which randomly picks one because counting y'all's hands is just way too much work okay so we're going we're going to log in again because we are allowing people to run random workloads here and I'm going to walk you through an application this is a very simple Echo server now the thing that's nifty about this is I'm going to toot my own horn a little bit we had a profion one of our engineers Richard Zach just landed in the last few weeks web assembly support into Tokyo and Rust and so this is you're seeing the the newest stuff hot off the presses but this is exactly the experience we want we wanted to just work out of the box with your existing language without having to do anything special no extra tools no modifying your application we want you to be able to use the existing tool frameworks that you they trust so we're using Tokyo which is an asynchronous framework for rust and really the only bit of code here that's specific to running in an arx is that line and the reason for this is because we're going to be pulling in from outside a pre-opened TCP listening socket so it's already it's already opened that's the only thing that's WebAssembly specific everything else here is just standard Tokyo and so what we're going to do is we're going to set the socket to non-blocking we're going to create the final listener and then we're just going to start our loop and we're going to listen for incoming connections when a connection comes in we're going to spawn a background task now this is this is async this is not threaded yet but this is asynchronous so we're going to spawn a an asynchronous background task to handle each incoming connection and we're going to come down here we're going to read bytes off of that socket we're going to handle when the socket disconnects and then we're going to write the output from that to standard output and back on the socket right pretty straightforward application and so let me show you how to compile it for web assembly normally and if you're doing rust you would do a cargo build if you want to do it for web assembly you add target equals wasm32 wazzy yes I can embiggin it all day long my mom taught me how to do this so yeah so normally you would run just a cargo build but we could run a cargo build we're just specifying a different cpu target web assembly is just a virtual cpu for those that don't know and web assembly is deployed in browsers all browsers today for many years so it's a it's a well-recognized mature technology there is a wasm64 in development so the standard is basically done but there has not been a huge amount of demand for it interestingly enough the there are some well so there's not there hasn't been a huge demand in relation to the to the amount of effort that it's going to take to implement it that's the way I should put it because we get a bunch of sandboxing stuff for free when you can have 32-bit pointers on a 64-bit system and so there's a whole bunch more work to implement it but people are working on this yeah it'll it'll be available so yeah so wasm32 wazzy and we have an application now that that's it so if we look in directory here you see this tokyo-echo.wazm and that's our application that's all we've done so we want this to be an exactly out-of-the-box experience and we have to have a little bit more and the reason for this is that when you're running an application in NRX you can't take things like environment and arguments from the host platform because the host is not trusted in this model right so you have to be able to grab those from somewhere else so we have a configuration file called nrx.toml and I'm just going to paste this in here and this is pretty straightforward we're just defining that we want standard in standard out and standard error as well as we want to open that additional listening socket now if you remember back here I said that this line of code right here is the only thing that's WebAssembly specific well this is because we are pre-opening this file descriptor number three and this is right here where we've created that file descriptor number three so that's the only thing that's really specific to this environment and so now we're going to pick our wasm file and we're going to click deploy I'm sorry oh the file the file is pretty small it's well that's the debug version let me do the release version you're taking me off the path in my demo this is where all the dragons lie so I'm just adding the release flag and this will just build the application with a bunch of additional optimizations including optimizations for size so yeah yeah yeah so it's two it's 2.1 meg but that's everything right think of it this way the entire nRx runtime plus the application is still smaller than the bootloader and linux in booting a VM right so just put that in perspective okay so we have our application running here over on the left unless it hasn't timed out yet and uh we're going to all right it's connected and so I'll say and there it is yep so now that's interesting but let's let's run that same workload on a on a different technology so we did intel first now we're doing AMD notice there's no changes whatsoever to the application and if you were deploying in this in kubernetes right it's really just like a matter of picking a different a different target that you want for for your application so you can be up and running in a vulnerability in a matter of minutes so we're going to do the same thing on s and p I'm connected again and this ah whatever you guys can see it now you can see that this is actually a distinct system that's running now I talked before about making sure that we could have an envelope where every time that data and code entered the system it stayed in the system encrypted and never left unencrypted so one of the problems that we have I don't know anybody saw the article recently that there's something like some crazy number of open security jobs in the world right now and a big part of this is because although we've learned how to do devops at scale for example with github and your cloud provider right we still haven't learned to do security at scale yet and so everything is like picking and matching these like different tools that you have for issuing certificates and getting renewals and making sure that disk is encrypted and you have to sort of glue it all together in a big ruby Goldberg machine and it's very time consuming and it's very error prone and if you're asking a junior guy to do it you have no way in hell that it's ever gonna be secure so what we want is we want to be able to we have people telling us all the time we should shift our security left how many of you have heard shift security left right okay the problem is shifting security left doesn't actually work and it just blames the developer for not being able to be perfect every single time is anybody perfect every single time in this room I'm not so I don't think that is and blaming the developer doesn't really help us we actually need to shift our security to the right we need to be able to scale our security the same way we've learned to scale our developer experience and the same way we've been able to scale cloud deployments and so let's see what it looks like to do an attestation and we'll do it on S&P again since it's right here and with our attestation we're going to we're going to enable the steward steward is the attestation service and we're going to change the protocol of our port from TCP to TLS and we are going to run the same exact application again so what's actually happening here now is that the the workload has gone and done an attestation as part of the startup procedure so it went out to that remote server gave evidence through its through its attestation report to the server that this is a tamper proof environment the server has validated everything cryptographically to make sure that all of the measurements are known and so forth and only once that's done does it issue a certificate for a private key which was generated or for a certificate on a public key for which the corresponding private key was generated inside of the TEE and so we can now do openSSL and connect to this and you can see that we got exactly the same thing now notice what I didn't do I didn't install a bunch of other additional tools and I didn't have to worry about whether or not my key is going to be compromised or whether my disk was encrypted because everything was encrypted always and so there was nothing to mess up right all I had to do was change the configuration and we were able to successfully shift the security to the right so let's do another demo this one is going to be a video demo just because I want to show a pretty complex attack and that requires me to have root access on the server so let me go back to here and so we wanted to show that you can actually do something that's a little bit larger than just an Echo server so this is actually a multiplayer wordal clone that we wrote and basically instead of trying to you know guess the word of the day you're actually trying to guess the words that the other players have put into crypto and so as you type the words in they show up so it's typing in the word world and there it is so forth I'm going to skip ahead here and so notice purple we got a match that second player had successfully guessed the first player's word and now this is important that the unlike wordal today wordal is generated for those of you know the word is sent to the browser and everything is done service or client side and so if you want to know what the word of the day is and you want to cheat you can just do that trivially right with any browser debugging but in this case all of the words that are being guessed by the other players are now held on the server inside of an anarchs keep and because they are inside of this keep everything is protected from the host and so we want to see what it would look like if we wanted to cheat in this game so we're going to put on our attacker hat and we're going to run the crypto application using web assembly using wasm time wasm time is not an arch now please don't fret I'm not picking on wasm time wasm time is fantastic in fact it's our web assembly implementation as well inside of the keep so this is not about about dising wasm time at all but it is however about pointing out how easy it is with root access to be able to find secrets in an application so we are we've launched it in wasm time and we're now just going to grep the memory for words so we found the process and we're off to the races so in a moment what we're going to see here is we're going to see the words that the other up there it is young is one of the other guesses that other players have made and so all it takes is one root level vulnerability to have this level of access to all of your virtual machines all of your containers all of your applications running on a host in a traditional manner that's a problem and we need to be able to do something better so we're going to show the same exact thing running again by the way we would get if you were running anarchs in debug mode you could scan the memory like this too right there's when you turn off debug mode there's there's no protections so we're going to do the same thing again here we're going to attack crypto but this time we're going to do it using anarchs and so we're actually doing an upload here we're uploading the application to drawbridge and then we're going to drawbridge is the application repository and we're going to deploy it right out of the application repository that's that line anarchs deploy it's actually way simpler than this today this is an old video oh I missed the good stuff so we're going to do the same thing we found the found the anarchs application and we're going to do exactly the same memory scan just for the different process it it means that I don't remember it's a kernel it's a Linux kernel thing and I just don't remember exactly what case causes that and that's it we didn't find any words so just simply by changing which host we were using whether we were using wasm time or whether we were using anarchs on supported hardware we were able to defend the application from attack now this is actually a call to action I somehow lost my slide share there we go we want to see your elite skills so if you go to anarchs.dev slash crypto we will basically give you everything you need you'll have the anarchs application have crypto and we want you to hack it and there will be cake and by cake we mean hardware and cash so please come give it your best attack and the important thing here is that profion is about to launch enterprise support for this and we want to make sure that it's ready to go so please please come and do that we will be announcing the winners of the first round at black hat which is coming up here very shortly and so yeah you have a little bit of time to get your submission and basically if you can run crypto in anarchs and you can successfully attack it from the host side then yeah you win last let's do one more demo so we actually didn't make this demo and I think it's important to show this demo because I want to show that other people are making stuff too and it can work in anarchs out of the box so this demo was actually put together by microsoft this was given at kubecon so we gave talks at kubecon for there was a web assembly day