One second. Yeah, look, have audio. Sorry about that. That's a great entrance to make. Just walk right in and say nothing. Hmm. Hi, welcome to the show. It's me, JP. This is John Park's workshop right here on Adafruit TV. We don't really call it that. And thanks for stopping by. I see we've got some people over in the chat over on the YouTube. Hello, everyone. Nice to see you. And let's see, I'm going to put on my glasses so I can read the chat because that's not going to do just guessing what those people are saying. Hi, Dave. Hi, David. Hey, Dale. Hi, Connor. Larry, what's going on? Dave Lindy says, just notice the bottom right corner. I didn't already hit that. Did I? I did not rage quit. I am here. But we are going to talk about rage quitting today and build a device for rage quitting. And if you're wondering what rage quitting is, we'll even talk about that. But you can probably guess from the name. And hi, thanks, everyone, for stopping by over in the Discord. If you are wondering where people are chatting from, that's a lot of our chat right there. So if you're tuning in on Twitch and it looks pretty quiet around there, or Periscope, or LinkedIn, or one of the other places that we're broadcasting, you might want to head to Discord. It's there. You can go to Adafruit.it, Adafru.it, that's our URL shortener, slash Discord, and you'll get an instant invite. And then you can hop over into the chat and say hi to all these nice people, including Bruce, Mr. Certainly, who's waving to his mom saying, hey, I'm Adafruit TV. Very nice. Well, thanks for stopping by. So let's have fun today, I say. We've got fun stuff prepped. And some of it's going to be on the fly because I came up with part of this idea at the last moment. And so I was trying to set up a laptop over there to see if we can do a demo, and we'll see how that goes. It should be exciting. First of all, let's mention our jobs board. We've got a jobs board and it is at jobs.adafruit.com. And if you head over to that site, jobs.adafruit.com, here's what you're going to see up at the top. In fact, there's a link for not only search jobs, that's the default, but available for hire. And if we click that and scroll down, you'll see that there are a lot of people looking to pick up work. So if you are looking to find someone to do something for you, head on over there. If you are looking for work, head over to search jobs. I saw kind of cool one in here. It was a, where'd it go? It was a, someone to build a rhythm machine, like a solenoid drummer. Did I make that up? I thought I saw that earlier. Let's see if I go back a couple steps. No, it's gone. Anyway, search through there. Maybe that job's already been filled. You better hurry if you're looking for a cool job, because apparently they go quick. So that is jobs.adafruit.com. Go check it out. Let's see. Next up, so as you may or may not know, on Tuesdays, I do a different show that's called Product Pick of the Week, JP's Product Pick of the Week. It actually looks a lot like that. That was this week's Product Pick of the Week, this rotary trinky. And if you tune in on Tuesdays, you can get a 50% off discount typically on the Product Pick of the Week. That was the case this week. And that discount is only good during the show. So you should tune in. It's on at four o'clock Eastern time on Tuesdays. Last about 15, 20 minutes to some demos. And this week's one minute reduction looks like this. Rotary trinky. It is a USB plug and play rotary encoder board that also has a neopixel for feedback and a capacitive touch for additional input. This sends USB HID commands. I can adjust that frame forward, frame advance, one frame at a time. I programmed this to turn on closed captioning when I click the little button. I've also set the cap touch slider as a play pause. It is the rotary trinky. It's a USB rotary encoder that plugs right into your computer and allows you to use a rotary encoder, click and touchpad. That one's on loop. We're just going to watch that all day if I don't get on the controls here. Let's see. Next up, what I've got for you is our circuit python parsec. Am I really muted again? All right, I get to go through this the second time. Thank you, everyone, for letting me know in the chat. Good. I didn't really want to be on the side of this thing anyway, so let's arrange this a little more nicely. Hold on one second. I've got an idea. You get to put up with me fixing something here real quick. Let's duplicate that. Workshop charades. What is he even talking about? Let's see. If I put that there and that there and hold on that there, that there. Yeah, that might work. I'll scoot that just a little bit. Hold on. We've almost got this. I thought I had this set up already. Pardon me. No, I don't want to move that. I want to move this. All right, that's going to work. All right, there we go. Get ready for it. Here we go. What is this thing? For the circuit python parsec today, what I want to do is talk about pulse width modulation. Pulse width modulation is a way of faking an analog output where rather than smoothly sweeping through voltages, we make very fast little steps that look analog to the eye. But the way they work is that we're essentially flicking a light switch on and off real fast. So the full on and full off, if we change the amount of time between those square steps, then we can average out to different voltages. So you can see here in my code, what I have is I'm importing time so we can pause, importing board so we get pin definitions, and I'm importing the PWM IO library. Then I'm setting up this pin on the LED pin, which is a PWM IO output on one of the board's pins. In this case, I picked a convenient one, the Mosey pin. And then we're setting a frequency as well as a duty cycle. Then we set the duty cycle to a particular value when it starts up, which is about half, which would approximate a one and a half volt being sent to this LED and the resistor that's there to protect the LED from over current. Then in my main loop, all we're doing is stepping through, increasing and decreasing the spacing between these full on, off square waves, which as those go in and out of phase or speed between them, we get this approximation of a smooth voltage. So you can see here in the case of this little cutie pie, I've got a big fat LED on there, and it looks like it is nice, softly, gently changing its values like you'd expect an analog output to do, except I'm not plugged into an analog output at all. I'm just using one of the regular digital pins and essentially triggering it on, off real fast. And so that is how you can use PWM in Circuit Python. And that is your Circuit Python parsec. Yes, someone mentioned in the chat that maybe setting up a rage mute rather than a rage quit would be good. I should have a big button for my muting. Maybe I need a huge knife switch right here for mute. So I always know. All right. So if by the way, someone had a question about, I think someone had a question about the PWM. If you want to head over to the learn system and you check for Circuit Python essentials, there is a link on the left for the PWM talks all about it and talks about the duty cycle and frequency that you use. And you can use this for things like servo motors as well as LED. So there's a number of different ways to use PWM to send something that seems like an analog value. So I hope that's helpful. Let's see. What's going on next? Okay. So I think, are we there? Yeah, we're there. So let's talk about a project of the day. So first of all, I mentioned that I want to make sure that we end on time today because Scott is going to be doing a deep dive at two o'clock. And I probably won't go that far, but yell at me if it seems like we're going long. So for today's project, I teased it here as this big rage quit key. So first of all, rage quit is what it sounds like. It's a word used by gamers to describe getting so mad at something that's happened in game that you just quit the game entirely. Sometimes you send a nasty message on the way out. Sometimes you just hit alt F and get out of there. So on Windows, the alt plus F4, the alt F4 will kick you right out of whatever application is currently sort of has focus or whatever app you're using. If I were, I'm on Mac, but if I were on Windows right now, broadcasting right now, my broadcasting software has focus. If I hit alt F4, bam, I would go away. So on Mac, we can send the keyboard combo command option shift escape, and that will do the same thing, not as sexy to say it's alt F4, but that will just boom, boot you right out. And as you can imagine, if you want to, in some games, there are ways to automate things like sending a message and then quitting the match and then restarting the match or quitting the game entirely. And so you can probably see where I'm going with this, which is we can do USB HID, both keystrokes and writing keyboard commands to send kind of anything we want.  Davo Desa mentions he had Boss Key. This is another name for this where you were playing Doom and you had to quickly make it look like you were doing a spreadsheet. So often there was a keyboard combo to switch away from the app or put up a fake screen or something like that. So Boss Key or Alt F4 or Rage Quit, that's what we're going to make. Now, you know I love the mechanical key switches and I've been doing a lot of projects with these little guys. So here's one of the Kale box red linear key switches, nice little key switch here. And so I want to do something with that. And as you can imagine, we could set up something like our QT pie and this key switch. We've got our feather, two button version of that sort of thing as a feather wing. We have the, let's see, I don't think I have it right here. Let me grab it. We have our Neo Key Trinky, which would work pretty well. That just plugs right into your USB port and you can use that. But for something more satisfying, I wanted to try a different build. So let me switch to my workbench over here. So you can see I've got another key switch. This is a Kale. It's also a linear key switch. But the big difference with this one is that the scale of it is just a little bit different. So as you can see, this is one big honking key switch. It's actually, if I grab my regular Cherry MX compatible Kale key switch, you can see there's a bit of a difference in size there. This one is four times larger in all dimensions. So 64 times larger in volume than the poor little regular one. And it even comes with a gigantic key cap. So you can see right there that is a satisfying, that is a satisfying key switch for building our Rage Quit machine. And so what I wanted to do first is show you kind of how I want to approach integrating a microcontroller with this so that we can send the key command from this big, gorgeous, beautiful switch. And by the way, a little word about this for a second. This is from Novel Keys, a company that makes key switches and partners with Kale often, K-A-I-L-H. And if I head over to my browser here for a second, this is the big switch series. These are out of stock right now, but you might want to sign up if you're interested in finding out when they come back in stock. And they make the yellow one I have, which is linear. They make a tactile bump one, which is the red one. And then they have a very clicky blue one, which is the one here on the right. And these were on mass drop at one point. They have some really nice photos of it. Colin Cunningham made a project on the learn system using a set of these, and he built a nice stand. Let's see. How about box switch? No, key switch. How about key switch? Colin. Colin? No, Colin. I found it anyway. So here you can see the circuit playground express here. And he's wired up the three switches. So this is really cool, nice macro keyboard here. So what I'm doing is a variation on this. And I love this stand and sort of macro keypad, giant macro macro keypad that he's built. I was curious though, since these things are so big, could we actually fit the microcontroller inside? And spoiler alert, the answer is yeah, pretty much we can. So what I've got here is a, zoom in a bit more here, if I can, I should do it. So I will, maybe I'll take one of these another time for you. So it's a little, it's not too hard. It actually opens just like a real key switch, but I found it works better to use a spudger. And I kind of don't want to pull it apart right now. But it's identical to a real key switch on the inside. And these are the two contacts here that are closed when this is pressed down. So this just shorts a digital pin to ground is how we'll use it. One other note I'll make is that they've even designed this to use a large LED. So this is this is how an LED on a traditional LED, not a RGB neopixel style one, but a traditional LED key switch like on your cap slot. The Cherry MX style has a little slot here for a, for an LED. And I just, I put a coin cell over here that I could hook this into just so you can see it light up because that's fun. So there it is all, all glowy. And if I can get that to stay there, I'm going to put my key cap giant key cap back on there. Look how satisfying that is, huh? Big and glowy. So I think ultimately what I'm going to do, I probably won't do it today, is add a neopixel. And we've got some eight millimeter RGB neopixels that are this style look like a gum drop style. So I have a little, I'm going to order a little set of those after the show and add that to this build so that we can do RGB under lighting. Since this is a nice clear key switch, you can actually also light it from below. So you could put a small neopixel ring maybe under the, the stem there if you wanted to. But what I'm going to do is you can, I think Colin might have soldered to these or use alligator clips. What I found is that our terminal connectors for arcade buttons actually fit pretty well on these. And if you bend them, you can attach them, let me put them like this, you can attach them with pretty good clearance just by sliding these onto those terminals.  So if we get those on there like that, now we've got a couple of wires that I'm running, again, to ground and the, I think, MO, MOSI pin, and then using that as a digital pin, not as part of SPI. And then if one thing I noticed taking this apart, pretty much all of the inside of the switch here gets taken up by the stem as that moves down. So you can't really put much inside inside it. But there is this little cut out here, which is where some underlit LED light tends to come through, depending on the design. That actually has just enough space to fit our, and let's see, how did I have this? I think I flipped this over to eat up some of that wire length. So that'll fit with one of our new little right angle USB-C connectors. That's often the trick with these things is how to get the USB-C plugged into something that needs to be in a tight space.  It's actually not used on this particular key switch, but that now fits. So I'm going to tuck those wires in like that. Tuck this back here like this. And then right before the show, I 3D printed a little stand I made to try to allow, because right now this thing isn't going to stand on its own. This thing is a little wobbly, because it's meant to pretend to be mounted in a PCB. But these have like a little couple of tabs that spring in and out to click into place. Let's see if we don't break this 3D print. Hey, there we go. So now we have kind of a nice little stand. It still shows off the insides, because it's kind of fun to still see that rather than hide it in a box. And then we can set that right on there like so. And now we have a nice stable button. So you could use this, let me switch my cameras speaking of using switches. So you could use this for something like a mute button if you want to. You could use it for an app switcher. You probably could go as far as to code it to do different things based on patterns. So if you want to maybe hold it for a second first and then one tap, you might be able to write your code to do more than just one thing with it. But I'm just going to do the rage quit thing with it. So let's demonstrate that. And I'll show you the code that I worked up for this. It is straightforward. And that's one of the nice things about once you've started doing some of these USB HID projects is that they are very, very similar. You can get away with reusing code really nicely. Let me rearrange some cameras here so you can see this big honker as I code that. Let me turn a camera to the side there. And how about that little view there? Got some blue tack from the last thing. Look at that. Yeah, I can't wait to get an LED in there. I just didn't have time today, but I will. So I'm going to plug this in and this is dangerous now. In fact, I'm going to click on something else. I'm going to actually launch text edit. So I just have that off in the side. I'm going to make sure that has focus when I plug this in just in case I've done something wrong and it rage quits it. It did not. Okay, that's a good sign. And so let's take a look at the code for this. So I'm going to pop up my atom here and let me move that other window real quick for you. Where is it? There we go. Okay, thanks for your patience as we set up cameras. So I'm going to open up the code that's on that board. This is it actually. So the code that I have on that board. Let me see if I can scale that a little bit. Read that. All right, more camera juggling. There we go. Now we'll be able to see it. Okay. There we are. So here's what we do. Import time in case we're going to do any pausing or things, especially with the neopixel, I haven't got that in there. So right now it's unnecessary. Import board so we can have the pin definitions using digital IO to read that switch. I'm importing neopixel, although I'm not yet using it. And then I'm using the Adafruit debouncer, which I love for this kind of stuff. It makes your life so much easier when just checking to see have I pressed a key and have I released a key. And then I'm importing the USB HID, the keyboard, keyboard layout and key code. And with those three things, we can do individual keys, we can do a macro set of keys that are a keyboard combo. And we can type stuff out. Then I'm setting up the key switch input as this digital IO pin on the board's MOSI, MOSI pin, as an input with a pull up resistor. And then setting that up on the debouncer as big key switch. Then I set up the keyboard. And then this is the code. While true, we update the debouncer. That's what that big key switch dash update is. The debouncer just checks to see, hey, has anything happened? If it falls, meaning it's been pressed, then we're both going to print the word press. And we're going to run this keyboard combo. So I'm doing command, option, shift, escape. And that is the magic combo on the Mac.  It'll go away real fast. And then that's it. And I'm actually not using the keyboard rise for anything right now, but I probably will once I set up the neopixel. So to demonstrate this sort of safely, what I'll do is I'll create a new screen capture. And let's put that on this layer. New screen capture. And I'll just, I'll do a monitor capture actually with this one. Just give it one second. Okay. So if you look here, I've now given this text edit focus. And I'm just going to type rage quit in there. How big does this go? Rage quit. There we go. So when we're ready to rage quit. And I'm going to put, I'm going to put this on a different layer for a second here so you can still see my key switch there. Okay, so I've got this app as the forward app. I come over here, I press boom, and it's gone. That just decimated the app. It just, it doesn't ask anything. It doesn't say, Hey, do you want to save? Oh, it's just gone. So perfect for when you get mad, just capoom. Someone asked, does that combo bring up the force quit dialogue? No, that's command option escape. So it's the addition of the shift that takes no prisoners and asks no questions. So yeah, command option escape will bring up the little force quit dialogue on the Mac. This thing, which Todd bought told me about earlier, because I was running this idea past him, command option shift escape, that is the closest to the alt f4 on Windows, which does the same thing. It just blows the current focused app right out of the water, which is awesome. I don't, I don't know if it'll work as a force quit if something is hung, it may also. Let's see, Dave Lindy says command w or command q may be closer analogy to alt f4.  So let's see, let me, let me check the chat. Any, any thoughts or questions? You're probably yelling at me like, don't do this, this is unsafe and terrible. Let's see, hey Discord. Let's see, Todd bought you, you just got a note, no spamming thing from the bot. What did you do? Oh, you're saying the same thing over and over again. You can't do that. Our Discord bot will get mad at you. Let's see. Yeah, Dave Lindy's, I didn't know about that one. It is good to check every once in a while about new, new keyboard shortcuts for the OS because they seem to, to come with new updates. Yeah, someone found a link to those. Todd bought us this Adafruit going to make a huge trinket for that. Can you imagine you just rip your USB port right out? This is a bad idea. Look at this thing. It's so satisfyingly big. I'm going to unplug it, actually, for safety. I'll show you it up close with my little table there, little coffee table side stand thing for it. And then I just wanted to try and see over on, I didn't get this set up in time, I think, but let me, let me run back over to the, to the workbench and see if I got this Mac to run Fortnite, which I thought would be a really good one to, to Rage Quit. No, it didn't. I only got that far. Okay. But yeah, we can, we can blow away this. So I'll show you this is zoom out a bit. So you can see here, we'll just plug in our little USB cable there. I really like these new USB cables. There it is set up. So whatever app we've got forward. Oh, did it not close it? Am I not plugged in anymore? What happened? Didn't work at all. Did it not actually recognize my, oh, there it is. Yeah, I didn't plug it in fully enough. This is kind of an old laptop. So it's got a bit of a janky USB. There we go. All right, so that should work now. Boom. Chrome is gone. What else can we launch? Let's take the terminal out. Will that work? Should. Boom, gone. So too much fun. Of course, it's highly dangerous because people will walk up to your desk and want to hit it. So maybe only plug it in when you're ready or have other switches on it so that you can toggle it on and off or modify it. I looked around briefly and I've seen there are definitely other cool projects with this kind of idea in mind. So look around. I saw someone had a selector switch on theirs. They used a big arcade button and they had a little selector switch and three LEDs to say, are you quitting happy, medium, or angry? And depending on which, it would send, I think it was like Starcraft or something would send a message that was nice, neutral, or not so nice before quitting out, which is kind of funny. Naradak asks, how about a giant kiddie paw, toe beans key switch? That would be amazing. Doctors say, I want that. Where get? Yeah. So go check out. Tell novel keys. We want these back in stock. I don't know. It's been a while since they've had them in stock. So hopefully there's enough interest to get them to start making those things again. And what else? I think that's going to do it. Yeah. So the main addition to this from now is to do a neopixel. Trying to see if I have those RGB neopixels that look like a big gum ball kind of one. See, I've got plenty of these. These are these nice 10 millimeter single color ones. And those will work great too. And that's actually a little more accurate to what a, or could use this crazy three watt thing and just melt your key cap right off. But yeah, that key switch is designed to have a through hole LED running right up through. It was the original style of like the Cherry MX. So it's more of a modern invention to put these neopixels underneath them. All right. I think that's going to do it. So thank you everyone for stopping by. I will be updating this and putting together a learn guide. And of course, one of the things about this is that, you know, it's a lot of fun with this particular switch, but you could use any kind of arcade switch you want. These terminal lugs that I'm using here to connect the wire actually meant for arcade buttons. So you could do this as an arcade panel type of thing, oversized arcade button like we have in the store. So there's a lot of options there, but this one's a lot of fun. Listen to that. So big. All right. Thank you everyone and stay tuned for Scott's Deep Dive at two o'clock. That's in about 22 minutes from now. And that's going to do it. So I will see you next week on Tuesday for another JP's product pick of the week. And I will see you next Thursday for another John Park's workshop. Thanks everyone. Bye bye.


So next we have, next talk we have is a human speaker talk. I would like to welcome Dr. Christopher Teganelli. He's an associate professor here at the division of active care surgery here at Minnesota. He definitely has a lot of feathers on his cap. He's the affiliate faculty at the Institute for Health Informatics, the scientific director for the program for clinical artificial intelligence and the co-director of the Center for Quality Outcome Discovery and Evaluation. He's triple vote certified, welcome to GCC. He's an active care surgeon. So he mainly deals with critical and trauma patients. So basically the course of action that he takes to take care of patients is really important. So his research team mainly focuses on using machine learning, AI, as well as informatics space tools for better patient care. So I will let him continue about how he does this. Welcome Dr. Christopher Teganelli. Thank you very much. I'm really excited to be here. And today I'm going to be talking about a rib fracture clinical decision support system. And later on we get into how we've integrated artificial intelligence into this decision support system. So I'm going to talk about my journey as a quality improvement researcher and how that's actually transitioned into becoming more of an artificial intelligence researcher. So when I started my research program back in 2016, I sought to reduce the third leading cause of death in the US. And that is preventable deaths due to medical error. Oh, there we go. You might be surprised to hear it, but preventable deaths are rather common. There's about 400,000 preventable deaths each year, making it the third leading cause of death. And that's up about four fold in the past 30 years. Now when we talk about medical error, what are we talking about? There's five different types of medical error. The first one is an error of commission. And that's when a patient gets a treatment and that treatment wasn't indicated to the patient. The next one is an error of omission. And that occurs when a patient doesn't get an evidence-based treatment. The next one's an error of communication. And that's actually one of the most common causes of medical error. And that occurs when there's typically multiple different services taking care of a patient and failing to communicate between them. An error of context is rather rare. This occurs when the provider doesn't take into account a unique patient situation. So for example, they order a medication that they can't afford. And then the last error is a diagnostic error. So making the wrong diagnosis. And this also includes a delay in diagnosis. You know, for sepsis, for example, every one hour delay increases someone's mortality by 2%. I'm gonna focus on errors of commission and errors of omission. I do a lot of interviews with physicians and I ask them, you know, why'd you order this treatment? Why didn't you order this treatment? One of the most common things that we find out is that a lot of physicians just weren't aware that certain treatment was the standard of care or the most evidence-based treatment that they couldn't deliver. You might be like, well, how is that possible? And I'll give you an example. So in my specialty of acute care surgery, we have to cover emergency general surgery. So that's appendicitis, heptic ulcer disease, gallbladder disease, hernias, trauma with new heart trauma, traumatic brain injury, blood vessel trauma, spine trauma, critical care, heart attacks, stroke, et cetera, and ECMO. It is impossible for any physician to keep up with all of the standards of care, all the guidelines, all the latest evidence for all those disease processes. You're talking about over a thousand different guidelines and evidence-based practices, for example, just in this specialty alone. And that's unfortunate because we spend billions of dollars on research to try to identify what are the best evidence-based practices. You look at those multi-site randomized control trials that are gone, this costs millions of dollars to do to find out what the latest evidence-based practices are. And we spend a trillion dollars in healthcare annually. And despite all of that, we repeatedly fail to translate medical knowledge and capacity into clinical practice. It takes 17 years for evidence-based practice to actually become ubiquitous in care. So the big paper, the big randomized trials published today in New England Journal of Medicine, will take about 17 years for providers to really bring that into clinical care. We only spend a couple hundred million dollars on actually improving healthcare delivery. So developing informatics-based tools, for example, that can improve taking all of these evidence-based practices and bringing them to the physician at the point of care when they're seeing that patient in the clinic. So my research objective is to develop informatic-based techniques to improve the delivery of evidence-based practice into clinical care with the goal of reducing medical errors and ultimately preventable deaths. We do this through three different tools. The first one is building clinical decision support systems. And that's a broad bucket. Typically, at least for this talk, we're talking about clinical decision support or CDS. We're talking about ones that are integrated into the electronic health record. And that's EHR, as I accidentally say that, electronic health records, EHR. AI-enabled tools, we're talking about a bunch of AI-enabled tools in the second half of this talk. And then also by developing local solutions, we've developed and deployed multiple local solutions. And we'll talk about those that much during this talk. But one of the most important things that we need to even do this research, one of the things I realized early on was I needed data. We needed access to very granular data on patient care.  It's very easy to get. What was it? A lot of what I've done and had to focus on is actually building data strategy and our data infrastructure in the last five or six years. I'm excited to show what that looks like now. We need data to find out what disease process should we start between build a tool for. We wanna find out how our outcomes compared to other hospitals. So we need other hospitals that too. How does our adherence with evidence-based practices, what's our adherence with evidence-based practices? If we deploy a solution, how do we evaluate that solution? And then we obviously need data for the training of the AI tool. So the first learnings that I realized, which I really wasn't expecting, I figured they have, oh, naturally you have a data solution kind of ready to go. And I saw this now as at UNC Chapel Hill, Michigan, and at University of Minnesota, is that you need to have a data strategy. I needed real-time clinical data. We had highly granular data. The data strategy has to encompass unstructured data. Notes, images, that's 80% of healthcare data right there. You need to collect social determinants of health. 40% of a patient's healthcare outcomes are driven by social determinants of health. Now, they're genetics. Number one thing is social determinants of health. You need to integrate data from multiple sources. A lot of people focus on just the electronic health record. That's only a small piece of the puzzle. So the historic, so what's the historic approach to healthcare data? What was it like when I got here? What was it like at University of Michigan? What was it like at Chapel Hill? There's large national databases that we have. There's also a lot of local disease registries. We've got cancer registry, transplant registry, et cetera. There's also these small and use case specific databases. Maybe colon surgery database, cardiac database, et cetera. Most of those databases, in fact, potentially all of them, they have a couple hundred data elements per patient, then poor longitudinal data, poor ability to follow patients across hospital transfers. You don't have a unique patient ID, especially in the larger national databases, value identified, oh, this is the same person just in a different health system. Limited access to provider level data. So if you want to look at the actual, the physician or the advanced practice provider is taking care of that patient, very limited access to that. Most of the data captures by manual check chart abstraction. So someone reviews the medical record, looks through, oh, did this happen? Yes, enter a zero, one in a database, for example. A lot of the data in these datasets comes from the larger academic hospitals. And there's very limited data for rural community hospitals. There's a long way to go to build a new dataset. If I said, hey, I want a disease process for, or a database for everyone who's had a bariatric surgery, might have to wait six months just to get that. Or if I say, oh, I want to add 20 data elements to the cancer database, might have to wait a couple of months just to get that. And then again with unstructured data, there isn't any integration of images and notes. People have image and note data. Everything's very siloed. You've got your structured EHR data, then you have all your notes. You might have all of your radiographic images, for example. There's not a lot of good linkage across those database, so it makes it hard to do multimodal work. That's why you might have seen in medicine, a lot of people really stick into the NLP, the new computer vision, pretty good modeling of structured EHR data. And this is an example of unstructured data. So this is a note for trauma transformations. Patients in a motor vehicle crash. It's all de-identified. And this patient, this note, how are we going to look at this note and extract from this patient-received adherence with an evidence-based practice? How do we integrate unstructured data into a database that we can use for AI in future There's two ways to do it. The first thing is that you can integrate the identifier for the note into your structured database. So for example, every note has an encounter ID. You can integrate those into the database, as soon as those will be ultimately did. But also we can take the features that are generated from natural language processing. We can integrate those features into our structured database but ultimately use those in downstream models. So let's say that you want to look through here and identify where there is a commission or omission. Well, you have two ways you can do it. So we can manually review the note and then decide where a certain evidence-based practice is delivered. So for example, if someone's heart stopped, did they get CPR? If someone couldn't protect their area and couldn't breathe, did a breathing tube go in? Or you can leverage AI natural language processing to read the note and analyze that note. So that's what we did, the latter option. So we made a list of indications for treatments. So if someone becomes unresponsive, they can't protect their airway, did they get a breathing tube in? Someone's heart stopped, did they get CPR? You also have to make sure that when you're doing this, that you train the algorithms so they can identify semantically similar terms. For someone's heart stopping, if I would say it's encoded or there are impulses or the heart stopped, those are all the same thing. And what we did was we extracted about, we took a whole set of notes we extracted a couple hundred procedures and extracted over a thousand indications or procedures. And typically when something happens like some heart stops, that's gonna be in the note a couple of times. That's why you'll see that indications for procedures is more than the actual procedure itself. What we did is we were able to take those notes and summarize them into these features and these concepts that are extracted out of the note. So a patient came in, they decreased level of consciousness. They were then unresponsive. If someone's unresponsive, they're supposed to get intubated. So right here, you see an air of admission right here in this patient. This is a real patient. They have lungs that have diminished, breast sense bilaterally, shallow breathing problem, hip retention, attempted IV access. There are two appropriate treatments there. This was all automated. This wasn't someone's gonna have to go through there and then decide if it's appropriate or not. Patient then had low blood pressure. They got blood, so that was appropriate, but they didn't get something called TXA. So just in this one example, you can see there are two areas of admission. And the other thing with this, this TXA, for example, that came out of the crash two trial that was published in 2010. This is seven years later. This is 2017, 2018. So you can see even seven, eight years later, a very well-known international trial. This standard repair is still not being followed. And what this did is it allowed us to, in an automated fashion, what we used to have to do in the manual process, we can now, within minutes of a note being done, run it through the NLP pipeline and pull out. Did someone get appropriate treatments? Did they get narrow admission, which is that middle column, or narrow admission? Did they get a treatment that they shouldn't have got? And then we've done this at the Royal Health Hospital, evaluated in the Linus Health Hospital's system. And then that second way that you could integrate NLP features from unstructured note into your databases. These then become features that can be added into your database. So for example, for each patient in there, you can have, do they get an appropriate airway or not? Do they get an appropriate IV, et cetera? So you can link the actual patient records to the note, and then you can also pull elements out from notes and put them into your database in a structured way. We've done this with COVID as well. So we've developed COVID pipelines that can read notes within 24 hours than being done and extract concepts such as the patient of diarrhea. Do they have a shortest of breath? Do they have difficulty breathing, et cetera? And this is live in NLP area. As soon as the patient comes in the hospital, these pipelines analyze the note and then pull out the symptoms that a patient has and then add it to our research database as structured elements. You have diarrhea, yes, no, they have it, et cetera. The reason we did that was because when COVID started early on, there was a lot of a thought that the symptoms that a patient develops are prognostically related to their outcomes. So I keep saying diarrhea, that was thought to be the worst symptom that you could possibly have if you got COVID. Then we've used these to publish multiple papers, including prognostic algorithms for patients as well. So here's our current data strategy. So we have, we collect, we build this data strategy on all patients that come to the ED, get admitted into our hospital or get any surgery or procedures. So it's over 3 million patients. So we have it linked to the state death certificate database. We also have the state immunization database, clinical trial, biologic data. So patients who do a clinical trial that can send to allow their biologic data to be used for future research, that gets integrated into the database. We integrate proteomics, botanolomics, epigenetics, et cetera. And that's probably one of the biggest growing areas. I've hired two Ph.D. faculty bioinformaticists and four Ph.D. biostatistics graduate students in the last year and a half or so. This is where we're really growing the most, I think. I'm electronic health record data. Obviously, it's integrated bioinformatics, home medicine, things like that. Live AI algorithms. So any algorithm that's deployed in the system, we're pulling out, what was that algorithm's prediction? Maybe it's a prognostic algorithm or diagnostic algorithm, et cetera. Patient generated data. This is an area that's also growing. So we're sending a lot of patients home with these mobile platforms that might send them surveys every other day and then they go home with, how are you doing? How's your pain? If you're taking your medicines, do you understand your meds? We're getting all that information back and integrating that. I mentioned note radiology data. So that's linkages to the actual notes or images, which sounds obvious, but it wasn't like that before. It was kind of a pain if I wanted to do an NLP task that I got to define in cohort. It was really hard to build that cohort, as well as pull features from those notes, either radiographic features or NLP features. Device data like EKG and TETX, those are all integrated. Manually extracted disease registries are integrated. And that's great. There's your gold standard for a lot of your AI tasks right there. So you don't have to sit down and reinvent the wheel and do a lot of manual extraction. Environmental and public health data, that's an area that we've been partnering with the school of public health and integrating data with. So let's say you're looking at biologic predictors of lung cancer, but you may want to also include dissipation lived in a high pollution, higher pollution area, for example, cost and reimbursement data and the social and terms of health. The database is set up in two views. There's a flattened version. So when we pull the labs in or the vitals in, we flatten it and we pull in, what was the maximum part rate that someone had for every hour, for example? And then we have a view that's a bit more granular. This is every part rate. I'll tell you 90% of people use the flattened version. And that powers a research grade database, any QI dashboard that we need, and then operations and QI database, QS quality improvement. So I leveraged this data infrastructure to try to identify where should we focus one of our first tasks. We identified that rib fractures we are a major outlier for. First of all, it's a common thing. It's about 10% of our trauma volume and our entire system is one of the larger trauma centers or trauma systems in the country. We've over 5,000 traumas. And our complication rate is pretty poor. For patients who come in with rib fractures, about 17.5% of them will develop respiratory failure and have to be admitted to the ICU. And 15% of them will be admitted to the floor and then also really get transferred to the ICU. And we're a national outlier for complications and serious complications. So the first thing that we did was we generated a planning team. Surround yourself in a multidisciplinary team. So we involve the system director of trauma, the various specialties that treat rib fracture patients, our IT team, nursing, respiratory therapy. We reviewed the evidence-based we came up with an evidence-based bundled treatment for patients. And then we piloted it. Always pilot QI intervention. So we worked at North Memorial Hospital. We piloted this intervention at North Memorial Hospital. We integrated the evidence-based practices and delivered it as a clinical decision support system. So what's that mean?   This patient should get an epidural, et cetera. So these are just based on nudges to try to remind someone of what the evidence-based practices are. We did a before and after analysis. There's more rigorous analyses you can do, like randomized control trials, but those are very costly. And in most QI, what most people do is these pre-prose analysis. They're very practical. We looked at outcomes, clinical outcomes and implementation outcomes. So the first thing we said is when we rolled this out, how's adherence with this? As might've just been beginner's luck because we had 80% adherence with this intervention when first deployed it, which is really good. I've never had anything since then that high on the first month. And then after about seven or so months, we reached 100% adherence with this decision support intervention. The next thing that we wanted to look at was behavior change. All we're trying to do with decision support is change providers behavior. That's why we're integrating AI. That's why we're making mobile apps. That's why we're giving these nudges. We want to change providers behavior to deliver evidence-based practice. And when we look at reasons why people will accept technology intervention like a decision support system, there's a couple of different frameworks or theories regarding why someone will follow something. There's the technology acceptance model. And then there's also, we typically use the unified theory of acceptance and use of technology or UT-AT. UT-AT poses that someone will use technology intervention if there's performance expectancy. So if they believe it, in this case, if they believe it is going to improve clinical outcomes, if there's effort expectancy, so it's easy to use. If there's social influence, if the bosses and peers want them to use it. And then there has to be facilitating conditions. So the IT department has to be able to actually deploy this. It has to actually work in the system. This is a part that I didn't think I would be doing when I started as a QI researcher and I think I'd have to be doing surveys and ultimately qualitative research. And this is again, 90% of the stuff that we do is trying to change people's behavior. This is the survey that we delivered. As you can see, we had really high performance expectancy. See what it does, yeah. Effort expectancy scores also had really high scores, social influence, a little bit of a drop in facilitating conditions and everyone intended to use the decision support intervention. This is an example of successful pilot study. When we looked at outcomes, we saw trend towards reduced pulmonary complications. We saw significant reduction in hospital aid this day. So we've scaled it, or we've implemented this so far at Warrior Hospital. How do we take that and scale that to the University of Minnesota system? It's 11 hospitals. It's a couple of hospitals in large cities like St. Paul and Minneapolis, as well as hospitals up in the Canadian border. So we used a model that was developed on a UNC Chapel Hill called UIS-AD model. Essentially what that means is you build a prototype first. We based our prototype off our pilot study. We then did qualitative interviews of end users and trauma surgeons and nurses and had them review the prototype on paper and tell us what they thought about it. What would work, what would they like to see? How would it work better? And do a system feasibility analysis. We then adaptively redesign that system. We then build it or program it into our computer system or EHR. We then do usability evaluation. We do usability evaluation a couple of different ways. We do expert-driven usability evaluations of the heuristics or cognitive walkthroughs with usability expert or end user evaluations. You could have an end user review this and walk through simulated patients, for example. We do both. We find that identifies the most usability issues. So we do a combination. We adaptively redesign it as well. Then we implement it at one site, four months later, two more sites, four months later, the entire system. Then we evaluate outcomes. So the prototype, this is an example of the prototype. It's important to look at the structure as well. It's a very linear structure. And you're gonna see that this is, we're trying to change this when we talk about the future of a lot of where integrating AI in. But right now healthcare is very, it's very algorithmized. It's very much like a big if-then-else state. If this, do this, if not, do that, et cetera. So a patient comes into the ED. They get restratified. They get admission order set. There's care plan. There's an early warning system in the system ends. The early warning system identifies if the patient's developing complications while they're admitted. We did qualitative interviews and we did thematic analysis. We identified eight themes. And this really changed the system that ultimately got deployed. Providers did not like alert fatigue. They didn't like all of these pop-ups interrupting their workflow. This bothers some of them. So we drastically got rid of that. Automation. The one that we didn't expect it wasn't described as well in the literature. They wanted this thing to be almost fully automated. For example, when you order a medicine called Toral to get an IV version of that pill, you need to dose that based on some of this kidney function. That's manual.  Why can't you just automate that? When you restratify someone, they need to look at this table and look at, you know, is the person, how many rects they have, how old are they? Are they mild or are they moderate risk? Are they at severe risk of getting complications? Why can't you automate that? So you added a lot of automation to this system. We reduced redundancy. We had minimalistic design. We had to make it malleable. So there's, the different hospitals have different resources. The hospital in Canadian order has a lot less resources than a hospital in St. Paul or Minneapolis, for example. So you had to make it malleable to the institutional resources. They wanted to be comprehensive. And this was a big, this was, this results in a lot of change. We kind of stopped when someone got admitted to the hospital. And now what we did is we were actually tracking how patients are dealing with it to 30 days after they went home. Minimizing errors and evidence-based integration. They want an integration of the evidence-based into the decision support system. So that you could see, I'm going to say it's recommending, you know, do X, Y and Z. You can click on it and actually see where that evidence that recommendation is coming from. So then we adaptively redesigned our system. This is the final system that was developed. We integrated automated risk stratification. We integrate a lot of evidence-based best practice advisories that went to the providers. We integrated a discharge order set, which also involves a little tool that patients take home. Those are mobile applications that they download onto their phone. We did a usability evaluation. We did a dual usability evaluation, as mentioned. We found 79 usability issues. The reason that we always do expert user-driven and end user-driven is because you look only 11% of issues were picked up by both methods. So there's a lot of benefit of doing, at least we find a lot of benefit in doing the combination. We improved the system after the usability evaluation that we implemented. What we saw was that there was a significant reduction in unplanned transfers of ICU. We saw reduced mortality. You can see the capillire survival curves. The trend towards reduced need for mechanical ventilation. We drilled in and looked at the actual evidence-based practices and our accident blockade, which says we're at the mural, and admission evidence-based bundle and following the recommendations from the early warning system. We saw the patients that received adherence with those three practices had the best outcomes. So are we done there? And can we do any better? The answers were not done there. That's really just the beginning. That practice, again, was very much, everyone got the same pathway, but people are different. So how can we tailor care or individualize care instead of just giving everyone the same pathway? So the current state, you have to do a randomized control trial to generate the highest level of evidence currently in medicine. So you look at the practice of like admitting someone to the ICU. Current evidence is if someone's got three or more ripped fractures, they're older than 65 years old, they should be admitted to the ICU. Okay, great. Well, what if someone's 90 with one ripped fracture? Or if they're 40 with six ripped fractures? And then think about additional dimensionality in this, right? What if they're a smoker? What if they got COPD? They're throwing a home oxygen. How to x-ray findings in form of treatments that people get? And then, sure, you could do a randomized trial now in a population of 90 with one ripped fracture. You could do a randomized trial of people that are 40 with six ripped fractures and so forth. But it's not feasible. Clinical trials can not possibly evaluate every permutation to identify who benefits from a certain treatment. That's why I personally believe that AI is the future of evidence generation in medicine. You could, for example, let's say that people that have a high probability of dying should go to the ICU. You couldn't generate a prognostic AI model that predicts who was the probability of dying and identify an optimal threshold when ICU admission improves patient outcomes. So it kind of gets us back to our database. If we had a prior we started this project only collected two, three, 400 data elements that we needed to kind of run this project and look at outcomes and add like 30 compounding variables, et cetera in there. We wouldn't have the data that we have to start training it out or at least to start individualizing care. And fortunately, the database we have has over 10,000 processed or flattened variables that we can use. And this is the future state. This is what we are building towards. We're building towards a person comes into the emergency department. The second they hit the emergency department, diagnosing AI algorithms start to try to predict what does this person have? If they get an X-ray, a computer vision algorithms reading that X-ray trying to predict in this case if they have a rib fracture or not. You decide, okay, now we got to admit this person into the hospital. Well, a lot of decisions need to be made. All right. Think about the different evidence-based practices and rib fractures. Do they need to go to the ICU or not? Do they need to get an epidural? Do they need to get surgery? Do they need to get IV tour at all, et cetera? Once they're admitted in the hospital, we need to monitor that they're gonna pick up an adverse event or develop a complication. And that risk is cumulative. So if we put a fully catheter in someone, every day there's a 6% probability that person's picking up a ureteria tract infection. And we can look at, does the person have a full in? How many days has it been? As well as other elements like their labs and vitals, et cetera, and predict the probability that someone's gonna develop a complication in real time. And it allows us to turn medicine from reactive, which is really how we are right now, to more preemptive. We could potentially intervene earlier or pick things up before a major issue is pretty obvious about, yeah, they got a bad pneumonia. Let's give them antibiotics. Once someone goes home, it doesn't end. You gotta predict who's gonna be re-invited into the hospital. So we can build a baseline model based on their discharge data elements, what methods they go in, how long in the hospital they get complications, et cetera. But by getting data from patients every couple of days, we can continually look at their longitudinal risk of being re-invited. And then once it goes over a certain threshold, we can deploy an intervention while they're still at home. We could send a nurse to their house or we could have them follow the primary care doctor the next day. Instead of them coming into the emergency department two or three days later and they're pretty sick and then having to get re-invited into the hospital. So we're actively building or in the process of validating these models. I'm gonna show two of these models right now. So the first one to talk about is our computer vision diagnostic algorithm. We've done diagnostic algorithms before in COVID-19 and I'll show some of that as well. So patient comes in the emergency department, they get an x-ray. We build an AI system that could read the chest x-ray and it's not just the peer vision, but also our integrated structured data and the patient's epic data through labs, vitals, et cetera and predict the probability that that person has a rib fracture. That would also reduce diagnostic error. So previously we've developed an AI algorithm that can identify patient as COVID-19 from their imaging. We also generate heat maps that can show exactly where along that source of infection is or at least why the algorithm is predicting that a patient has COVID. We have a pipeline that will pre-process the image and identify the lung space and then we have another pipeline that will identify outliers and throw away outliers. And then we have a pre-trained neural network test network 21 that can classify patients as either having COVID or not having COVID. We did this in combination with us, Emory, Indiana, University of Florida, Gainesville. We also compared it to radiologists performance. And not only can we build these algorithms, we can actually deploy these algorithms into real care. So that COVID algorithm was deployed into real care. And the way that that works is as soon as a patient gets an x-ray, it shows up in our PAC system. Within five minutes of that x-ray being done, it's pulled out into Epic, that's the electronic health record that we use is Epic Server Meditech. But that algorithm is pulled out, I'm sorry, that x-ray is pulled out. The algorithm runs on it, it exports a flow variable zero to one, and then that gets put back into our electronic health record and then we can trigger decision support based off of that. They've been information card saying this person has a high probability of having COVID, confirm with the test, et cetera. We can also share the basis for why these algorithms are making these predictions. So here's the results of our RIM fracture algorithm which is currently undergoing a external validation at University of Florida, Gainesville and UNC Chapel Hill. So we train the algorithm on about 15,000 ED visits, 4,000 at RIM fractures, so a bit of an imbalance. But real care isn't 50-50, about 50% of patients don't have RIM fractures. Test, validation, we train a couple of different models. We are heavily moving away from only building uni-modal models. Every model that we build, at least in the program for clinical AI, we're constantly asking, can LNLP be integrated? Can structured data be integrating? We're not at the point of integrating biologic data, but that's next too. So an image only model has performance of AU rocket 0.77, 0.8, see the sensitivity specificity at point score. EHR only model actually does pretty well, which makes sense if you think about, you know how patient presence is where fractured the elements that were included. That a multimodal model is barely, probably not even significantly better, but multimodal model is essentially about the same benefit of the multimodal model. You'd also get that heat map showing on the lung of where that actual fracture is. So it does have some benefit, at least explainability versus just an EHR only model. Another algorithm building is a, we're validating some predictive algorithm that can identify who's developed a blood clot. And this is being externally validated by Duke University right now. So Duke and Vanderbilt previously published back about nine years ago, an NLP algorithm which can identify a VTE essentially blood clot, like deep based on those are DVT or pulmonary embolism, a VTE phenotype. And you know, one thing is don't reinvent the wheel if someone's already built it, let's take what they've done, let's use their lexicon to optimize on it. So we've developed a neural network which also uses EHR features, labs, bioels, et cetera, as well as natural language processing of the notes and has an AROC of about 92.8%. And you can see the precision recall and F1 scores for that model as well. And no institution could do it alone.  That's critical in the AI space, I'm sure you're all pretty aware of that. But one of the things that we've had to decide on is how are we gonna collaborate with these other institutions? You know, we have two options. One option is we centralize all of our data. So everyone says, all right, let's share all of our data, let's do a centralized solution then we can train models and all the data will pull together. When healthcare has a lot of issues, we run into issues, we try to do that. We run into issues with privacy, data sharing, bandwidth, we were talking about petabytes of data. Just our images alone are five petabytes. We also run into issues of every institution wanting to be the one who held all the data. So that turns into some interesting debates or discussions. So what we settled on was a federated approach. Everyone keeps throwing data and we use a federated learning platform that allows us to send the models externally but the data stays behind everyone's firewalls. And I'm gonna talk a little bit about this platform that we use. So in 2020, we launched the healthcare federated learning collaborative. It's us, Indiana, Emory and the University of Florida, Gainesville running three additional institutions on currently is in collaboration with NVIDIA and it's being funded by Cisco. And essentially the way that it works is all of our data stays in our system or our therapy system behind our firewall where we have our GPU capabilities we use Azure. We have a central server with the actual model six. So I'll show it a little better on the next slide. And then each institution has their data themselves. So you can see it here, each institution, let's just say division only, each institution has their X-rays, for example. We have a central server that has the model. The model gets sent out, it goes to each institution, trains on each institution's data, learns the hyperparameters, sends the hyperparameters back centrally and the process continues, the hyperparameters are also newly and ultimately the general model is trained. And we have under review a paper that shows that using a federated approach is comparable to using a pooled analysis for at least computer vision in this case. So we also had a hospital in Spain collaborate as well. So what's the next step? Once we build this future state, how are we gonna just throw it into practice? You can't just build AI and just throw it into practice. You gotta validate it, you gotta make sure it's equitable. You gotta make sure that it actually improves care. You gotta make sure over time that there's not long drift. So the way that we're gonna do this is a multi-center randomized control trial. And this is currently in the works, we're collaborating with the National Trauma Society to develop this, but we're gonna randomize a set of hospitals in the US to give the AI-based care for rib fracture patients versus other hospitals to give just the standard care, which gives a bunch of the skip that I'll stop. Everyone gets the same care or very similar care. And then we're gonna see which one out reforms. And the way the AI works is you don't have to follow the AI. The AI is providing a recommendation. So if you're a provider looking at the patient emergency department, you're like, should I admit them to the ICU or not? The model will give you, you know, based on historic data, if you admit this patient to the ICU, their probability of dying is 5%. If you admit them to the floor, the probability of dying is 8% or something. There's a confidence intervals, et cetera. There's the performance of the models when you decide. That's kind of how this would work. So let's not forget where we started. Starting with the five types of medical error. I personally believe that AI-enabled clinical decision support can decrease preventable deaths associated with incorrect treatment, treatments of commission or errors of commission, non-adherent treatment and misdiagnosis. And I really think that there's variation in care. Healthcare is an art, medicine is an art, and people do different treatments. Someone might be like, you know, in my experience, this treatment really helps with fracture patients, for example, and maybe no one's studied it before. AI can leverage that variation of care to possibly identify novel evidence-based practices. So in summary, I think that a forward-thinking data strategy is critical. You really need to create a 360-degree view of the patient. We're constantly thinking about what other data elements we integrate into this database. Integrative or multi-omics expertise is increasingly important. We look at the different types of data. There's medical imaging data, radiology images, ultrasound, robotic surgeries. We're now obtaining consent from patients to record and save videos of a patient's surgery for downstream robotic automation tests and ask you to do videos, EKG analysis. There's natural language processing of notes, and there's also evidence in deciding to talk about this piece yet, but so you have NLP of notes, and then there's biologic omics analysis. And again, I mentioned, we're starting to, this is probably the area that we're growing in Vegas. You know, I've added six staff on, just to tackle this piece. And the future, the area where I think the future is is right in the middle. We need people that can take all of these, they're a jack-of-all-trade that understand enough in these different pieces that can help build models that can put all of this together. Plans and career trajectories will change. So, RASA, don't fear learning new things. I never heard of federated learning three years ago. Surround yourself with experts in multi-disciplinary training. So when we, when one of the health informaticists that we were working with brought forward, hey, why don't we, when we realized centralized solution wouldn't work, why don't we do federated learning? So, okay, sat down and Googled, you know, the University of Minnesota federated learning and found a professor in computer science engineering who's an expert of that technique and added that person onto our team as well as his grad student. Continually reassess what is new and look for ways to cultivate those technologies. So if you build it, they'll come, we built that federated learning collaborative and now online institutions are reaching out to us. We just presented about it for the CDC. There's the possibility of hospitals in other countries, other continents possibly join the platform. Built bridges, you're gonna need collaborations of industry and external health systems for progress and informatics. And thank you. That'll take any questions. Yes. Yeah, so anything that's done for quality purpose. Could you repeat the question for online? So the question is, what are the medical legal implications of this, right? So when you say medical legal implications, I guess there's a couple of different ways that you could think about it. The first thing is just like as I showed earlier, we do quality monitoring. This happens whether you do an automated 3LP or someone manually reviews a chart. Anything that's under quality is actually not discoverable. So whether it's NLP or a person that reviews a record and determines that an error of commission or omission were done, we actually have a peer review. So in trauma, it's actually mandatory. We have to review every death in trauma and every complication. And then a whole panel of people also have to determine if it was preventable or not. And that's a conference that we do. And all of those decisions are not discoverable. On the AI piece, what if someone dies because of AI? Is I'm not sure exactly which piece you were referring to? I was referring to, let's say, someone there who worked in health practice. Complications happen. And the ways to communicate to the family is honestly, only the error, or at least that's how it should be communicated to the family. All the medical records are discoverable. So they'll be able, regardless of, whether we internally know that an error was made, they'll be able to obtain the records and have a physician review like an obvious error was made here. So, yes. That's a great perspective. I think my question is really seeking a little bit. You showed a couple of cases where you could imagine the some ways of being in Black Fox and the more that they're trained the more experience that they have, the better decisions that they're making. And so some of the tools that you show seem to be sort of monopsying. There are students given some set of the immediate information that you've had, right? But in the last slide, you showed me something that's quite complicated from that sort of scenario that if it seems that it's immigrating and boss have to have, we only know what we have to have. Not only can they have that, but we also can't have that. And so, I mean, that's something that's way from that. But what I want to do is provide some context where we are specifically to the latter because for me, that seems like the whole ground. Right? You're immigrating across all the information and seeing what in any one's life. In that sense, it's kind of- It does. No, absolutely. So I think the first thing is it's kind of two pieces on the continuum, right? Like we're trying to get to the latter and we're not even at the first piece yet. Not even at everyone giving just that algorithm, you know, approach to medicine right now. So a lot of what we're doing with just kind of the decision support system that we've deployed now is just trying to get people up to standard of care. But we're already starting to work on pushing the boundaries to individualize care and leverage all of that rich data to try to give more tailored and generalizable care to people. We aren't at that latter piece. I think that there's a bunch of people that are kind of doing similar work, but we're definitely in the infancy right now. A lot of the AI systems do you Google the different AI tools that exist and have been integrated into practice? They're for like a single situation or a single question like, again, does this person have COVID or does this person have diabetic retinopathy? That was a model that Google and Mayo made, for example, or does this person benefit from this treatment? Those are single point decisions that need to be made along the continuum of care for a patient. We are trying to build a pathway where multiple models can be integrated together to answer those critical decisions, those evidence-based decisions that are possibly in a way that leverages the rich amount of information that we have. I don't think that we're gonna get to the point where we have like an AI system that can help with every single possible small decision that gets made. But if we can at least build AI models around the key decisions, so in rib fracture patients, I've kind of already said that, like, just don't get enough of girl that they know the ICU, are they decompensating, things like that. We can build integrated AI for those in a single use case. I think that would be a step towards the Holy Grail. The Holy Grail system is just as constantly analyzing everything and making all of these additional predictions or recommendations you haven't even thought about. So the goal right now is just to get, is to develop a system that can tailor the seven or eight key decisions that need to be made for a disease process. Yes? So while we balance the amount of work and investment that's needed, upstream versus downstream, so things like approaching a hospital-acquired spectrum, that's one of the areas we have had, you could tackle up to England work, which means the data standardization so that those critical data are captured at the time of the data collection. Or do tackle it as if there was downstream problem, do you spoke quite a bit about machine learning and AI? So try to tap into unstructured data if it's not going to get lots of people working on an area. So how do you best balance and then in Canadian setting there's a specific, well, incentive is something that can't be ignored. So within Canadian setting, currently there's no incentive to use AI-assisted services. I believe that in the US there's a separate veto for AI-assisted services. So there's this compensation and incentive structure that's in place, whereas in Canada we don't have that. So what is the incentive of, trying to get a whole bunch of people working on this AI-assisted care? One is that there's no way to get that compensation. So I find that balance very tricky. So I'm kind of curious how you can deal with that. I think that from the compensation front, we're not doing this, at least my group is, and maybe the health systems like, yeah, please do this so that we can build more for like AI care, for example. We're doing this because we believe that if we can deliver tailored care, that we're gonna further be able to push the needle on those outcomes, that we're gonna be able to reduce mortality, reduce hospital utilization, only give people an epidural, for example, that actually needed an epidural. We've actually found there's a bunch of, there's populations, if we give them an epidural, they should have a longer life this day. We're trying to preserve IC events, only send people IC that need to be there. So the hope is that AI, when deployed, will actually reduce resource utilization and not at the cost of worsening care, but actually be fair. So that's the incentive for us in our system. On the upstream versus downstream, it's kind of more like agile what we're doing. We're kind of doing everything in parallel. So we have all of a team that's in charge of building out this database. That includes our hospital part infection database as well. All of those elements are integrated into this database. And that team includes two ETL people, three analysts and a project manager, and then a lot of clinical subject matter experts. And those people are just in charge of building the database that ultimately is used for all of these downstream projects. Then there's the whole program for AI team, that is building different AI tools and evaluating them and implementing them into care. Then there's everything in between. So we were kind of doing everything in parallel. There isn't kind of a very linear path that we're following. Is that helpful? That makes it a problem? Yeah, it's very helpful. Hold on. So the data team, I think it was about seven people, I just said, so about seven people. The OMIX team is currently at six. Our research team of like physicians, MD, PhDs, PhDs is well over 30 people. So it's a pretty large team. Yes. Let's say you come up with this AI and there's this scenario and wait for the traditional training that they have. They know that they need to give them this medicine. But then the AI predicts that the other medicine and they should give better choice. Yeah. So we do. So like before we deployed the COVID AI algorithm, we actually go through a training program with all the ED providers. We not only make tip sheets to show the monitor performance of what it is. There's also like email blasts and things like that. We also have group meetings where we present it to them. We explain what it's gonna show. We also explain here's the sensitivity, here's the status, here's precision, et cetera. We show how it performs based on the prevalence of disease. So we look at different prevalence of disease when we do the evaluation. So they know, and we also explain all that needs. And so they have an understanding of what the limitations of the model are. Most people, if they disagree with the algorithm, they're gonna do what they think. But the algorithms there, they hadn't even thought of it. It's really helpful if someone's like seeing something like, oh, this is heart failure. It hasn't been finding on chest x-ray. It couldn't confuse with COVID. This is heart failure. This is heart failure. I'm not worried about it. Now it was like 82% chance that they had COVID. They might be like, what?  So that's kind of where it can come in. It's something that they hadn't even thought of. But if they're completely disagreed that they're like, this is the treatment that I need to get. They're not gonna follow the AI. And that's why at the end of the day, when we deploy this clinical trial, we're not gonna force people to follow the AI. Some patients will get hurt. We don't wanna do that. We want to just give them a nod to your say, here's what the model's thinking. This is the basis also for why it's saying that. So with every model, it'll show a heat map for this is what it's looking at. And maybe there's an error for whatever reason. Maybe it's looking at the clavicle or something and saying that this is why it's calling that someone has a rib fracture. Well, they'll know that's not a rib. So I'm not gonna follow this, for example. So it's really important that FDA actually requires it that you show the basis for any AI ovens, whether they're deployed. Yes? Yeah. I have some low level questions about the NLP involved. In particular, when IBM wanted to jeopardy challenge with Watson, they thought they were gonna swoop in to this field and they had solved the problem. Is anything that IBM and Watson done useful to you in particular, their un-structured information management architecture and the C-take system that's built on top of that, which is clinical text extraction, something like that? Short answers, yes. So some of the algorithm that we use back in the motor vehicle crash example, that was an ensemble approach. Sort of an ensemble of C-takes, clamp, metamap, biomedicus. And what it did is identified, it picked the different annotations system that performed best for the task. So let's say that we're extracting like a patient age, each system C-takes, metamap, clamp might have different performance on extracting age. And maybe you'll find that all C-takes is really good at finding someone's age, just based on how this trained or outperformed, et cetera. Then we'll use C-takes for that task, for example. We might find that all biomedicus is the best for this other element. So short answers, yes. We do use C-takes. I'm not a developer, so I can't get two in the weeds, but I kind of know high level what they were doing. So, yes. So in terms of AI strategies that you've mentioned, Dan will be mentioned computer vision. I was curious if you can take it over to reinforcement learning. So this is a class of algorithms that you know, observe these permutations given superhuman performance on games like Go. And Penn Medicine has actually employed this in a system where, you know, they were trying to decide until we patients all have ventilators and sort of observing the decisions that the commissions were making and eventually over time would start to make its own decisions and they realized they were able to, if they were able to listen to the system, that they cut their mean ventilator time down by a half a day as a pre-print available. So just wondering if you've thought at all about reinforcement learning, if you've seen any potential applications for that trauma setting. I've heard the developers and PhDs discuss using reinforcement learning. I don't know if any of the neural networks that they've deployed are reinforcement learning networks, but if you send me an email, I'm happy to put you in touch with them and they can answer that question. In the imaging space, they almost always use convolutional neural networks. Yes. How do these algorithms actually make it to the condition when it comes to what's often being used as a deployment? Yeah, so the way that we deploy it is typically there's two ways you get to play clean decisions for it. If we pass it or to be interrupted, there's a push to kind of move everything to pass it, just if people don't like these pop-ups so learning them and interrupting their workflow. That being said, when we evaluate it, we find that adherence with best practice and optimal care is actually when you use the combination of passive and interrupted, whatever that's worth. But typically these algorithms are done through an interrupted mechanism. So a provider will be evaluating patient in the emergency department and a box will pop up on the side, for example. I'm saying, if this person looks like they have a high probability of having with fractures, for example, acknowledge it, agree, disagree, something like that.  So we had a COVID banner within our electronic health record and it had a lot of COVID specific information. And one of the things that was integrated in there was this COVID prognostic algorithm. So that is, you know, they go in there and they're looking at a patient, they just peek to the left and see, you know, if this person is COVID, you know, what's the probability that they're gonna have a really severe course or not. And that's really important too, if you're in the emergency department, we have metrics or thresholds that say that we've shown that it's safe to send someone home or we should possibly send them to a COVID cohort at hospital. So, yeah. Yes. How much resistance did you see from the doctors in terms of like not wanting to change the algorithm? So I think we lucked out, oh, lucked out. I think that we benefited from a lot of the early work is when we really got up and running and really deploying stuff, it was late 2019, early 2020. I think people who are just very happy to get any tools that they could get to help with COVID. So we actually had really good acceptance with ripped fractures as well, these various tools for having really high acceptance as well. I think that some of it is, we have a very aggressive training program. We have many sessions with the providers. We actually have a whole education team that meets with them. We do booster education sessions, we answering questions. Also very transparent. That's a really important thing when you're doing any of this is being really transparent of like what's rolling out, how things work, why we're doing it, what our outcomes are. And then we also listen to them. So one of the things that triggered the early warning system is pain. And that's actually the national system for like early warning systems called the PIC score, the pain inspiration and cough score. But our internal data show that pain triggered this decision support system like 75% of the time. And when it triggered, it wasn't ever associated with like a worse outcome. And the physicians hated it because I think the thing triggered like 10,000 extra times that were needed. And they said, can you get rid of this? But we took pain out even though the system's called PIC score, so not the PIC score, but so we listened to them and then we also use our own internal data to decide should we move away from practice or not. Other questions? This is an online question. What are you doing to ensure your models and predictions don't recreate existing biases? Example, a long ratio or gender lines in the medical system? Excellent question. So this is a huge area of focus. We have a whole team that looks at equity and fairness of AI models that's led out of the division of biostatistics. Whenever we develop an AI model, we get multiple validations. So the first validation we do is a temporal validation. The second validation that we do is we implement it into the system in the background. We look at how it would have performed and of course we do an external validation. For all of those, we do a couple of different checks for equity. First thing that we do is we do subgroup analyses to look at how would the algorithm perform by race, by ethnicity, by gender, by age groups. Those are the main groups that we look at. And then we use the long test to see if there's any statistically significant differences in the AU rocks and the APRCs, the performances between the different groups. The other thing that we do is we do a McNear test to look if there's any significant differences in sensitivity and specificity across different tests. We also always do a regression algorithm to look at if race, ethnicity, gender, different age groups are independently associated with the scores and the outputs that the model gives. And you can actually look at the radiology AI paper that I showed earlier. We go through a lot of those different analyses that we do. But we're also, our Biostatist Department is also looking at novel ways of defining fairness of AI models. So we'll see what ends up happening with that. There's a large PCORI grant they're submitting. But this is a big area of interest for us. We also have a committee that reviews AI algorithms that are deployed and they evaluate for model drift. So they'll review the model every three or four months and they'll also run similar analyses. So we do the best that we can to evaluate, but I think it's a really important piece. I'm glad that that question was asked. No further questions, please give a big hand. Thank you.


You know in communication. There's nothing we like more than a good process. Everything is a process and it's true there's a transactional model of communication and That explains how the communication process works and we also have a a process where we can explain What happens during perception when we perceive something when we're experiencing something? So I'd like to break that process down for you a little bit so that we can understand What's happening when we engage in perception and when other people engage in perception? so The process of perception starts with selection selection just means you know we're identifying which stimuli we're going to pay attention to There's so many stimuli that were bombarded with at all times that that we can't pay attention to all of them So we have to focus in on which ones we choose to so Selection just says these the stimuli these this is the thing I'm going to pay attention to right now and I'm going to kind of ignore the others as much as possible But this is what's going to receive the bulk of my attention right now. That's selection then we engage in what's called? organization Organization is basically just kind of categorizing things as we as we as we come across things as we experience things You know we see a dog coming towards if we say that's a dog. I've seen a dog before that's what it looks like It's not a car. It's not a you know a desk. It's not we know whatever it's it's that's a dog Right in the same way we see a person coming at us. We know that's a person and a car We know basically what they look like. So we just kind of broadly categorizing things in organization Then interpretation we start to apply and assign meaning to those things right to what we're seeing We start to get a little deeper and apply Some sort of meaning and and first some sort of meaning from those things So when we see that dog coming towards us Interpretation is what says oh, I want to give that dog a hug I love dogs because of my experiences with animals and dogs and I just love dogs I want I want to pet that dog and I want to you know play fetch with it and all that kind of stuff But it's also the same thing that makes somebody else see that dog and think well I've been bitten before I've had bad experiences. This is a you know caution caution caution This is a fight or flight moment. We need to be aware of this dog. It's dangerous right could be dangerous So so we come to these different interpretations based on our own experiences and our own knowledge and our own Feelings about that situation and so we all come to you know This is where we really start to see a lot of divergence in Perception is in interpretations when we apply that specific meaning because we're applying it from our own perspective We also then can engage in negotiation This happens a lot of times in groups Especially if you think about a group memory that you may have like a time You would out with your friends and we're with you know a family story that's been told over and over again Over time that story will change just slightly right a little bit each time and and but you kind of need the the Agreement of the others to do so to say that you know Well, that was funny or no it wasn't funny or you're funny or you're not funny or whatever You need the agreement of other people you have to almost kind of negotiate these things with them negotiate a shared understanding of What's happening then right so we can sometimes engage in that process with other people that as part of this process We're gonna focus on the first three for the for the purposes of this video We're gonna focus on selection organization and interpretation just understanding the negotiation is there at times as well But it's what I'd like to do is to show you a few pictures and run through a little exercise We'll show you how this works because these things happen just like that like the snap of a finger We move through all three of these stages and we get from selection and interpretation just so quickly So let me let me show you some examples here So take a look at this picture for me if you would just take a look at this picture. What are you seeing? What are you seeing? This is one of those pictures if you haven't figured it out already it has a couple of different images and it's kind of a You know it's got those dual images in it. So some of you may be seeing a Young woman what we would describe as a young woman looking off into the distance, right? You're only seeing a part of her face other people may be seeing an older woman and And you can see her face in profile You can see fully half of her face Which looks like an older woman there some of you may be seeing both and that's okay There's nothing wrong with you if you're only seeing one or the other if you're seeing both That's that's not the point, but there there are multiple images here, right? So but so for those of you if you've seen the older woman, let's let's take that as an example Tell me about her What what can you tell me about this older woman? First of all we're calling an older We don't even know that right but but first I mean so selection says pay attention to this image Selection says pay attention to this image. This is what I'm seeing. This is what we're paying attention to right now Organization says that's a person. It's not a dog. It's not a rabbit. It's not a car. It's not a building. It's not a table It's not a desk. It's whatever it's a person, right? I've seen a person before and that's what it looks like So organization says okay. This is a person and then interpretation starts to take over and tell us about that person And and so is this an older woman and what does older mean? The older I get the less old she seems to me because I'm getting closer to her age, right? But so but some people would describe her as an older woman Some people say which when I ask that question some people say she's a witch and then I ask why and they said well She's got a bump on her nose and a big nose and that makes me look like a witch She just looks like a witch and she looks like which some people just say well, she looks sad And looks like she's you know, maybe in the morning. Maybe she's in a funeral But wait and the truth is we don't know any of this, right? This is all interpretation. That's what we're at with interpretation We start to find individual meaning we make meaning out of what we're experiencing what we're seeing what we're hearing what we're tasting So forth we make meaning out of those things through interpretation But everybody's gonna end up maybe on a slightly different page Even if we agree that there's an old woman to young woman We would probably maybe disagree about little, you know details of what we're seeing there Let me give you another example Thank you. Look at this picture. Let's this image You have seen this one before it's getting some of the one of those dual images So if you if you're seeing Both things then you're seeing, you know kind of a cup or or a chalice in the middle there and In the black and then in the the outline of that in the white You can kind of see two faces especially toward the bottom the bottom half of the image There's two faces kind of facing one another and So we can take this either way What you know people some people call a chalice some people call it a cup some people call it a vase And and so, you know, what is it? I mean, I don't know it's in the picture It's a kind of up to interpretation right selection says pay attention to this picture organization says that's a container of some sort and Then interpretation says well, that's a chalice obviously It's you know, it's like out of Arthurian legend or you know something like that or it's a vase If you're florist, maybe you see a vase if you're a history buff, maybe you see a chalice for old days You know just depends on the person we get into interpretation or we get into those faces What are those faces doing? Are they getting ready to kiss or they're getting ready to fight? Are they just staring at each other or they you know, is it to men or is it to women? Is it man or woman? Is it we don't know I don't know but interpretation takes over and where our interpretation goes Leads us down these roads based on our experiences and our our own history and our knowledge and so forth Well, I'll come to a slightly different interpretation probably each of these things Okay, those were some silly examples. Let me just show you some pictures real quick actual pictures of people Okay, I'm gonna go through these fairly quickly. I just want you to take a quick glance and And and see what you can see about them just gather some general impressions, you know So what do we see there What are we seeing? So the first three we kind of you know fairly well dressed right first three were fairly well dressed as opposed to this last one The last person here gets a lot of comments about you know, it kind of looks homeless Some people say it looks like a gym teacher I get that but most people say yeah He kind of looks a little Little rough a little disheveled, you know, so if you saw this person How would you react differently than if you saw those other people that I showed you or the first three people that were dressed in You know suits or nice outfits How would you if you saw them walking down the street toward you or you were walking toward them? How would you react differently? Would you react differently at all? and you know, so What will we do? Well, it may surprise you to know and so we may figured that or may have seen this already But the first person that you saw and this person are actually the same person That they they brought in this is actual He was a homeless person a homeless man and they brought him in and and you know clean him up a little bit groomed him and Combed his hair and and gave him a suit to put on to kind of make a point You know the longer I leave this picture up there the more you'll begin to see that There are some ways you could spot that and maybe you did spot that the first time around um, you know to Would be able to to see that maybe this picture wasn't quite right but the point is our our You know through selection Organization to go interpretation then you know leads us to these different impressions these people selection said pay attention to these pictures Organization said well, that's a person these are people. I'm showing you pictures of people But then interpretation said okay, that's a business person. That's a business person This is a homeless person and what you know, what does that mean to you? For some of us it means, you know Across the street because we don't want to get hassled and we're worried about safety and things like that For other people it means, you know, you get out your wallet and you give them something or you offer to Bring them some food or something and just you know and do what you can Some people just keep walking don't do either. You don't cross the street, but you don't stop either You just keep going It just depends on the person depends on where you're at in life depends on your interpretation of the situation Your comfort level all that kind of stuff. So um, so we go through the same thing with people though selection organization interpretation And it does happen just like that just as quick as you can snap your fingers We move through all three of those Those parts of the process selection organization and interpretation I mentioned before that there's you know, those are the steps selection organization interpretation negotiation And uh, and you know, if we were walking down the street together, then we could negotiate kind of what you know What that experience was what that person was how we should have handled it and all that kind of stuff, but um Again the first three are what we're focusing on in this video And uh, and this is something we do, you know millions of times a day probably when we're perceiving things every time We engage in perception But remember so is somebody else and their interpretation is going to be based on Their own experiences their own life their own knowledge their own all of that kind of stuff, right? So be aware of that that interpretation may vary greatly depending on your frame of reference depending on your background If you have questions about the perception process or anything else related to perception Please feel free to email me I'd be happy to answer any questions that I came in that way in the meantime I hope this gives you kind of a new perspective or understanding on what happens when we perceive And especially the understanding that when we get to that last section that interpretation Things are going to vary based on on on your own outlook your own history your own frame of reference then so Take that into consideration the next time you wonder Well, that's not how I see it. How could that person possibly come to that different of an interpretation? That's because it's all part of the perception process