You're witnessing a miracle unicorn event right now because I'm not supposed to be here. But here I am. Hey.  We have slides. Okay. So let me just tell you a little bit of the history of DC 101. It's gone up, it's gone down, it's changed. But really the idea is when we created DEF CON or when I created DEF CON, everybody just kind of learned from each other and we've gotten so large and we've gotten so many sort of myths and stories that we want to assemble a team here to sort of explain what's real, what's not, how to get the most out of the con, what we're trying to do with the con, what we're not trying to do. And you'll hear this phrase over and over at the con and it's kind of cliche but it's sort of like the con is what you make of it. And what we're trying to say with that is look around, like how many thousands of people are here. You'll get lost in a sea of people unless you find a way to make the con your own experience. And we try really hard to do that. So for example you'll notice you'll be like in a room of thousands of people and you're like how am I going to, what am I going to do? Oh I like physical security, oh I could do tamper evident contest, I could do lock picking, okay lock picking, oh I think I want to try low security locks. And next thing you know you're at a table with ten people picking locks, right? So you went from like tens of thousands of people to ten. And we try really hard to give you all these sort of like off ramp experiences where you can go find other people that are interested in what you're interested in.  And so if you expect the con to come to you it might not work out so well for you but if you just show a little interest and get into a couple of these off ramps you'll meet amazing people. By show of hands who was in line con? Yeah, okay were you scared? Did it turn out okay though? People were probably pretty cool to you in line con, right? That's like the earliest moment of socialization, right? It's okay to ask questions and it's okay to make new friends. So that's kind of what's going on with line con. Now it wasn't planned that way, it just happened because you had to sit in the line con for so long. But in the end it's part of what it means to sort of attend def con. So with this panel we mix it up every year but we want to show you how different people have experienced the con and their advice. There is no one magic perspective, right? It's so large. Around def con 3 I think at first I was really angry with myself because people were coming up to me at my own conference and telling me about things I did not see. And I felt like I'm missing out, like what is the point? All these people are having these great experiences and I'm hearing about it second hand. And then at def con 4 I'm like oh no, that's a good thing, right? More of that. And so now sometimes I'll just secretly scroll on Twitter and I'll be like awesome, that was cool. I have no idea what just happened, you know? And I think you have to sort of take that approach. Okay, so I want to hand it off to Nikita who def con would probably not happen without her organizational capabilities. So with that, Nikita. Thanks, boss. Yeah, so I just wanted to introduce some of the people that are on the panel. Actually we talked about maybe them introducing themselves. So we're going to start with... Hi, I'm Mar. Okay, hi. You've got to be really close to the mic. Hi, I'm Mar. I am the badge designer this year and I do art around con. And you can see it in the hallways and on the signage and stuff. Hey, I'm Kirsten Renner. Everyone calls me Krenner. I had a cooler handle back in the day but didn't stick. I'm mostly known for helping start the car hacking village. This is our ninth year. Good evening. My name is Dilo and I'm the chief of staff for the SOC. Hi, I'm Five Penny and I'm the chief of staff for the SOC. Five Penny and I'm the vendor lead. Hi, I'm Megan Wu and I am the DEF CON workshops lead and also in charge of DEF CON workshops CFP. Thank you guys. I said you were the merch lead and I heard and I'm a liability. Okay. Mar, would you like to come up and talk a little bit about the badges in the art? All right. Hi. Okay. So, like I said, I'm Mar. I did the badges this year. So I just want to talk to you a little bit about how you can interact with them and what they're all about. Okay. So, okay. So the badges, if you've noticed the shape of them, there's a difference between the inhuman and the human badges.  The reason for this is they're penrose tiles.  They're an example of aperiodic tiling. So in physical space you can actually put them together into like a big mosaic. So I wanted people to interact that way with each other with the badges. So that's in physical space. If you look on the back of your badge, there's a URL there, it's spux.art. And if you take the little insert that's inside your badge pack and type in the UUID into the website there, there's a huge mosaic that's growing right now. People are doing it and every time we add a new person it just expands. So we're all making this huge collective piece of art together. And hopefully by the end of the con it's massive and beautiful. We'll see what happens. So once you place your tile, you can place it again as many times as you want. So you can get your buddies together and create actual art instead of just sticking it somewhere. But we'll see what happens. Okay, so what's next? Oh, okay, so the lanyard puzzle. I don't have a lot to tell you yet about the lanyard puzzle. But if you look, there's some characters on your lanyards. There's different characters on the different lanyards. So try to put it together and see what you come up with. Okay, and yeah, so overall what I'm trying to do with the badge here is just endless customization. So you have this chamber on this type of badge, even on the other type of badge, if you didn't get one of these thick plastic badges, there's a little sleeve. And the inserts will also fit into that little sleeve. So I pre-released the specs for this little thing. I'm colliding a shard or an insert, call it whatever you like. Add on. Oops, let me drop it on the floor real quick. It's gone forever. No, I found it. It's right here. So if you look on the back of a program, there's a little pop-out insert. So you don't have to create a PCB and be all glowy at con. The ones that I've seen are really amazing and you guys are super creative. But you can draw your own. You can cut it from acrylic. Really anything you want to show off who you are at con. Okay, and then on that note, the last thing I want to talk about is galleries and self-expression. So there's so much art that we see in the hallways and so much creativity. And hackers and artists are just so similarly brained in a lot of really interesting ways. And I think that we should be seeing more of that and give it more of a dedicated space. So we've been talking about having like a more dedicated gallery space for the kind of art that you're making in coming years. So keep an eye out for that. And that's all I got for you. I'm sorry I can't stay for the panel, but thank you so much and I hope you have a blast. Okay, hey, I'm Kay Renner. I'm going to tell you a funny story and I'm going to roll into how that story, you can take it however you want on how you may customize your experience and kind of enrich and enhance your engagements, everyone that you're meeting, whether it's at a talk or in a line or at a village, some of the best engagements are going to be just in the hallway, how you might not do what I did unless you want to have an interesting story 15 years later. And then I'm going to give you a little peek behind the curtain into Village Ops. And then you can kind of imagine when I saw name a couple things that kind of are involved in setting up a village and I'll forget a whole bunch of things. So multiply it in your head and then imagine that there's 40 some odd villages. So I'm telling you about one and then multiply that time is 50 and then whatever the appropriate multiplier is as you compare all the things I've got going in my head compared to what I don't know Nikita has going in her head when people don't even let her walk down the hallway. So there we go. So here goes my story. The person who it's about promised to pretend they remember. I hope they do actually. So I'm getting on the plane. Somebody handed me some books and they were like, you should read this. And one of them was Art of Intrusion. And I was like, what am I doing? What does hacker mean? What is DEF CON? And I'm doing my research. And I actually read hackers for dummies. And I'm looking on websites and now don't boo me. At the time I was a recruiter. I was given a list of people to recruit. And the first rule is, well, first of all, raise your hand if you've never been here. It's frigging awesome. Welcome, welcome. OK, so someone said what you're going to do is you're going to connect to some people that have done this before. And those people will kind of help guide you through and everything. So that's my first tip for you. If you're following Ray Redacted's redacted hunt, flag five is to remember two things I said, all right? So connect to someone who's been here before to kind of guide you through stuff. They'll let us know if it's an actual emergency. In the meantime, I'll talk faster. All right, let's go. All right, so fast forward. I met some people who were amazing. I was very fortunate they kept bringing me to cool places. And I was just kind of soaking it all in. And I end up, DEF CON was at Rio at the time. This was DEF CON, I want to say 18. I highly recommend, by the way, going on YouTube and watching Kristen Padgett's Extreme RFID, Extreme Distance RFID demo. So that demo was happening on the 30th floor of the Rio. You actually remember? OK, all right. And then I just ended up in the room, right? So I'm looking around and it's exciting. There's camera crews and everything's fun. And I see two people that look familiar. And I'm like, maybe these were people I was supposed to recruit or something. I don't know, but they look familiar. So I just walk right up to them and go, hi, I'm Kirsten. You look familiar. And it was you and you were talking to Dan. And you were very polite about it. You were just kind of like, I don't know if you probably don't remember. He goes like this, perfect example. And then they just went back to their conversation. So I'm glad I added to your content there. But so just real quick, so that's funny. That's cute. Gave you something to talk about. But I could name a whole bunch of people.  I could just say their first name.  And I'm not going to go down a whole list of people because I didn't get permission from them all. I'm not speaking on their behalf. This is my opinion, but I did get permission from a couple. So if you see Jason, you expect a hug. And he's kind of obliged to give you one, right? That's a thing. And if you see Katie's beautiful pink hair, you want to pick her brain. I can tell you with 15 years of experience, everyone wants to do that. And she's very generous of her time. This is a community united by a passion and a love for teaching and learning. So everyone that did the research to build the content, to give the talks, they're constantly trying to build content for you. And they're very generous of their times. And the bulk of this is on the backs of volunteers, right? So you're going to notice things that are broken or wrong in your experience. I kind of recommend, well, letting us know. We need to know if it's broken or bad. And then you could offer to help. And then you could start becoming a contributor, right? You can start becoming a part of the community by offering your ideas and your suggestions and things like that. I keep thinking it's like I said a code word, and it's like, you got a point.   I am winning. All right, so that's about it. I don't know if this is in the Big DJO slide. Still investigating? Already. OK, and then the last thing I'm going to say is everyone you're talking to, make sure you take a moment to look around you and meet people.  Could be your next best friend. One day, I accidentally met the CEO of my next company. Pretty cool, huh? So you're surrounded by greatness, and enjoy your show. I know you. Hi, everyone. My name is Dilo. And like I said, I'm the chief of staff for the SOC department. Here with DEF CON. I've been with the SOC for 13 years this year. In my real life, I'm a teacher. So I believe that you should be able to read the content without me reading it to you. I'm just going to add so that you get value add instead of being talked down to. So the 3, 2, 1 rule, these are guidelines that we ask you to follow. Three hours of sleep a day, two meals, and a shower. Friends, this is Vegas, and it's hot. Please shower and use deodorant, and we love you. Also, because we're in and out of air-conditioned rooms, we're in and out of the heat. Concrete is real. We all go home with the chesty cough, and it's just gross. How can we work against that? Follow the 3, 2, 1 rule, and definitely take those showers. Wash your hands. Use sanitizer, hand sanitizer frequently. And then also, maybe limit handshakes. Maybe do an elbow pump or a fist pump or something. I work with kids, so we do toe taps. You know, keep it fun and maybe not transmit so many germs. Do I just call it an additional? I also talk really fast, so sorry. We also have this safety third rule, so know your boundaries, know what works for you. We are in the desert, but the SOC is really the Department of Fun Enforcement. So we want to make sure that you have fun, and in order to do that, we need to ask you to do a couple things, hydrate, hydrate, hydrate. And also, drink some electrolytes. If you're gonna drink some alcohol, it is DEF CON, people did shots. Follow that with some water, alternate back and forth. Make good choices. And then also, try not to put yourself in danger. If you find yourself in a situation, it happens. Look for a red shirt. All goons wear red shirts, but not all red shirts are sock goons, so if you find yourself in a situation, ask a red shirt to put you in contact with a sock goon. We will gladly meet with you, talk with you, find out what you're upset about or what kind of help you need. And in addition to that, as a part of our SOC team, we have the hotline.  Here are some reasons to call the hotline.  Are you feeling scared or worried about your physical safety? Do you want to voice the concern? Would you like to report a code of conduct violation anonymously if you preferred? We have a phone number, it's up there, all the places. We also have a Discord channel called DEF CON Hotline and there's an email that you can email your concerns to at safetyatdefcon.org. Those are all the ways you can find the hotline. So also in terms of safety, these hallways, we've got stanchions and we've got flagging tape and people are walking in different directions. Just take a minute, put your phone in your pocket, look up, look where you're going and try and stay to the right. That'll help traffic move easily. And let's talk about some respect, friends. Because you know what, we are a community and we all just really are here to learn and get to know each other and get to know more about the community. So let's be mindful that when you're taking pictures, please, if you're gonna do it, I mean we know you do it. There's a photo policy, we ask you to follow it. There's a whole code of conduct thing, so maybe read it and try and follow it.  If you don't wanna be in it, turn your face away.  If you noticed that someone took a picture of you and you were very uncomfortable with that, please say something.  I really wasn't prepared to be in that. Do you mind deleting it? Friends, if someone comes up to you and says, hey, could you delete that? Be nice, be respectful and say, you know what? Sure, not a problem. So that we're respecting each other. That would be really great. Let's see what else. Don't be afraid to get to know the people around you. As you're introducing yourselves, you're standing in lines and you get to meet new friends. Be approachable, hey, you know what? It was so great to meet you, can I give you a hug? Consent is key, my friends. Say it again for the people in the back. Consent is key. So, if you're gonna hug, if you're gonna touch someone in some way, please, please, please ask for consent because if you don't, then they call us and then we get to find and talk to you. That was it for me. So, I'm glad you guys are here. Thank you so much for asking me to be a part of this panel. This is, like I said, this is my 13th year but this is my first time speaking but I have to go back to work after so I did not take a shot. So, everyone could raise a drink and take one for me. That would be great. Thanks. Hi, I'm Five Penny. I am not much of a public speaker so I'm gonna do my best here. Okay. So, this year we have vendors as well as exhibitors. Exhibitors are a new department for us. They don't sell anything. It's more kind of like a classic trade show and they sponsor the non-profits that you will see throughout the vendor room this year. The vendor room also has, the vendor room has a non-profit corner that you guys should definitely check out and visit. You see. Thank you guys for being so supportive. I'm definitely an introvert. Didn't know I was gonna be doing this. So, we also have communities in the non-profit area of the vendor room this year. Some you will recognize such as Wisp and in the vendor area, you'll see a lot of stuff out on the table. The vendors are very friendly. They want to talk to everybody about their products but that doesn't mean what you're seeing is free. So, definitely when you're talking to the vendors, engage with them. Make sure that the swag is available to take and it's not actually a product. I think that's it. I think that's it. I think that's it. Hey everyone. So, as you all know, we have a lot of content across all of the properties. Here's at Caesar's, Hera's. Link? Link and Flamingo. Obviously as the workshops lead, I usually don't leave Flamingo a lot since that's where our workshops are at. But what I would recommend as we get into the main days of the con on Friday and Saturday and then before closing on Sunday is take some time and review the program, the schedules, the list of things, the talks, vendors, contest and events, all of that and try to pick a couple of things that would interest you. Download the HackerTracker app so that way you can stay on top of things and also hop on to Def Con Social or the forums and try to find out what other folks are interested in, what they're doing here at Con. When I started coming to Def Con, Twitter was pretty brand new so that was how we would like find out where everybody was congregating, which bar everyone was hanging out at, what talks folks were most interested to see, what villages were doing the coolest stuff and then we would meet there. So Def Con Social is kind of taking that place now and if all else fails, have a backup plan. So we have DCTB playing across a lot of the hotel properties and we have five channels and it really beats standing in line, especially like Saturday morning when you might be a little hungover or the end of Saturday afternoon or Friday afternoon if you're feeling really overwhelmed, especially as a lot of like introverts and amberverts where we're surrounded by people, they're constant stimulation, we just need a moment but we don't want that FOMO. We have DCTB and we also have Twitch. So at main stage we have four main tracks with about 120 talks. There's something for everybody but not all talks are for everybody. So as well as being the head of the workshop CFP, I'm a member of the main stage CFP review board and what we try to do is take all of the proposals, review them and try to figure out what's interesting, what's novel, what do we think that you as the attendees might have the most interest in. So we ask that you guys keep that in mind and don't heckle the speakers, especially since a lot of us, like I said, are introverts and amberverts and it's hard enough to just get up on the stage. So if you have a question or a comment, save it for after the talk, approach them, be like, hey, let me buy you a coffee or beverage of your choice and let's, like I have some questions or I want to talk about your talk. Most people are really receptive to that and it's less hostile than being heckled during like your first DEF CON talk. Then we also have workshops. So this year we are located at the bottom floor of the Flamingo. We have five sessions a day on Thursday today, Friday and Saturday. They usually run from 9 a.m. until 6 p.m. And we cover everything from red team, blue team, mobile app sec. We have our first policy workshop this year and so we're really excited about that. Most of them sell out within minutes. This year we had a couple that sold out within hours. So it's really hard to get a seat in them but on the forums we do have a thread that people, if they're canceling their pre-redge, they let folks know like, hey, I have a ticket for sale, not for sale, but for swap and I can transfer it over to you if you want to go to this workshop. Otherwise we don't have a wait list. So please don't fill up the hallway. We would really appreciate that. And now it's Nikita's turn. Thank you, Megan. Okay, I'm gonna talk a little bit about some pro tips, some advice I thought I would put together after asking someone who it was their first DEF CON and these questions came up. So that's why they're here. After I finish my portion of the talk, we're gonna open up to a short Q and A. There are, there's a mic in the center aisle here. So if you have a question that you wanna ask, you might wanna consider lining up for it in case it gets long, I don't know if it will. So one of my advice is don't forget your con badge. We're not going to replace it if you lose it. If you lost it, we're sorry for you, that's really bad but we're not gonna give you a new one. If you are walking around with a counterfeit badge, we're gonna take that out and Sock's gonna have a couple of conversations with you. If you are coming down to the parties, bring your badge. It is required. A lot of people dress up, they don't think about it, they're wearing nice clothes or dresses or cosplay or whatever and you forget to bring your badge and then you have to take the long walk back and it sucks, ask me how I know. So the gentleman I talked to about said, what's the one thing you used to forget to do all the time? And I said, not get too drunk. So after I was done working, I would go out to the sky boxes or the parties just like anybody else and before you know it, you might have had a little bit too much and one of those experiences in the Riviera, I was the mobile bartender. So I had a messenger bag, I don't advise you to do this, by the way, at a messenger bag lined and filled with ice, disposable cups, mixers and various forms of alcohol. So people would come up to me and say, hey, make me a drink, sorry boss. And I would do that and I would take the straw and I would taste it to make sure that it was okay, yet never sample your own product. It hit me like the wave of the heat when you exit out of the airplane. And I said, I gotta go, I gotta go, I, peace out. And I almost didn't make it to my room. I ran into a goon. I wanna say his name was John Cue public, but I'm not entirely sure. And he was like, oh yeah, we're not gonna leave you alone here. So he helped me get to my room, but if something feels wrong, like if you have a suspicion that I've only had one drink, why do I feel like this? You might want to go and take a rest, maybe talk to a goon, say I don't feel good, something's wrong. And then they'll take it from there. They have a whole process and checklist. But yeah, that was a funny story for me. Just passed and they advised on. Sanitary supplies, tampons, menstrual pads, they're in both bathrooms. You can use whatever bathroom you identify with. This year we have a sticker wall. Please use it. Please. Yes, please use it. And the last thing that I wanted to tell you about is how to get the most out of DEF CON.  Like you make a plan and you're like, no, I'm supposed to go here, but this vortex wants me to go hang out over here. Just go with the vortex. If you go where your heart wants, you don't have to feel guilty about missing a talk. We'll publish it online for free. You can watch it later. Don't feel bad about not getting to visit one of the 32 villages. It's a lot. It's over, it's half a million square feet. So there is a lot to walk around which reminds me wear comfortable shoes.  They have to get you miles. Some of our goons put on 14 miles a day. So it's a lot of walking and it's a lot of moving about. But back to that. Some of you are here because you're employer. Yeah. Okay. He said it twice so we could get applause twice. Yeah. Okay. No, no, no. I was sorry. So some of you are here because maybe your employer decided to foot the bill this year. And that's fair, especially if you're new. It's a safer bet to give us a chance to experience it when your employer is paying for some of the bill. No judgment. However, don't think about what does my employer want to see from me when I get back? Like what did I learn?  So instead, try to look at it as what did I learn? I can learn. What things interest you and inspires you to make something better at work or for your team or to inspire you to take on a new project, which leads me to this question. When you're walking around DEF CON, I want you to think what do you want to spend the next 12 months doing for your employer? Until you come back. A lot of you will flip from one end to the other and you'll see lots of things and get inspired to want to make a contest or do that thing better. We will be opening our call for contests almost immediately after the close. Thank you. No. I'm talking. OK, thank you. It's funny, it's like I'm the person that can text to make that go away, but I'm on stage. OK, so yeah, our call for contests will be opening almost immediately after the con is over. We have forums that currently have a whole bunch of different channels where you can interact with the contests now. If you want to contribute to what they're doing, grow upon it, add upon it, reach out and interact. There's nothing wrong with introducing yourself. That's just normal hair. Nobody is too unapproachable or too big or too rock star. That's not the culture that we support. We support knowledge and engagement and walking into a village knowing nothing and walk out knowing at least one thing. So no, I think we have a lot of time left on the clock, but if we'd like to open it up to any questions, you could yell really loud or go to the mic. Thank you. Questions? Oh, OK, go ahead, sir. Let's see if they'll turn on your microphone. Testing. OK. Thank you all for putting this on. In terms of FOMO, is it pretty safe to assume that all the main stage talks will be recorded? And then what about the villages and the talks there? Yeah, so all the main stage talks will be recorded and captioned in English and then published later at some point as fast as we can. And some villages, I think you do know which ones? Yeah, there's a few villages that are having recorded talks in our Track 5, which is kind of like near the contest area. So some talks from Car Hacking Village, AR, XR Village, a few others. And there are some individual villages recording like AppSec and I honestly can't remember, but they do exist. OK, thank you very much. One question. OK, anybody next? If not, we'll just tell you stories. OK, so raise your hand if it's your first DEF CON. I know. Raise your hand if you're international. If you're what? OK.  Ah, I saw it. Oh yeah, how do you decide when you add more villages? So you might have noticed some people talking. We constantly are sort of renewing and changing villages. And villages come and go based on their size, the maturity of their group, maybe the people who put together have moved on with jobs. So everything's in sort of constant renewal. But we've tried to put a focus on villages to be hands-on. So the villages aren't just another speaking track, but you can walk into a village, do a thing, and then you're like, hey, I just learned how to do the thing. And that's sort of the idea of a village. And so sometimes people will tell you about how it was three years ago, but we're really trying to get more hands-on in villages and trying to move conversation talks and other kinds of events into different spaces. So what we do is we look at all the villages we have, a new submission will come in and we'll see, well, what's the history of the people involved? Is it a good idea? Like, what was it? Like the automated marijuana growers village. You know, and you're like, what is that? And you look at it and it's like, oh, it's automation, it's industrial control, it's all these microcontrollers. And you're like, oh, that's pretty neat. And you try it for a year and you realize nobody really wins. People got those skills in other villages. And you're like, okay, well, that didn't work. So we'll give that space to somebody else the next year. So a lot of times we give you just enough rope to hang yourself and try it out. And if you show improvement, we'll give you more space. And that's why when you see villages that have a lot of space, that's because they've been repeatedly really good at delivering. And small villages don't mean they're bad, it just might mean they're new. And we follow the same thing for contests and events. But as Nikita will tell you, there's only so much space and so we have to make really hard decisions about which villages we're accepting. There's a bunch of villages we would want to accept we don't have space for, essentially. Boy, the base is really kicking. So I actually bothered to come find a microphone.  What is it? Does it get a prize, some swag for the answer for a first-timer? I thought maybe you were saying first in line, in line. No, but you can buy a car-hacking village badge in the morning if you get there fast enough.  Thanks, guys. Yep, thank you. Yeah. Hi, I went to ask, so I have some experience in things such as safe spaces and stuff like that. And so the SOC team sounds interesting to me, even though this is like my first time at DEFCON. So my question is, how would someone join that or in how much experience would they need to have for it? Yeah, so your question goes more to how do you become a goon? And it'd be interesting to hear everybody's stories on how, if they are a goon, how they became a goon because there are many paths to goondom. Yeah, so let me just take that. Okay, hi. So there are a lot of different departments that have goons. Like I said, all goons wear red shirts. So some of you have a lot of interest in certain things and like you said, you might be interested in being a SOC goon. What I recommend is that I personally recommend that you tend a minimum of three DEFCONs as an attendee before you think about even coming on staff. And the reason is this, every year DEFCON is just a little bit different and we grow so much in that time and you have the opportunity to see and experience and learn new things. And what you think today might be really interesting. Next year you might learn something else. You're like, oh, but I kind of like that too. So it gives you a chance to learn and see and really kind of get to know what the con is all about. And then I would say, meet the people who do that job. If you are interested in becoming a SOC goon, meet us, say hi to us, ask us why we do what we do and find out if it's really for you. Because it sounds really fun until you realize that friends I got, I did not follow the three two one rule. I got one hour and 38 minutes of sleep yesterday because it took that much to get my department going. I have a department of 150 people that keep you guys rolling through the weekend. So there's a lot to do. If you enjoy being on your feet for 12 hours a day and standing in a three foot space, answering the same question 85 million times a day, then maybe the SOC is for you or info booth. But if that doesn't interest you, then there's a lot of other things that you can do. But again, I would say observe, experience, ask questions. Thank you. Hi, sorry. I just have a quick question. There's a few talks that I'm really interested in. They are the virtual hospital in space, hacking reproductive health, the microbiome. They all said that they're virtual talks, but I can't seem to find the links for them. And I'm just wondering when I can watch those. Have you checked on Hacker Tracker? Yeah, they're on Hacker Tracker. They just said they're virtual, they're listed for midnight last night, but there's no other information. Which village is it? I think it's the bioengineering. Oh, the biohacking village? Biohacking, sorry. They most likely might be on the Discord or on one of their own Twitches. You might wanna check out the info booth and see if you can get a little bit more information from them. They may be able to help. Okay, thank you. He asked if there's more than one way to, more than one path to becoming a goon if we were interested in talking about that? Yes. Oh, ours. Okay, he wants our stars specifically. No, we've never had a talk about that. I can tell you my story, it's kind of funny. If you wanna hear it. My first DEF CON was DEF CON 18.  So, again, don't do what I did. But I became a shadow goon. I basically kind of watched what they did and I thought, you know what? I have a lot of really interpersonal experience and I have a lot of crowd control experience. Once upon a moon I worked for Disney so I can move a crowd of people. And so they just were like, all right, clear this hallway. And so I did and they were like, okay, cool. Come back tomorrow. So I did that and then I have a personal motto and my personal motto is how can I make your day better? And so that really kind of, I kind of went into this talk with, hey, how can I make your day better? And they were like, hey, how can you not make our day better? And I basically started, this was back before the no outside food or beverage was really enforced. We all remember those days. I used to hand make homemade sandwiches and lunches for the sock goons back when we were at a department of 75. So I made 75 meals for four days, 75 meals a day for four days for our team and that was how I could make their day better which was to keep them fed and hydrated while they were on the floor. And as I did that, more and more things became evident that things needed to be done within our department and so where there's a hole, I'm gonna fill it. So I just started taking on new responsibilities. And here we are 13 years later. I'm the chief of staff for a department of 150 and we're one of the largest of the DEF CON departments.  Hi, yeah, so this is my first DEF CON and I was pretty surprised to see so many people and especially so many people carrying cash and paying in cash. So I was wondering about the backstory of managing cash. So first, did you ever think of switching to something else like for example, Bitcoin or just outright credit cards or whatever? And also, how do you get all that cash out?  Is there really a problem? Yeah, that's pretty sus of you to ask. Yeah, I know, I know, but that's why it's entertaining. I mean, I think it's entertaining. It would be entertaining to know. It is a production, but it is a classified production. But I would say that one of the reasons why that we have our registration system as a cash-based by design is to protect the privacy of people that are otherwise unable to or concerned about putting their name in a registry that could then be subpoenaed or collected. That kind of has always been the culture for Jeff. Sorry. You know, things are really good now. There's support and encouragement. We've got policy, village. We've got people wanting to interact with us. But back in the days when Jeff started this and for quite a while, it was, you didn't want the feds to know who you were. They were a little bit out to get you, my goodness. But as far as cryptocurrency is concerned, no. Thank you. Thank you. I think it was mentioned in the DEFCON documentary. At one point, the hotel we were staying at, Alexis Park, there was some sort of, we never knew what it turned out to be, but it was a counterterrorism investigation, supposedly, and they pulled the records of everybody that stayed at all the convention hotels and anybody that paid with a credit card. And we never even got a subpoena because it was so well-known everything was cash, there was nothing to get. And so that probably saved us hundreds of thousands of dollars of legal fees by just not collecting it in the first place. So I'm a big believer in this whole like, if you can't protect it, don't collect it, because it saved us. And real quick, to the person who asked about the schedule, Hacker Tracker already heard your question and are already fixing it. Yes? We want people waiting in line at the microphone to get priority just so everybody can hear them. Adi, I've got a question from the Discord. Someone wanted to know about what went into the decision for forming communities and how does it affect the communities, villages, exhibitors, et cetera, divide. And then also what communities are there? Wow, spiking is weird. Yeah, so the community idea is sort of a work in progress, but the idea is, what's the EFF? We put the EFF in the vendor room and they would, for donations, I'd give you some stickers and stuff, but really they're there to grow their community and explain to the community what is the EFF doing? What are they working on? What are their priorities? And then you would have other community groups, right? You would, oh, say again? Yeah, you had women in WISC, Women Information Security Professionals. You would have other groups that are organizing and trying to find other people interested in their projects. You would have, say, maybe, what was the university? Yeah, UAT, but you would have people that were clearly not selling something so they're not really a vendor and they're not really running a contest, but it's really important that they find their people and communicate with what they're doing, ACLU, whatever. So we decided, what we were doing is we were taking communities, essentially, enforcing them to fit in either the vendor zone or a village and it just, it wasn't working. And so instead what we need to do is listen to what their needs are and if it's just tables and a place to meet and talk, then why force them to follow all the requirements for a village? So that's what we're trying, right? We're trying to figure out the needs of communities that wanna meet and organize, but aren't really selling a thing or running a track of, you know, hands-on. That's the idea. Hello, hi. First DefCon, glad to be here. Thank you for everything you guys do. I've got a kind of important question, though. So in the book that I got, it basically said, okay, goons are wearing red shirts and they have on-hours and off-hours and when they're on-hours, they don't drink and then they have off-hours and they can drink. Obviously, everybody wants to buy a goon a drink, but that didn't really discuss all the rest of the staff that aren't wearing red shirts. A number of you up there aren't. What's the policy that applies to the rest of you as far as drinking? Yeah, so I mean, I don't drink during con-hours. I don't know if Nikita does. She probably shouldn't.   And so to your first part, the red shirt. Some people, we don't wear red shirts because if we did, we would never be able to get down the hallway. And so as like the HQ staff, the staff that's responsible for paying the bills and making some of the stuff happen, it's really important that we get around quickly. For operational departments, yeah, they need to be visible. And sometimes people do drink and if you see it, I don't want you to knock them out, but we definitely can't have drunk goons in red shirts. That is no, no, no, no, no. That's a huge liability problem. That's not what we're about. So if you do see violations, please let us know because we need to be held accountable. And if you don't see us being held accountable, then you're not really gonna back us up, you're not gonna believe what we say. So we try to create a system where we have a code of conduct, we have a transparency report to show you that we're holding ourselves accountable. And you'll see in that transparency report every year, usually we let a goon go, right? Because they've misbehaved. So the rules apply to us just as it applies to everybody else. I can definitely say that there's a reason I didn't take a shot. Like I said, first time speakers take a shot, but I have to go back to work. And so I felt like that would be violating the piece of paper that I signed when I came here that said that I won't drink while I'm on duty. So I hold myself accountable to that. I hold our team accountable to that. I don't let the SOC goons drink in the red shirt. If you see them off duty, we all have the red badge. So if they're in a red shirt, that means they're on duty and we're here for you, especially the SOC team. We are the Department of Fund Enforcement. We wanna make sure that you are having a good time and that you are safe. And we can't do that if we're inebriated or under the influence in some way. So we just try to hold ourselves to that standard. Other departments I also believe have kind of the same conduct because we signed that code of conduct that says that we're not going to be inebriated or under the influence while we're working. So, but again, I can't speak for other departments. I just speak for SOC. Okay, well, thank you. We're gonna wrap it up now. Last question. Oh, yeah. Oh, I thought it was you already. Yes, go ahead. Sorry. Yeah, just one clarifying question about how badges work. Like, I also knew that cash was the main payment method, but seemed like these year people that pre-register, play with card got like a full real badge, but people that pay a bit cash got the paper badge that doesn't even have like the puzzle or any of their interactions. So how does that work? Well, it's not supposed to work that way. Unfortunately, that's how it's worked so far. So what happens is one of the things that we can count on is when people pre-register, we know they're coming, they've given us money. And so we know how to allocate the resources. And so as we were getting bad shipments and then we started having problems with the badge manufacturer, we had barely enough for the people who even pre-regged. We don't think of the pre-regged people as anything different. It's just, it happened that the number of pre-regs was just slightly less than number of real badges we got. It's not a policy or anything that they get it and others don't. And so I'll talk about it more tomorrow. We're still getting more information from the manufacturer who has been very, very slow, I should say. Right now, we're investigating flying somebody in an airplane to physically fly on Saturday with more badges from the manufacturing facility.  And as we get more physical badges over the next couple of days, what we'll do is we'll swap out the badges for real badges and we'll just keep doing that until we've run out of people on airplanes. But, you know, thank you. Okay, oh, you're sneaking in a question? No, you're not. Okay, great, okay, so I wanna thank you all for coming. We're all approachable if you see us, but if you see us heads down running down the hallway, please don't be in our way. We might storm right over you. Again, it is what you make of it. I hope that's something you take away and, yeah, no, no, I thought you were gonna say something. No, no, no. That's okay, and so one thing that we try to do when you're thinking about the difference, what's the difference between DEF CON and say any other InfoSet conference, when we do look at villages or contests or events or anything, we always say, is that hacker? Like, it doesn't have to advance your career. Being here is not necessarily about upskilling yourself for your next job. Here's where you recharge your creativity and try something new and meet new people, and so that's why we sort of try to be, that's what we mean when we say more hacker. Your company doesn't care if you learn how to hack a PS3, but here it might be really fun, right? And so that's what we're about. The mindset, not necessarily the latest, you know, attack against an Oracle database. So thanks a lot. Thanks to the panel. Thank you. And see you around the time. Thank you.


Hello, I'm Catherine Unsworth and I lead the skilled workforce development team at the Australian research data commons. I feel like the song I'm so excited to be playing in the background, but then maybe not. I can tell you, though, that the Digital Research Skills Austral Asia or DRISA project team is so excited. We have now officially launched DRISA, so over to the team for their reflections on their journey with DRISA and its value to the sectors. The best part of being part of the DRISA working group and collaboration is really the ability to build a tool and contribute to its development. That will allow us to showcase the amazing talent and skills in the data science and digital skills training space in Australasia. And I'm really looking forward to learning about all of the trainers and courses and materials that are developed in Australia, really having a central place to go and look at that, to showcase the work that we do, but also really to see what others in the community are working on and towards. Hi, I'm John Simon from the University of Newcastle. I first heard of the working group through an ARDC update me. At the time, we were having internal discussions about how we could enhance our existing training programme for researchers, particularly in an environment where funding is quite heavily restricted. The working group seemed to be an excellent way to see how other organisations were addressing some of the challenges. The first thing I enjoyed about the working group was meeting and working with people from across the country. As a working group growing momentum, I was delighted to be able to find that we would be able to provide a tangible end result, DRISA. Thanks largely to funding from the ARDC Enforcement. Not only that, I was able to have a small part in steering that development. Being part of a working group that can provide a tool like DRISA is rewarding in itself. All I'm also knowing with DRISA or something like it will assist us as we develop our own research training programmes. My name is Kaye Steele and I work in a smaller dual sector regional university with research and teaching across a wide range of disciplines. This portal provides a one-stop shop for schools development for researchers, so that across the research sector, we can optimise use of institutional time and resources for maximum benefit to both trainers and participants. The best things about being a part of the DRISA project are the opportunity to discuss a common approach to making it available, a portal for researchers' skills development and being able to provide input to the site architecture so that it is clear, easy to use and helpful for our researchers. There's also been the satisfaction which has come from seeing the portal develop so that researchers will have increased access to schools development opportunities. Hi, I'm Melissa from Australian Biocommons. For me, the best part about being part of the DRISA project has been seeing the international and national communities come together and share their skills, knowledge and expertise to solve what is actually a common problem for all of us. Everyone's brought so much enthusiasm to the project and I think that's really allowed us to build something that I think will truly help people find the training materials and expertise that they need. Hi, my name is Anastasios and I'm the E-Research Training Manager and Lead Research Data Scientist at Intersect Australia. The best thing about being part of the DRISA project is to demonstrate to the greater community what can be achieved when a group of enthusiastic and passionate people put a collaborative effort and share ideas and knowledge. For Intersect, research faster is our main purpose. DRISA Web Portal will not only make a great range of digital skills, training events and materials discoverable and accessible, but hopefully will also demonstrate how important and valuable all these training courses and events are to increase research efficiency and productivity. This is just the beginning of this journey and I'm very thrilled to be part of it. What the ARDC and its partners are striving for at the national level is a coordinated and targeted digital research skills and training agenda that ensures our researchers are equipped with the skills and knowledge to push boundaries and lead the world in shaping a better future for us all through leading edge research. DRISA is a game changer in that quest. On a personal note, I'd like to share a quote attributed to Henry Ford. Coming together is beginning, staying together is progress, working together is success. This for me sums up my experience working on DRISA, working together with respect, collegiality and a commitment to getting the job done. It's been a flipping crazy ride, which by the way is by no means finished, but this group of people are phenomenal to work with. It's been a career highlight without a doubt. Hi. What's the best thing for me about being part of the DRISA project team? Well, it's getting to work with a group of active and enthusiastic users and project donors. Because as a software engineer, that is what really makes a development project feel worthwhile. Hi. I'm Ann Buckhausen. I'm the education and training manager with the Plessey Supercomputing Research Center. I remember attending my very first ARDC skills summit in just after joining Plessey in 2019. And at that time, a national training registry and calendar was seen and voted as a national priority. Fast forward to 2021 and we see the launch of DRISA. I see the value of DRISA as making real the F in fare. That is making or focusing on findability. So it makes findable training and events and materials and providers, training providers nationally for users who want to take training. Being part, switching gears a little bit, being part of the working group of DRISA has been an amazing experience in and of itself. I'm not experienced before such collaboration and that's real collaboration nationally across more than a dozen institutions. I feel privileged to have been part of this group, this group of committed and keen training leaders across Australia. DRISA is already populated with some events, materials, providers and trainers, but we need critical mass in content and users to be sustainable. That means we need you to make DRISA your go-to place for digital research training. So check out the portal at DRISA.org.au if you haven't already. We're actively seeking feedback. Please let us know how we can help you take advantage of DRISA. Contact us at support at DRISA.org.au. Thanks for your attention.


Well, it's wonderful to be here and I will add to I'm going to try to earn my grade back here Brian So we had a cancel last spring. I felt horrible about it because we really did think that we were going to be finished at that point But what Brian didn't realize until he just asked me two minutes ago Is that we actually had the first national release of the findings at Vanderbilt yesterday at 1 p.m? The research team Brian kept on going Matt. Why are you you know? Why do we have to get it done? Why we need the report this way and I hope we can edit this part of the video so it doesn't sell me out here But I just you know, it's time to get it done We figured it out and but it was really because I couldn't cancel on Brian twice So I hope that will at least get me a quarter point back The the real agenda for today at least from my perspective is to talk about the design and implementation Some of the threats to validity in terms of the experiment in which we are conducting To go over findings which are quite interesting. Maybe contrary to what some advocates thought in many expected it in some lights and then to See how this fits within the general policy arena Is anybody else getting an echo a little bit? I move over here In terms of design and implementation We kind of have to take a step back into 2004 2005 time frame because the way in which this experiment was set up is we implemented the project We had negotiated for two three-year period when it was implemented part of the part of the negotiation For for the project to launch was that we wouldn't talk about results We wouldn't do any analyses until full implementation was completed full implementation to us meant that that final check went through payroll services and we were done and No matter how many people asked us. We really didn't look at any outcome analysis except for calculating the teacher Effect estimates in order to distribute bonuses in those Final payments went out in November. We then launched into the last nine months or so Trying to figure out what the story is within the data at the beginning of 04 05 time frame the focus of pay for performance in particularly for advocates of pay for performance We're saying we have this great thing now. It's called value added all we need to do in order to approve teaching And I'm simplifying this quite a bit here is to offer a significant financial incentive, which we've never done most pay programs implemented prior to 2005 had a maximum bonus amount of about three thousand and even three thousand in Many respects was on the high end And that we need to tie the performance standard the performance measure to something that's objective We can't just have professional development or somebody come in the classroom and give you a check mark In value added in a sense brought this to the table At the same time there's reports coming out and surveys of fortune 1,000 companies that from 1987 until present had increased six-fold In the number of their employees whose performance bonus component was based upon an individual level So these things were informing the project and I think it's important to say at the same time that when we design this project This isn't a project that we would think we would implement in practice, right? This isn't the optimal solution We're trying to answer a policy that had been debated in many regards since the 1860s We can go back and see superintendents quoting and saying pay for performance is the way to go And all through the 20s in the 20s. There were some sort sorts of pay for performance programs 1921 the single salary schedule comes in which compensates teachers based upon years of experience and degree held by 1950s 90 over 95% of all school districts in the United States were operating the single salary schedule if we use this to 2002 schools and staffing survey again 97% of teachers were paid off the single salary schedule And in fact if we wait that by the teacher estimates it goes up to like 99.5% It's it's important to remember too within this project that there's There's different types of pay for performance programs or there's different theories that's driving behind it We're testing what people are going to call the behavioral or motivational aspect of it many advocates of pay for performance are going to say a Financial incentive will cause teachers to work harder. I think teachers quite personally work very hard But it may encourage teachers to try something different that they hadn't in the past The other piece is the selection or compositional effect So those who are most likely to be rewarded under the metered activity are likely to come into the profession And those who are most likely to be rewarded are to stay in the profession As well as those who do not receive an award it sends a strong signal either to improve practice or potentially find other opportunities of course we test the motivational aspect So the project was called point the project on incentives in teaching It took place in Metropolitan National Public Schools, which has about 72,000 students 125 130 schools we were based in middle schools grades five through eight There's of the individuals who were eligible to sign up 296 teachers signed up That was about 70 percent eligible. We had at least one control and one treatment teacher in 38 of the 40 middle schools in Metropolitan National Public Schools And that was just in the first year as I'll show later. We had quite a bit of implement of attrition It implemented for a three-year period 0607 to 0809 And teachers could earn up to a bonus of $15,000 the average teacher salary in the district is roughly around 42 And that's per year. So we just randomize in the beginning of the project I want to show this because a fear of mind in terms of the results coming out is is There's two sentences to the story and it's potential that media only get the first and the first which I'll tell you in a bit The second is that pay reform can come in many many different structures We can conceptualize an incentive program in many many different ways This is a taxonomy we developed for a project we did for the OECD But you can see incentive structure, you know comparing a Rank order tournament to a fixed-price contract unit of accountability We could think of a hybrid system or we're rewarding individuals and teams What are the standards and thresholds is it going to be linear? Are we going to talk about some step function again all of these could play? Could play a role in whether they have an effect or not at the beginning of the experiment as well I should say that that this was just not You know Vanderbilt and ran and a couple a team of researchers moving forward and saying we want to do this in a Superintendent saying all right go ahead It took a roughly two and a half years in negotiations That included getting buy-in and support not only from the superintendent the school board From the local public education foundation from the business community from the mayor But also a key partner throughout all this was Metropolitan National Education Association So the teachers association both at the local and the state level When reporters would call and say you know, how did you get them on board? Are you really being serious about this? They were key players And it really came down to the executive director of the Tennessee Education Association Making a statement during one meeting and he goes we just want to know He's like it's been too long. We just want to know whether this is an effective policy to increase student test scores other things that that were laid out throughout the negotiation process is that Teachers couldn't compete against other each other. So we had a fixed-price contract. Obviously this gives us a pretty big financial exposure Awards would be made to individuals not to teams and we actually see something that quite contrary in practice in other places But but Metro was very specific to say we do not want a vote of seventy percent of teachers saying yes To have to make those other thirty percent participate And so they let us implement at the individual teacher level We're gonna evaluate performance based upon a gain score or their growth excuse me their progress over time in the performance threshold certain a bonus this was Difficult one to determine but obviously with a fixed-price contract We didn't want them so low that if everyone got above the bar we would go bankrupt essentially We also didn't want them too high that they You know would almost demotivate those who had no chance at all to reach them and Maximum bonuses were large and hopefully that's pretty clear We developed in terms of trying to figure out what the standard was What's the performance threshold in which a teacher had to meet in order to earn a bonus?  I'll explain that in just a bit We looked at what the point estimate was at the 80th percentile, which was about three point seven at the 85th percentile And then the 95th percentile and that's how we arrive at these five ten and fifteen thousand dollars This was also important to the district and to many involved because This was it was fixed for the rest of the time for a three-year period So essentially teachers couldn't prove over time at the beginning of the following school year They all received a teacher bonus report that clearly explained an outline, you know, how they performed relative to that benchmark And various documents that we provided so in terms of the bonus formula itself one thing that we encountered was In this when we submitted the grant in prior no five and we heard in the spring of 06 Is we were told that every teacher for the most part in middle school who teaches math just teaches math When we received the data and we had we basically started in July of 06 We had to have this implemented in September of 06. There wasn't much time We found actually the great majority just didn't teach math and so the way in which we were conceptualizing of The performance measure in order to calculate bonuses could leave a lot of room for opportunistic behavior in Game and in that sense is if I teach 50% math and 50% English language arts Am I just gonna teach a hundred percent math because I can now earn $15,000 so what we did is Created a bonus formula that would actually adjust downward if a teacher's students didn't perform in a Non-math subject at the average level of performance within the district for that school year so Basically what we did here is if we say that teacher earns 10 it has hit the 85th percentile of performance obviously We would have I subscript M1 be one I Mscrew up to be one that's gonna get us to 10 PED is gonna fix it to say that this teacher had a hundred students 75% of them were and I would say we're in math 25 of them were enrolled in which one's English And they didn't make it so Essentially, then we adjusted to that they get 75% of the 10,000 This would be this would be district average progress Yeah, and it came up quite a bit actually that that Adjustment took place although there wasn't drastic drops In terms of the performance measure that we use again. I mentioned we use a simple transparent value added measure that I'll walk through right now the We Is everyone sure knows here Tennessee has TVOS it's existed since about 1995 TVOS reports come out The only individuals who can access at least as of the time we implemented this experiment were The superintendent the principal of the school And a designee of the school board TVOS data comes back to the state of Tennessee in individual sealed signed envelopes So we're making good use of our data here when you open up a TVOS envelope you get a single page piece of paper It has about it has a somewhat of a table on it 10 6 numbers the most information is it is a disclaimer at the bottom That's a paragraph saying what these numbers shouldn't be used for For the most part I Don't think teachers ever saw these numbers and in fact throughout the experiment at one point we had a teacher call who Near tears I mean we have interesting phone calls throughout the experiment, but near tears and said I always thought I was one of the highest performing teachers in the district and She had around a negative 27 on a very simple measure of value added which put her bottom the second percentile of performance She'd have been a teacher in the district for 15 years TVOS had always been there But it's now this translation of this information that seems to put something in that she reached out to us to say What can I do to become better? Obviously as researchers we don't want to interject and in train But we pointed her to the math mentors and back to the district So she could get the necessary help in terms of our measure. It's as simple as this Please note these are fictitious names, especially if anybody is here from the IRB See people like academic jokes. This is good Jay Smith is actually a person who used to work with the Center. So so we made him our example And Jay Smith The way we calculate this scored a 250 on the math t-cap in 2006 We could say that he was a fifth grader at this time We would then say we would then have to wait until summer and The state would calculate for us every single student in fifth grade who had it 250 in 2006 what did they score in 2007 on average and that's the 270 We say the expected gain is going to be 20 points and in this particular case Jay Smith did very well He scored a 285 which we could essentially say it's going to be a 15 points above 15 points value added. That's what you get in the final Wrong button. That's what we get there We do that for every single teacher That is enrolled or every single student that's enrolled in a teacher's classroom from the continuously enrolled from the 20th day of school Until the time of spring t-cap testing. We chose that definition because it aligned with the state's NCLB policy And we didn't want to design a system that went against with the structures that were already in place This breaks down for everyone the bonus awards by year We start off with in 0607 we have about 143 treatment teachers And the first thing you'll notice is we end up with 84 in 2008-09 we had a significant amount of Attrition and I'll show you in just just a moment where those individuals tended to go At the same time, it's pretty clear that each year we range between 40 and 44 teachers are earning a bonus And so You know some may think that that something could be happening here on average We spent right around nine to eleven thousand dollars was the average bonus. We paid out 1.27 million over the period of three years We had spent weeks and weeks and weeks before we implemented this project and we projected there's no way it's going to go over one million and You know going into year three in the spring and in already Having talked to most of the major Foundations across the US and we were kind of doing transatlantic flights at this point as well They weren't interested in supporting teacher bonuses. They wanted to do the The research or the program being implemented, but not teacher bonuses We were very fortunate that the individual who put up the money for the bonuses was also Promised to be our financial backstop and he covered the rest and so I'm very very very indebted to him the piece and in throughout the project that that has Been incredibly helpful is I'm sure many in here know is that that district data systems are not built around performance management Nor are they district data systems warehouses in which all the information from HR from federal programs From special programs, whatever it may be is kept in one place Nor is it the case that it's kept in the same type of information management system We were fortunate to hire somebody who was in metro national public schools as a director of data quality and assessment Who actually was a graduate of Vanderbilt to work 75 percent time for the center and his soul He would say a sole purpose was to charge forward and help us get access to the information that we needed in order To to gain as much information that we could to track and to get the data as accurate as possible The best part is that his last name is pepper and he is a doctor Our response rates and surveys that we gave in the fall and spring we had a Between 92 and 100 percent response rate, which was wonderful in year one We only administered a survey in the spring simply because of timing and the amount of Effort and where we were in terms of the implementation of the project But then we began to also include non-participants obviously because of IRB reasons The 30% who never signed up we weren't able to get approval in time to begin to survey them But through working with IRB and making sure that we Followed appropriate rules. We were able to start surveying them in year two We also conducted a series of interviews both at the end of year one and a series of interviews this past spring I just get teachers perceptions Tremendous amount of survey data that comes from instruments We used in the past on our own research studies as well as the brand corporation who was a key partner here in studies as well and Brian actually Has added some things on as well and it's been great We have principal perceptions of teacher effectiveness is another thing we did we asked math mentors who were in the district to Fill out logs about how often they came in contact with teachers. It wasn't always controlling treatment It was just teachers across the district We had teachers take The LMT and so we had a measure of teacher not content knowledge And on and on in the district was wonderful to open all this data up to us Because it was critically important as we started trying to answer this question one of the pieces as well is that When the private foundation reached out to us in the spring you said, you know, how do I know you have the right? Linkages between students and teachers, right? This is the foundation it for us to be able to calculate a value added measure of teacher effectiveness And we said well, we have monthly course snapshots We have we I know where every single student is at this date in the month and when a change occurred In this transactional history and he says, how do you know it's accurate? But it's data. Yeah, it's in the system. And so finally I gave up and I said, how are we doing audit? And he said that that's what I was getting at And so we sent every single year and this was only the treatment teachers because there was a different incentive for the control teachers And there couldn't necessarily be a comparable Comparison here. We sent them a list of their roster every single student that was enrolled in their classroom. They were masked By period and who was in their classroom at some point during the year and would not be counted for bonus purposes What's alarming is that of the 143 rosters that we created in the first year 55 come back with appeals The appeals process was then turned over to the district to solve Oftentimes it had to go down to the site level to figure out whether a particular student transfer was true or not a lot of times It was inter-school transfers or pullouts or some type of special services But we changed 153 students solely based upon that The other thing that came up quite often it came up in about seven instances in the project as well It's whole classes were matched to teachers that didn't teach those classes And there's a fantastic quote from the Dallas Morning News about a teacher who didn't receive a $10,000 bonus because of his poor French scores And the quote goes on to say that that he was very discouraged because he doesn't speak French Nor does he teach French? But he didn't get his bonus because of it and it's simply these data systems That that we have to be very careful about the weight we put on them It states particularly on a race to the top move into high stakes personnel decisions whether it's firing Or whether it's rewarding and the promise can be firing The first thing that's gonna be challenged is the quality of the data and the information in which that decision was made The good news is the data systems became a little bit better, and we see this go down over time quickly. We randomized Obviously treat to get the 296 and the treatment and control We stratify the sample to 10 groups based upon a school effect estimate And then we looked at clusters of teachers. The reasons we did clusters is if we saw a variation In the scaling also of scores in fifth and sixth grade seventh and eighth special ed and advanced algebra There was another piece in here in it didn't come it It's essentially not not relevant in the context based about how things fell out But we're very concerned about cheating or Opportunistic behavior since we obviously had at least one treatment one control teacher in every classroom There was an opportunity that the principal and the teacher could collude and That they could find out some strategic matching of the best students who are optimal in order to Maximize that teachers chance of getting a bonus and This this came up as well that during the recruitment process we had three teachers run up So during the recruitment process we put Train research assistants in a school for the entire day. We had an FAQ document. They could sign up Everyone asked questions except three people They ran into the room signed up in my colleague Dale Baloo who who Is Just been an incredible partner on this project says you don't have any questions and if you know Dale there's just a great look on his face and They go no they go we teach third year ELL students and they said in year one they do Oh, you know, they don't do well in the high-stakes test year two They don't do well in the high-stakes test But by year three when they're picking you know when they pick up the language they get the mastery They're used to the test huge gains And they're like we're taking home $15,000 They didn't it's what we learned But anyway, it was a good story There's really when we talk about threats to validity. There's three things we're mainly concerned about obviously randomization failures Particularly with small groups. We may they may end up not being equivalent. We see this at the grade level We don't see it overall the purpose of Simon is the other piece here as well Which I just kind of spoke to a little bit and then teacher attrition obviously The good news is overall the treatment and control groups were balanced across a large number of Student and teacher characteristics But as soon as soon as we go to a little bit lower level particularly the grade level we start to see some imbalance In those instances when we ask David made the models at the grade level Obviously we put in a tremendous number of covariates into the model The number of sensitivity tests here, and I don't outline all the tests that we did Were fell into the situation where they told the same story It was actually quite nice to see the same story just come up over and over and over There may have been one or two, but by chance We might expect a little bit of that We we asked at the beginning of the of the experiment that principles run the schools The way they always did Obviously we implemented a few weeks into the first of into the first few weeks of the school year So assignments had already taken place This was a huge concern for years two and three when they returned and they knew their principal and they knew who'd be making those assignments We also asked that that the teachers who participated in the project Not tell whether they're in the control or treatment group part of the concern of many people involved in the project Is that it could break down Collaboration it could cause resentment and so so we asked that they did not share this information and They actually actually signed something saying that they would not share that information So so in terms of purpose of assignment and we followed this over time every year in I'm not going to go through the numbers, but really we're looking at proportion of students who switch out of a teacher's classroom You know are we pushing certain kids out or into your class? Are we looking at dropping students with unexpectedly low beginning your performance? dropping students who have a downward trajectory and In part of this we're assuming that a teacher would really go out and try to figure out What the likelihood of a particular student is in terms of the performing credibility well in year three? Year three is a generic number here, and so we're going to expect regression to the mean They do in a sense the teachers in a sense have some access to this information Is with the T-boss data system? They actually do projections, and so this projection information Does individual student projections for everyone who's enrolled in that teacher's class? So it's it's It's plausible this could be happening But during this period of time login rates were very very low into that system And we're also able to track in who logged in So as you see here, we have control and treatment status school year three years of the experiment found in 2006-7 school year we lose two and three teachers if I already announced this once I apologize, but we only lost one teacher because they asked to be removed all Other attrition occurred because they changed grades, and they're no longer eligible under the criteria That we had set forward which is quite interesting because everyone you know the popular media is going to say Teachers do not like pay for performance You know they don't want to do any of this and I think it's important to remember that these things can be done and teachers will remain supportive Reporters have asked me for the last day now You know were you surprised by the findings and I said no, I'm surprised that we even made it this far I mean we were literally on the edge of our seats well into year three of just whether this would would last It's not part of the larger attrition here in year two is One of the clauses we had is that a teacher had to instruct at least ten math students and that were expected to take the spring high-stakes assessment We didn't have those data and those linkages at the beginning of the project because we're still in our data Negotiation pieces and so there's quite a few who in the spring, you know the students take the test We have the data now we can look at and they never qualified to be fair We let them continue in the experiment. They were not bonus eligible that year and a Number had a dropout because they meant I think it was 14 But we can find out the exact number here and so this breaks down control treatment and reason for attrition Just kind of the way we categorize it here is change in assignment versus other ability criteria The two places where we see somewhat of a difference in the end is the other left the district altogether More control group teachers left the district and down here. We see more treatment teachers left teaching But stayed in the district Typically they move to an assistant principal ship or a principal ship Sometimes to a coordinator coach type of position out of grade was you were no longer in grades five six seven or eight There's one middle school I believe in metro Nashville that actually has fifth grade in it, but it's elementary and a teacher changed and We mistakenly did not know this and so we try to drop him and he goes no no He's all very deliberate here because there's fifth grade. That's middle school. So We honored his request so in terms of evidence have been balanced. There's there's really three ways We're looking at this and that's at the beginning it do we see any differences between the treatment and control group Do we see differences between treatment and control group over time and do we see differences prior to? point being implemented I Know there's quite a long list here But where we do see differences and I'll get to some of the student ones next as well, and this is only a set of examples of the type of Variables that we were able to collect on teachers But we did see that that from year one to year two that we ended up with a slightly greater proportion of female teachers in the treatment group Which actually goes against some of the the survey literature saying that males would be more likely and more inclined to participate in an incentive program And we ended up with slightly more more black teachers remaining in Year two year three again, it goes up a little bit higher for female and Then ELL ELL was the only place in which we had slight imbalance in year one after the randomization and then Year hired I guess it is by year three But overall for the amount of attrition that we went through losing nearly half our sample The balance remained in place pretty well We also thrown a number of control variables just in our models obviously because the attrition Number of sensitivity tests and again these are some of the ones that we'll look at it's important to note here, too That for things like t-cap scores These would be the scores from the first year prior to implementation Obviously if we use their score during the year of implementation that essentially could be endogenous with treatment. I Mentioned that an issue With the randomization is small lens, so as soon as we you know, we we randomized on average We're looking at the average treatment effect, but but if we start thinking about doing analyses by grade Are we still balanced at the grade level and you'll see that when we use Achievement prior to point We have slight imbalance here and as you can see the size of the coefficient is quite similar Incoming score of kids in the classrooms that would be there That would be the normalized math test score for students In the year coming in the fifth grade they're in treatment control Between the two groups and this actually would be four So this is seventh grade, so that's going to be their sixth grade score two years prior These are levels Yes, and we I'll show a few charts that we do The other piece and now I'm going to earn back my final quarter point here I do want to make the observation that says Thanks to Brian and Lias Brian and I actually first met over this project I think he was in Chattanooga at the time in the great state of Tennessee And he got my email I called him and I just said look I need somebody who knows how to do this You're the only one who's ever done it actually you created this this in your Chicago study and This this is going to be a big ding on us if we don't have this covered in our proposal Brian was it was generous is he always is and signed up and and did a lot of the suspicious Answer string patterns as well as looking at different forms of of particular cheating or gaming We didn't see any that Would alert us to to having an issue the other thing we looked at every year too of course was was a sorting of students and and Again, nothing came up. The only time that that something did come up. We actually had a teacher call and say Two treatment teachers in my club in my school are cheating I'm like, okay, and we had a very systematic protocol of who could talk to treatment teachers They had a designated line that call into it would come to two of us We were discussed it as a team have a very systematic formal response back and and what she said is she goes They're planning their curriculum together And and last time I checked that was called team teaching So so we followed up In a nice way In terms of dependent variables We have t-cap math scores transformed to rank base these scores Typical things you would see we did sensitivity sensitivity test with the benchmark gain scores that we showed you for Calculating the value added scores There's a number of other things we looked at as well We also added in reading ELA science and social studies another piece in this too that that was incredibly enlightening is is how English 101 was ballet And we are fortunate to audit the hundreds and hundreds and hundreds of different course codes and course descriptions before We did our first year of calculations and we were able to set up a Kind of a protocol in order to consider something an academic subject because the the coding they use often I Can't see ballet with reading Treatment status we look at really in three ways and I'll present these results and that's just simple the average treatment effect One zero one being treatment Treatment status interactive with student grade We're looking at whether there's differences Based upon grade as well as treatment status interacted with year and we only did this one pulled across all years In terms of findings We did not find a statistically significant effect an average treatment effect overall There was essentially no difference between the performance of students in Control group classrooms and the performance of students in treatment group classrooms Years year prop it would be the year prior to There'll be yeah, and I can shoot but if you're in seventh grade it goes back a couple years Yeah, we also found that there was no significant differences for students in grades six through eight When we estimate separate effects by grade We did find a significant positive effect in fifth grade in years two and three of the experiment However, when the student when students Metriculated into sixth grade. We no longer saw this if that saw this difference And the effect is quite large. It's a half to two-thirds of a year's worth of academic growth But given that it does not persist We feel as though we can't place too much weight on it The one caveat here is that and I said at the beginning we're still waiting for the data to come back From the state they changed our testing system. It's delayed We will go right in and look at the same exact thing to see whether the sixth grade The students in fifth grade last year their their effect persists In terms of teacher attitudes, this is probably some of the bigger takeaways. I think I'm trying on my points back Significant gain between for both treatment and control groups or Were you a reviewer It's a critical point here is actually the gains in the school district went up quite a bit during this period of time I cannot tell you why they went up. I could I you know, we probably bring up 20 different reasons one potentially Important thing to note is that in year two the district was facing state takeover There's a lot of newspaper coverage about the state takeover would it get defaulted to mayoral control What was going to happen? There's no superintendent It's potential you could make the the argument that the incentive effect from state takeover Was greater than what we may have seen from the incentive effect of this monetary award We also looked at whether this was true because it was pretty large gain and again, we're normed off the state so We see this in third and fourth grade as well So it's just not the treatment grades in which we see that we see very few differences in terms of of teachers perceptions of different instructional practices attitudes how they're approaching their jobs their positions between treatment and control groups We do see that that teachers in the in the treatment group were more likely to report some somewhat of a positive Outlook towards their school environment and their principals and the control group teachers were For the most part though, there's really no differences that we see over time for lack of a better example, but you know if a teacher said something like I Contact parents more regularly and there was a significant difference between treatment and control Then the next question said Do you contact how often do you contact parents and the next one say never There was no common story there's really no common theme as we try to tease out what all this information was saying Now those weren't exact questions. So but Teachers of all years were generally supportive of the project We had no complaints about bonus calculations. We had no challenges that this measure is unfair And as one may expect novice teachers were much more Likely to think that the program was alright compared to veteran teachers The big aha moment right here Comparison between teachers who didn't did not win bonuses those who won bonuses were slightly more favorable than those who didn't win bonuses This is a good audience. They laugh at my really bad joke Teachers who want a bonus revealed increases in positive perceptions of the point across years as well Do you that's good, that's good That's five bonus points Brian


I'm with the Touch Graphics folks here. I've got Steve Landau and Lindsey Yazzolino and everybody I had lunch with said we ought to come over and see what Touch Graphics is doing. So what are you guys doing, Steve? So Touch Graphics is all about tactile information presented in a universal context. So we're trying to make materials that are usable by everybody but are fully accessible to people with disabilities including visual impairment. It's still accessible to the sighted as well. Exactly, and fun for everybody to use and informative and helpful to the general audience. Okay, so what have we got here? So this is an example of one of our talking maps. Lindsey do you want to talk about that a little? Definitely, definitely. So this is actually a map of the Google campus, the Google headquarters located in New York City and it is actually in the process of being mounted on that campus. So the idea behind it is that the map itself is tactile, so when you touch it you can feel the different rooms and the pads, some of the office spaces, but as you can also hear it's also providing audio information. So if you for example were to touch one of the elements on the map, so this is for example this bumpy stuff, that's an office cube and if I were to touch it a second time, 4C327 is the actual room number.  This is a giant Android tablet, what is it about? Two, three, two and a half feet across. It's a 24 inch diagonal Android tablet. And it's got an overlay on it that's in color, that shows us the office numbers, but there's also these little bumps that she's touching that then allow her to have it read to her of what the different things are on the map. Do we want to show an example of the overlay on its own? Let's show them the tactile skins, you'll see the whole picture. So this is the one of the tactile skins and they're holding this, it's basically the same kind of overlay but we're feeling it with our hands just in space. Yeah, so this is one of the skins and yes the idea is that you would set this on top of the map. The skin itself actually has an NFC tag on it so that when you lay it down on the map, on the tablet which has an NFC leader, it detects the code on the map and because the tablet has software which shows what's in each of the coordinates of the map, it will automatically then load the data for that map. I was going to ask, wait a minute, how does the tablet know it's on the overlay but no, you jump right to it. And Steve, did you want to explain the process, kind of the idea of the technology behind the printing? So we've discovered the kind of an interesting new way to make tactile graphics and tactile graphics as I think Lindsay has explained are raised line and textured diagrams, pictures, maps, all kinds of different spatial representations that we can now feel by touching with our fingers. It's not just like a Braille map, it's more than that. Well it uses Braille in some places but it's lines and textures, not Braille and that's because we're trying to show people for example where the stairs are, where the elevator is, where the map you are here point is, where the bathrooms are. And for each one of those is a different tactile symbol that you can feel very easily and we make these maps with a very special kind of printer that can lay down a gel, a liquid which is then cured. You know when you go to the dentist and they put that purple light in your mouth?  Yeah, so that, we like it, because what it does, that ultraviolet light cures the liquid that the printer lays down and it turns into a hard raised line and so we can control the height of those lines and create kind of a hierarchy where we have some very light things, these textures are very low and light, these travel routes are kind of high and smooth and then the steps, try to feel that little step, you'll see it's kind of like a little staircase. Yeah, just a little tiny thing coming up versus the travel route which is much bigger. Exactly, so once we've created this overlay we can place it on the Android tablet and because this material is so thin your finger still activates the touch screen right through. Oh, that's the real trick, so that's okay. And we struggled with this for a long time, we developed a lot of different prototypes to try to get it thinner and thinner and thinner, but we were doing it with like engraving on a plastic sheet and it started cracking, it was so painful and frustrating but then we discovered this new printer and it solved all of our problems. Wow. And the lines that it produces are so precise that not only are they easy to feel, but when I first saw them I was just blown away by how crisp and how nice aesthetic. So easy to understand. You understand and I just kept wanting to touch it because it was so clear. I like it, so it's a perfect UI basically for you. So who do you market to, do you sell these to regular companies or how do you produce your products? Well, we do lots of different markets because we're in a funny kind of business where these very small groups of users sometimes, so in order to really build up our business we're trying to reach out to museums, to schools, libraries, businesses. I had a lady from the CIA over here at the table earlier today and she thinks that this would be a really nice technology for their headquarters in Langley. Of course, I told her all the tablets have a microphone so you can talk to them and she said, oh no, no, no microphones. I'm not surprised, not surprised one little bit.  Exactly, that's what we're trying to accomplish. And so right now we're working with a great company, Google, who is sort of you know, interested in innovation and demonstrating what they think are good technologies and promoting things like that. So we're hoping to maybe do more work with them. All right, so let's take a quick break here and we're going to move scenes because I want to find out why there's a horse head behind me. Okay, we're going to tell you all about the horse. Okay, we've moved over by the sculptures. What are we looking at, Lindsay? So we're looking at several things. The one in particular is this sculpture. It's a Buddhist, oh well, it will tell us what it is actually. Listen, it's going to talk to us. This is a reproduction of a Chinese Buddhist sculpture. The actual work of art made around 1150 AD appears elsewhere in the museum. The original sculpture is extremely rare and one of the most important Chinese objects at the San Diego Museum of Art. You may touch this reproduction with your hands. As you explore it, you will hear the names of parts that you touch. If you continue touching a single part, you will hear a description of that part and explanations about the sculpture and the people who created it. Begin touching now or wait for more general information. So have you guys created this? Yes. The San Diego Museum. I'll start touching it. Go ahead and touch the feet. Is it internationally known as the feet? I'll keep touching it. Guan Yin stands on bare feet. Notice the toes, heels, and soles of the feet. So I'm moving my hand around. Okay. All right. Can we make it stop? Because I want to hear what you've done here. This is very cool. This is a 3D-printed replica of a priceless Chinese sculpture from the collection of the San Diego Museum of Art. So they came to us and they said, well, we have blind visitors who come to the museum. We'd like to make some versions of some of our most important artworks that are really accessible for this population. So we started by going to the museum, carefully removing the glass case on this precious artifact, and then we scanned it digitally using our iPhones. It's an amazing thing. You take your iPhone and you take hundreds of photographs of the object from all different angles, all different directions. All those photographs are then processed on a computer program and it creates a digital model of the form. So we created the digital model, then we sent that file to a 3D printer. The 3D printers are in wide use right now and they're really magical machines, but they can create objects like this. Once we had the 3D print, which by the way comes out as a monochromatic plastic object, we painted it with a special kind of paint and that paint has an amazing property. It can actually detect where you're touching on this. So it's through the paint that it's sensing that I grabbed her hand. Correct. Wow, that's pretty cool. Well, it's a really versatile, simple, and inexpensive way to make objects like this that people can experience all different ways. We know that little kids love to touch things. We all love to touch things. Yeah, adults do and I think most adults just have had to learn not to. Our first words are don't touch. I know adults somehow think you can get boring and not touch things anymore but we want to show that that doesn't have to be the case. I think that's a perfect place. You guys have a lot more to show but if people want to learn more about Touch Graphics, where would they go? They can go to touchgraphics.com on the internet or they can call us at<PHONE_NUMBER>. Great, thank you very much Steve and Lindsay. This was fantastic. Thank you so much Allison.