So I'm giving this talk on behalf of myself and Dr. Atanas Atanasov, he's my lead engineer on our team. Initially we had been looking at in-node work placement which has big impact for many workloads including cloud native applications. In this talk we'll look at some examples from the domain of cloud native benchmarking, HBC, telco and discuss what questions tools and techniques can help with this problem. So first I'm going to flash up this picture. Dr. Atanas and I, because we started off in the HPC world, realized that the current set of Kublet default managers are potentially leaving a lot of performance on the table. They do not allow the level of fine-grained control we are accustomed to in HPC. We are currently two to three weeks away from releasing the CPU control plane, which you can talk to me about later, to allow users to mix shared and pinned cores and to have new affinities to workloads. We will later add in isolated cores as an option as well. These examples come from our experience in attempting to benchmark new hardware platforms at Intel using this control plane. And now that we had this fancy new tool, we needed to prove that it was useful. So benchmarking server platforms in HPC is the custom. We want numbers. So we had a lot of confusion in how to do these benchmarks. So we started looking for toolkits. So first, do we want to execute manually? No. We want reproduction to be simple. Engineers get things wrong, including us, and we wanted to automate. Batch tools play a big role to increase the ROI of benchmarking. We wanted to enable users to study application regressions and save everyone time. So we then had to go and hunt down tools. We wanted to, we could use batch scripts, but that seemed hard. We also wanted to be able to schedule benchmark execution, maybe similar to slurm. Cloud-native queuing frameworks can be a valid alternative. We used Ansible to provision underneath. In benchmarking, we also used Ansible to handle workload deployment, validation of the workload properly executed, and automatic error detection logs. Further, benchmarks as well to handle the post-processing of the benchmark. So now we have the tools. Now we needed to figure out what benchmarks we were using. We could use synthetic benchmarks and evaluate system performance, but this does not well represent the workloads we are running in the cloud today. Cloud users have complex applications consisting of multiple microservices connected over the network. There may also be availability requirements, such as P95 latency access to services. So we did some research on workloads that could be more realistic. The examples are still rather simple cloud applications. At the end, let's discuss what other benchmarks people are using to evaluate cloud systems. So this will be a question I actually post to you on what you guys are doing. The first workload we looked at is the microservice-based Google microservices demo. This application is the classic three-tier app, including a front-end to receive the request from the clients, business logic services, and a database store, where we store the transactions for the customers. In our benchmarking, we evaluated the throughput of such a system on a distributed system with four machines, and only four machines. We placed a load generator on a separate machine, and we had three worker machines to front-end business logic and transactions. Our goal was to optimize the throughput of such systems under the given latency constraints. The second benchmark we used was the Duststar Bench hotel reservation system. Again, this is a microservice-based software platform that provides a search recommendation capabilities for hotels. We see here a clear split in three tiers. The difference in this case was that we had separate databases for different parts of the data model of the application and the caching layer. It turned out that these two applications had very different scaling behavior and reacted very different to pod placement strategies on the cluster. Hotel reservation had a clear bottleneck of the database components, and if you run just one instance of the database, it's turned out to be a bottleneck. Best effort QoS, the business logic, was able to handle the increasing number of front-end requests. We observed, hold on, to fix the bottleneck issue, right? We executed two instances of the workload with two database layers. Still, this was not the final optimization. We also had dual nicks. So we further isolated each workload instance on its own socket with careful network configuration. We were using Multif, but it was very hard-coded. So this isn't available easily today. This is hard. For Google microservices, the best quality of service did not provide us any benefits. The workloads suffered under noisy neighbor problem, which we managed to fix by pinning of the services and again isolating on a separate socket. All unused cores on the socket we used the remaining group of services which were not sensitive to the cache-related issues. So this particular piece is a summary of why extending these primitives is so important. So we went from a 40% utilization to a 78% utilization. This was true with both of them. Both of them had similar performances. One was not better than the other. And we really want to be closer to 90%. So there's probably more we could do regarding the database as the bottleneck. We're not sure what other optimizations we can do to get that core utilization up. These two workloads are still very far from actual applications and what the users deploy on the systems. If we look at some examples, genomics, AI, HPC, and apps, and more, users start to use the cloud platform to run those workloads. These applications are performance critical. And usually, they are optimized for a certain placement model. HPC and AI applications apply pinning and affinity. Configuration mechanism to get out the max available compute from the hardware. So currently, a lot of these applications are still using slurm. That includes internal data intel for fine-grained performance. Last but not least, we are also looking to measure other things. So it's insufficient to analyze them on one to two nodes. We do this on four, but it's still very far from reality. The customers are running on hundreds or thousands of machines. So what can we measure? We're starting to look at throughput, latency, throughput under latency constraints, but we'd also like to understand how these metrics behave at scale. Does latency go down if we add pods or do we increase the throughput? What happens also with the system are we using all the available resources. So similar to prior, we're not using all the compute available. If not, we can pack multiple workload instances, workloads on the given VMs. What about the memory? Are we accessing memory in an optimal way or are we racing cycles due to the wrong placement? It's about where we are. I have some questions on what people are using for benchmarks if they have better ones than what we were using, but this is the best we could find. Is anyone using better benchmarks? That's a no. The talks, did we have any questions? Thank you. Hi, very nice talk. A multi-part question. When you're running your benchmarks, are the nodes that you're running on virtual instances or are you running on bare metal? We're running on bare metal. Okay, do you have plans for running on virtualized nodes? And if so, how do you plan to manage the core placement with the hypervisor? Different questions. It depends on the hypervisor. So we need to know more about what our users are doing. So if people are using something like Mesos with virtual cooblets, that's gonna look a little differently than if they're doing it in a VMware type platform. So we have to know more about what we're gonna be running on but isn't the plan. We just haven't gotten there yet. Thank you. Any other questions? This kind of goes along more with your benchmarking questions. I'm more putting this, I guess, to the Kubernetes community, but I was wondering how, what kind of infrastructure is in place for performance testing for Kubernetes?  So that was the problem, right? Is there isn't a lot in place? So there's the Google microservices and there's that star bench and that's about the extents as far as that sort of workload. There are people who've run Linpack across Kubernetes, but those are very targeted, they're targeted benchmarks. They're not meant for real performance for your application. I cannot also because I used to work on scalability of Kubernetes. There are also benchmarks ran for evaluating the supported scale of Kubernetes workloads. But that's not running real applications, that's what Marlowe showed here.  And there are also some benchmarks that like test other dimensions, but like on 100 node configuration. But you try to, for example, throw every significant put density or the issue, see if the network traffic, but these are very synthetic benchmarks. Any other questions? I was wondering if you were also thinking about multi-tenancy, like running multiple applications of different nature. So that has to do with your scheduling components. So we have telemetry where we're scheduling where we can actually monitor behavior on the node. And so if you have a second tenant, there's no way today to guarantee your tenant's going to behave with regards to bandwidth as an example, right? So you actually are going to either have to have something working externally that's checking to make sure things are behaving and then either throttling it or throwing it off the node, or you have to have some assumption that if your latency starts to go down or your throughput isn't what you expect, that you're going to be rescheduling that particular service. So that has to do with, yeah. So that's really on your scheduling side. That's less on your node resources side, but you still need something on there monitoring. Is that an answer? Any other questions? We're just throwing up on that in the cooking piece. I mean, network is a compressible resource like CPU, right? Right. You can give it to a pod and you can take it back. Yep. And there are mechanisms like Linux in the Linux stack to set these things up. I'd like to basically set up for sharing between. So you don't really necessarily have to have like an external component to monitor traffic. You can just set them up in a similar way that you set up sharing for CPU cores. Maybe, but you still have jitter. So when you're looking at HPC applications, you're assuming not having jitter because you don't have other workloads. But you do end up with jitter switching back and forth between your two processes. It mostly controls basically throughput, but not. But it doesn't help jitter, so you still end up with the processing issue. Thank you so much.


Is Maxwell House the best coffee in the whole world? Well, your father says so, and your father knows best. Yes, it's Father Knows Best, transcribed in Hollywood, starring Robert Young as father. A half-hour visit with your neighbors, the Andersons. Brought to you by Maxwell House, the coffee that's bought and enjoyed by more people than any other brand of coffee at any price. Maxwell House, always good to the last drop. A wonderful bird is the pelican. His bill will hold more than his belly can. He can take in his beak, food enough for a week, as for flying, you know, gosh darn well he can. With this bit of deathless verse, we highest to Springfield in the White Frame House on Maple Street. There we find the Andersons at dinner, and there, as usual, we find a problem. It's all about socks, or is it a dress? Well, it's about something like this. So this little girl said to me, Kathy, she said, I'm going away. I'm going out west, where there's wide open spaces, and cattle, and rustlers, and bandits, and everything. And I said to her, you are, and she said to me. Kathy. Yes, Daddy? Pass the butter, please. Yes, Daddy. They only cost two dollars, Dad, and that's for the cap and the socks. It's a bargain, bud. That's what it is, an absolute bargain. You mean I can get them? No. This is a very good roast, Margaret. Oh, thank you, dear. If it's a bargain, why can't I get them? Because I'm not giving you the two dollars, that's why. Oh. And she said to me, I'm tired of being treated like a criminal. That's what. And I said to her. Kathy. Yes, Daddy?  Oh, yes. And I said to her. Who's treating you like a criminal? And she said, Father, it was $15.95 less than a week ago, and now it's only $8.50. Betty, I wasn't finished. And if you can give Bud two dollars for something dopey like a hat and socks that light up. They don't light up, Betty. They just look like they light up. I wasn't, Mother. It's in that new shade of gun metal. And if I don't get it, I'll simply get it. If I don't get it, I'll simply die. We'll miss you, dear. So, Father gave him the two dollars, and you can see Joe coming two blocks away. I don't care if you can see him clear over in West Springfield. So I said, why do you have to go out West? And she said. Kathy. What? We don't care. You don't? Kathy, you haven't touched your dinner. I'm not very hungry. Kathy, we have an appointment to spend the evening with Mr. and Mrs. Smith. We haven't got all night to fool around.  What if I was riding along on my bicycle and the truck driver didn't see me? What truck driver? Any truck driver.  Yes? No. Father. I am not going to give Bud two dollars for pink socks. And I'm not going to give you $8.50 for a dress. Now, is that understood? But Father, I need it. All right. Then it's up to you to figure some way of getting it. That's what I was trying to do.   If your entire life depends on getting a dress, go out and earn the money for it. Dad. When I was a boy, I worked for the things I wanted. And I didn't feel I was too good to work at anything that came along. Dad. I washed dishes, shoveled snow, dug ditches. Dad. Bud, will you please stop interrupting me? But I want to ask you something. If I earn the money, then can I buy the socks? Yes. Where was I? You were building the Panama Canal. Betty. It's all right, Margaret. Let's just be thankful that we have one sensible child. Yes, dear. Which one would that be? Why, Kathy. Kathy. Me. That's right, Kathy. She wanted the doll, but did she bother me for the money? No. She's earning it by selling those Pelican banks. Kathleen. I'm sorry, Mommy. I guess it slipped. Kathy, what's the matter with you? Why, nothing, Daddy. Jim, something is the matter. He's shaking like a leaf. I'll get it. You stay right where you are. Bud. That isn't the doorbell. It's the foam. Answer it. I'm supposed to answer the doorbell. Why do I have to answer the foam, too? That's what's wrong with this country. We're a nation of specialists. I suppose he expects time and a half for answering anything but doorbells. Jim. Hmm? Oh. All right, Kathy. What is it? I'll go away, Daddy. I'll go out west and start all over again.  What brought that on? Daddy, it's for you. Thank you, Bud. I'll be right there.  Wouldn't it? You see, Dad, all Betty had to do was answer it in the first place. Sit down and finish your dinner. Holy cow. Kathy, what have you done? I was trying to save them some trouble, Daddy. That's all I was doing. Save who trouble? The Pelican people. They said it was worth two dollars, so I didn't think they'd mind. What was worth two dollars? The doll. Now look, Kathy. Jim, if we're going over to the smiths... This is just as important as going over to the smiths, Margaret. If Kathy's done anything wrong... But I haven't, Daddy. And why all this hogwash about being a cattle rustler in the wide open spaces? Because they said I did, and I didn't. Well, while you're trying to figure that out, I'll go upstairs and comb my hair. You do that. Go upstairs and comb everybody's hair. Kathy. Yes, Daddy? Start from the beginning. About what, Daddy? The Pelicans. What did you do? Well, they said in the comic book that if I sent away for 12 plastic Pelican banks and sold them for 50 cents apiece, they'd give me a doll worth two whole dollars. And you sent away for them? Uh-huh. All right, up to there, it makes sense. Now, how many did you sell? Four. Only four? Well, everybody else sent away for them, too. They were trying to sell them to me. Kathy, if you can't sell the other eight, just send them back. I did. And they got mad and sent them back to me. Well, that doesn't seem logical. As long as you sent them the two dollars for the four you sold, you did send them that, didn't you? How could I? I used that to pay for the doll. Kathy. They said they were going to send me a doll, and I just thought I'd save them the trouble and I bought the doll myself. Where is it? The doll? The doll. It died. Kathy. I was playing Red Cross with Patty Davis, and that was the doll we used for blood confusions. She means transfusions, Dad. Thank you very much. You're welcome. Kathy. Confusion means something else. Like, when you see a boy on a bicycle and he isn't wearing a fluorescent mask. Wonderful thing has happened. Mary and Swift has a cold. Well, that's nice. Maybe if she's real lucky, it'll turn into pneumonia. Father. Well, what did you expect me to say? Mary and Swift has a cold. But don't you see? She was going to babysit for the Hathaway's tonight and Friday and Saturday. And now she can and I'm going to. Well, that's more like it. We're finally beginning to show a little ambition around here. I've got to be at the Hathaway's in five minutes. Where's Mother? I'll tell you, Mother.  Oh, Father, I've never been so happy in my whole life. That beautiful dress. Say, I've got an idea. Well, treat it gently. It's in a strange place. What an idea. I'll make a million. Betty, hey, Betty, wait a minute. I want to ask you something. And he wanted to tell me what confusion was. All right, Cathy. Now, where were we? They were going to put me in prison. No one's going to put you in prison. But the letter said I was responsible and I have to send them $6 and I don't have $6. I'll give you $2 and you can send them that and the rest of the banks. They don't want them. The man said they've got enough banks. Oh, fine. What am I going to do, Daddy? Well, there's only one thing you can do. You've got to sell the other eight pelicans. But I tried and I couldn't. Well, you didn't go about it the right way. You've got to create a market, build up a desire for pelicans, make the entire neighborhood pelican bank conscious. How do I do that? Well, it's very simple, Cathy.  Or, uh, you can bank on Cathy's pelican. You mean then they'll buy them? Of course. With men who know their banking best, it's pelicans two to one. Watch the pelicans go buy. That's the idea. Pelicans are better than ever. Now you're rolling. Be a pelican of distinction. That's enough, Cathy. What this country needs is a good five-cent pelican. I said that's enough. I'm sorry, Daddy. I guess I was just carried away. That isn't a bad idea, either. Jim, I'm ready. Just a second, honey. Now are you all straightened out? Oh, yes, Daddy. Good. Jim, while you're getting your coat, I want to tell Ben. Where are Betty and Bon?  But every once in a while they do hit home. Daddy told me how to sell my pelican banks, Mommy. Oh, that's fine, dear. Jim, Betty and Bud haven't even started on the dishes. Where are they? Oh, I guess I forgot about the dishes, honey. But you know that little talk I gave them tonight? Well, Betty has already gotten a job as a babysitter. She has, has she? That's right. And Bud? I don't know. He went off screaming that he's going to make millions. Really took it to heart. Now don't take your coat off, Margaret. I'll be ready to go in two seconds. To go where? To the Smiths, naturally. And who is going to take care of Kathy? Oh, no.   Stop arguing, Kathleen. This is one time when your father knows best. These days, coffee's making good news in grocery stores everywhere. In city markets and country stores alike, grocers are featuring Maxwell House coffee at lower prices. Yes, now the coffee with that wonderful good to the last drop flavor comes to you at the lowest prices in months. And it's a mighty welcome occasion for everyone. You, your grocer, and for us Maxwell House people, too. You see, we have one aim with our coffee to bring you the most in flavor and enjoyment every pound you buy. To bring you truly good coffee at the lowest possible price. That's our way of doing business. And it seems you folks like it. You've made our Maxwell House coffee, America's favorite brand. Because you can count on that famous flavor every cup you pour, because you find more flavor for your money, vacuum packed in that familiar blue tin. And at today's prices, Maxwell House is more than ever today's coffee buy. Look for it. Featured these days at the lowest prices in months. Enjoy the coffee that's always good to the last drop. It's sometime later in Springfield. We don't know exactly how much later, but believe me, it's later than you think. Bud is well on his way to a fluorescent cap and socks. Betty has her dress. And Kathy, well, Kathy has her pelicans. Put them all together in what his father got. That's right. A headache like this. Oh, yes, Mrs. Lawson. We guarantee all our babysitters. Bud, will you please get off the phone? Just a minute, Mrs. Lawson, please. Dad, Mrs. Lawson is a brand new customer. I have to be polite to her. I would like to use the phone. You bet, Dad. Hello, Mrs. Lawson? No, it was just my father. God. Well, I can give you Joe Phillips. He's one of our best babysitters. Bud. Oh, yes, ma'am. He's as good as any girl babysitter in town. Bud. That's eight o'clock Friday. Okay, Mrs. Lawson, he'll be there. And thank you very much. Isn't that wonderful, Dad? That's three for just Friday. Let me have the phone. But I have to call Joe before he goes out. Let me have the phone. Holy cow, Dad. This is business. I wouldn't care if it was... Oh, no. Excuse me, Dad. Anderson Babysit is incorporated, but Anderson's... Now I've heard everything. Oh, sure. He's right here. It's for you. Whoa. How did that ever happen? Please don't talk too long, Dad. I've got important things to do. Yes. Hello? Oh, yes, Mr. Bryant. Kathy's principal? Uh, just a minute, please. Yes, Dad? Go away. But I have to... Go away. Good gravy. Try to run a business around this place. I'm sorry, Mr. Bryant. I was just telling my son... They have? They were? She did? Well, I... Well, naturally, you would be upset. Yes. Of course. Oh, I promise you, it'll never happen again. Thank you, Mr. Bryant. And thank you for calling. Good night. Kathy! Did you call me, Daddy? I just got a call from Mr. Bryant. The principal? The principal. And he tells me you've been defacing public property. I have? Well, haven't you? Oh, no, Daddy. I wouldn't do a thing like that. Mr. Bryant said you've practically ruined the entire west wall of the school. Oh, I didn't know I did was write signs on it. Kathy. Well, you told me to advertise, Daddy. I just wrote slogans about the pelicans. I told you to write little signs on paper. I didn't tell you to put chalk marks all over the school. It wasn't chalk, Daddy. It was paint. Oh, I don't believe it. I didn't think anybody would mind. And I sold practically all the pelicans. Pelicans. Morning, noon, and night pelicans. Kathy, why do you do things like this to me? But it was my best printing, Daddy. I thought you'd be proud of me. I've got the pay to have the whole wall cleaned. Maybe if I told Mr. Bryant I was sorry. That'll do a lot of good, telling him you're sorry. Mr. Anderson? Good evening, Officer. Now what have we done? May I come in? Oh, sure. Just come on in and make yourself at home. Thank you. Miserable night, isn't it? Yes, miserable. Don't go away, Kathy. This probably concerns you. Yes, Daddy. Go ahead, Officer. Mr. Anderson, we understand that you're operating an employment office at this address. That's my son. But the king of the boy babysitters. I guess it isn't anything I did, huh, Daddy? All right. Go back to whatever you were doing. Thank you, Daddy. Mr. Anderson, your son has been operating an employment office charging a fee of 10 percent and he has no state license. A license? Well, they're just kids. I know, but the state law says... Officer, all he did was organize the boys of the neighborhood as babysitters. And they pay him a fee. That's the point, Mr. Anderson. They pay him a fee. And in order to collect that fee, he must have a state license. There's a fine of $500. But... While we naturally feel... But... You want me dead? Come in here. Now, there's no reason to be upset, Mr. Anderson. I'm sure you can explain to the judge. Judge? You mean I have to go to court? Well, according to the law, I have to give you a summons, but I'm sure that if you explain... But... What's the matter, Dad? You name it and we've got it. Daddy puts paint all over the school. You run an employment office without a license. A what? Father! You'll just... Margaret! Monster did to me. I didn't know you needed a license. My new dress. My beautiful dress. Is anything right? That's the question. Nobody said you needed a license to be a babysitter. All I did was sit on the bench at the bus stop and look at it. Pelicans! Thee, Mr. Anderson. I'm coming, Daddy. If the girls can be babysitters and they don't need a license, why do we need a license? You were running an employment agency. You were getting jobs for people. Just babysitting jobs. Did you want me, Daddy? Kathy, I... I could strangle you. I didn't do anything. Look at your sister's dress. My pelicans, it worked. Mr. Anderson, I... Don't they look wonderful, Daddy? And it was my idea. The first time I've even had the dress on and look at it. Kathy, what was your idea? Well, drawing pelicans on the benches with chalk. Then when the people sit down, they get pelicans on them. Oh, Kathy. And then when they walk around, everybody sees them. Isn't that the way you advertise? Kathy, go up to your room. But you said if I wanted to sell pelicans... I said go up to your room. Gee, baby sat for three whole nights and look at my dress. Betty, stop acting like an infant. It's only chalk and we'll have it cleaned. I was going to wear it tonight. You can wear something else. Jumping creepers. I'll talk to you in just a minute. Officer... Mr. Anderson, let's just forget the whole thing, shall we? But you said... About the summons? Brother, if I gave you a ticket, I couldn't sleep tonight. Couldn't, huh? I don't have any kids of my own. Would you like some of mine? No. I was down at the station that everything's all taken care of. Well, thank you, officer. And there won't be any more of, you know... The employment office folded five minutes ago. That's the idea. Thanks again, officer. Good luck, Mr. Anderson. And brother, you can use it. Good night. How did I know you needed a license? Nobody told me. Jim, what on earth is going on down here? Margaret, five minutes ago, I found my way to jail. What? A $500 fine. That's what it was going to cost me. Jim, that's impossible. How could a little girl like Kathy... It wasn't Kathy, it was Bud. Bud? Oh, Bud, how could you? How could I find it? Look, Margaret, it's all over now, so let's just forget about it. But the next time anybody says babysitter or pelican or... Bud, what have you got in your hand? In my hand? What is it? What is it? Bud, will you stop repeating everything I say? What have you got in your hand? A pelican. Take it outside and burn it. But it's Kathy's, Dad. I found it on the dining room table. I said burn it. Put it in the incinerator. I never want to see another pelican as long as I live. But this is the last one. Good. Let's make sure it is the last one. Mom, do as your father tells you, Bud. You mean a perfectly good pelican? Bud? Holy cow. Jim. Honey, we've got to do something about the children. I know, dear, but... When you get to the point where a policeman feels sorry for you... Well, that's the end. Jim, I don't know what you're talking about, but I do know that you've broken Kathy's heart. Well, that isn't where she's going to be affected next. She's been working very hard and she's been doing it all for you. For me? That's what she said. She didn't have to paint signs all over the school to sell eight puny little pelicans. Eight? She's already sold eight dozen. What? Eight dozen. And she's getting more orders every day. But why? What can she possibly do with that many dolls? She's going to sell them to and give you the money to pay for the damage she's done. Well... Frankly, I think you owe her an apology. Margaret. Kathy. Honey, you don't know what she's been doing. She smeared paint all over the school. Do you want me, Mommy? Your father has something to say to you, dear. Okay. She put chalk pelicans all over the bus benches. And if I have to pay for cleaning the clothes of all the people who sat down on them... Here I am. Go ahead, Jim.  Oh, I have, Daddy. I've sold ninety-five. And when I sell one more, I'll be all finished. Well, that's fine, dear. I, uh, I'm very proud of you. I've got almost forty-eight dollars. And when I sell the last pelican... Gosh! Well, what is it, Kathy? My pelican. What happened to it? Kathy. But I just saw it. It was in the dining room. And I went upstairs. We won't worry about that pelican, Kathy. I'll give you the fifty cents and we'll consider the whole matter closed. Oh, Daddy, you have to help me. I have to find the pelican. Sweetheart, one little pelican isn't that important. This one is. That's where I hid all the money. The coffee section in your grocery store. That's the place to see welcome news these days. In stores everywhere, grocers are featuring Maxwell House at lower prices. Now all you folks who drink Maxwell House every day can enjoy it at the lowest price in months. And for those of you who haven't been getting that wonderful good-to-the-last-drop flavor, now's a good time to open up a familiar blue Maxwell House tin and start enjoying truly good coffee. Find out how much real pleasure comes to you with that famous flavor. Flavor you can count on every cup you drink. For we'll never compromise on the quality of a single pound. Tomorrow, look for Maxwell House at the lowest prices in months. Now more than ever, today's Coffee Buy. You'll get your money's worth and more with Maxwell House coffee because it's always good to the last drop. Digging into our book of Proverbs, we find a happy facet to our sad little tale. All is well that ends well. That's what they say. And so let's listen in at the Anderson Breakfast Table to a happier conclusion than we might imagine. Like this. Jim. Margaret, I'm in no mood to discuss anything. May I please be permitted the pleasure of suffering in silence. But Bud has something very important to tell you. Oh, yes, sure it must be. Well, go ahead, Bud. Dad, I'm sorry I disobeyed you. About what? About the pelican, but gosh, I didn't see any sense of burning a perfectly good thing like that. What? He didn't burn it, Jim. He didn't? I found it up in his room. With the money? Yes, dear. Oh. I'm sorry, Dad. Bud, don't ever disobey me again. No, Dad. It's the worst thing a boy can do. I know, Dad. When I give you an order, I expect it to be carried out. Yes, Dad. So here's two dollars.  On the other hand, there's another proverb. Everything hath an end, it says, and a pudding hath two. Kathy, it would seem, is the other end of the pudding, like this. Daddy! Kathy, don't you think you've caused enough trouble around here for the time being? Oh, but this isn't any trouble, Daddy. This is just wonderful. I won the prize. What prize? I sold more pelican banks than anybody in Springfield, and they gave me the grand prize. That's fine. And you know what it is? A real live pelican! Tell the kids it's a hot favorite with hoppy. That's hot post-tweet meal. You shouldn't let youngsters go to school without a hot cereal these mornings. So just tell them how hop along Cassidy goes for hot post-tweet meal and watch those kids gobble it up. Post-tweet meal has a rich nut-like flavor. It's crammed with solid nourishment. It cooks in just three and a half minutes. And tell the kids it's hoppy's favorite. You'll see, you'll all agree, it's the best hot cereal you ever ate. Join us again next week when we'll be back with Father Knows Best, starring Robert Young as Jim Anderson, with Roy Bargy in the Maxwell House Orchestra, and yours truly, Bill Foreman. So until next Thursday, good night and good luck from the makers of Maxwell House, America's favorite brand of coffee. Always good to the last drop. Father Knows Best was transcribed in Hollywood and written by F. James. Now stay tuned in for Dragnet, which follows immediately over most of these stations.


We can start in, and we posted this in free public places, right? And on the website, an emailed interested party. So we're following the open meeting law. Welcome, Mrs. March 27, 2023. And first, we are starting with a special town meeting to discuss the town of Rochester voting whether or not to authorize cannabis retailers. And integrated licenses, and this is vote will be on May 2nd. And we're here to sign the warning on that and ask any questions if anybody has any comments or just to put it out there that this we've had a request to authorize someone to have a retail establishment. And the first step of that is for the town to vote whether or not it wants to host that or not. So, any comments, and Nancy? Depending on how the vote goes, if it goes yes or no, does it start the process immediately? Or are there regulations that retail establishment can be? Well, if it's no, then it's just a non-starter. If it's yes, then it goes to the planning board to see if the location that has been suggested is going to work or not. It can't be near a school, either. No, it's law. And the state, the process of the state, which I said four months or something? Yeah, so this is the first hurdle. And as they approached the planning board about this and said, well, the first step since the town has not had this issue come up and had a determination on it, this is the first step. Are there any other surrounding towns that you know of that have Randolph, Pankock, Campbell Pass? Randolph passed it to, I think they all passed. Yeah, Randolph's up and running. Yeah, Randolph up and running. Okay, yeah. Martha has a question. Hi, my question is now, obviously, I heard just heard somebody say it obviously can't be near a school. That makes sense. But you said that they have to go to the planning board to see if their location that they're suggesting will work. Do you, can you say anything about where the location is? Is it in the village or is it in another part of town like North Hollow? Or can you say that? It's not in the village and it's, I think that we just go with this first step just to have the discussion whether or not the town will approve it or not. Yeah. Okay, so am I correct that it's going to go on the issue is going to go on the same ballot as the school vote, right? No, it's a separate ballot, but at the same time. In time, excuse me. Okay, excuse me.  And now are they voting on the school vote by Australian ballot? Is that why I'm, am I understanding that correctly? We're going to talk about that a little bit. We're going to talk about that later. Okay. Okay, excuse me. And okay, so if it's approved, then they go to the planning board to see if the location will work and then it goes from there. Yeah, yeah. Okay, thank you. Sorry to bother you. But does it go back to the state? Does the state have to go through this? I haven't even been to the state yet. This is, this is the first step for them. Yeah, the first hurdle, yeah. Is there a license involved like a liquor license? Yes. Through the state. It's a lengthy process for getting their state license.  Do we do that? Is it a town or is it just the state? It might be just through the state. It's through the cannabis control board, but there is like a fee that the town does receive. We currently receive for a different type of license. This is an opt-in. So there's only out of like six different licenses that you can apply for. Two of them require this opt-in vote. The rest of them can just go through the cannabis control board through the state. And they, and because of that, the town will receive $100 for each license in our town. Can you say who it is? Is it somebody local? You can get them, you can look online because they're under different names. So I don't, I wouldn't know. No, just who's ever asking? They're, yeah, they, yeah, they do live in town. Okay. Yeah. And it should also be noted while we're having this discussion that there was a point in time when some of the tax generated from retail sales was going to be distributed back to the towns. But we understand that did not happen. So the town will not be receiving any of the tax dollars from the cannabis, it's a 14% cannabis tax. And then they'll also have to charge the 6% sales tax, so it's a 20% tax. It all goes to the state and we get $100 from the fee. Is there something about federal, the federal government being involved in this too, that the banks can't accept money from this? It's not legal on a federal basis. So you, that's true. You know, if it's, if the bank is FDIC insured, they can't do things that are still considered illegal. You also would not, you're not supposed to go on to the Forest Service land and smoke pot. So you do have to pay attention to federal guidelines. Interesting. Martha, can you not hear Pat when she speaks? Okay, maybe we can move that closer to you. I don't know, but it seems like, whoa, and you're on. How was that? How does this sound, Martha? Could you say something? Do you have a question, Martha? She's on, now you're on mute, Martha. We can't hear you. This is fun. I could hear everybody else fine. I just couldn't hear Pat. I'm sorry. I'm sorry too, with this ongoing problem we're having. You're still very, very faint. I can barely, I can't, I can barely hear you. If that thing gets any closer. I'm not kidding. It's weird. You're fine. Everybody else is fine. I can hear Nancy fine. She's behind you, but I can't hear Pat. Well, she's has a soft voice. She just has this dandy voice, I guess. So, okay, on a federal basis, cannabis pot is not legal. Therefore, banks can't take money for retail sales because it would be considered an illegal activity.  On a federal basis, cannabis is not legal. It's only legal on the state basis. And the state issued, as far as I know, I understand that the state has a 14% cannabis tax that they charge, and then they charge the 6% state sales tax. So, when you buy cannabis, you're paying 20% tax, and none of that comes back to the town. I just think the town should know that when they go to the voting booth. She's nodding. I do believe she heard some of that. Anybody else have any questions, comments? Sounds like a lot of hoops. A lot of hoops. And if it comes up for a license every year and you can deny it, no brainer to me. No. But does the town have to approve the license, or is it the cannabis board? We don't know. You've got a lot of data there, but I don't know if. We have a lot of data there, but I don't know if it covers that part. But we could look into it. I think that's wise, because if you have the control over it, the select board has control over it. The initial thing is like, it's just another business. Well, the town could vote, again, to repeal that permission in the future, just because you decide to allow it doesn't mean that's forever allowed, if it doesn't seem. So the liquor license thing is in the town? Yes. They still have to satisfy state. Per view. OK. Yeah, state requirements. But yeah, just clarifying it. We have a lot of learning to do. Yeah. And is that it? To start this conversation, I think, we'll just put it out there on the radar so people know that this vote is coming up and what it's about, and if there's no more comments on Zoom or in the room, I'd move to close this special town meeting.


This is the first question and the second question, this is the harder one. And the simple one is, any chance of seeing in your future your debates with Ben Shapiro or anything like this, yes, because he's a big fan of this, so Judeo-Christian morality Ben, if you're listening, he said that he's interested in debating, so I think it's going to happen and it'll be a lot of fun because he's smart and it'll be interesting to see what he says when I say what I say. I don't think he has an answer, but that's the reality, but he's smart, so I think that's going to happen. Again, I don't know when, and I don't know if, let's assume he doesn't chicken out. I certainly will not, that I will guarantee you. In terms of cronyism, so first, there's no such thing as crony capitalism. Never use it, never talk about crony capitalism. Cronism is a feature of statism. It's crony statism, but all statism is crony. There is no non-crony statism. Now why is that? Capitalism, as I said earlier, the feature of capitalism is a separation of state from economics. If you separate state from economics, there can be any cronyism because the state has no control over business and business has no interest in the state because the only interest this business has in the state is what? It's the fact that the state has control over the business. I'll give you a great example. In the mid-1990s, Microsoft, Bill Gates is Microsoft, you know how much they spent on lobbying you know, crony activity in Washington D.C.? How much money did Microsoft spend? Zero. Yeah, exactly zero dollars. Nothing. No cronyism. No lawyers, no presence in Washington D.C. Nothing. Nada. They left Washington alone and the politicians got really pissed off because in those days Microsoft was the biggest company in the world based on market cap. Largest company in the world and they were not lobbying or put it another way. They weren't bribing any politicians, right? They were really rich and they weren't handing out anything. So Microsoft was invited to Congress. Did you see Zuckerberg in front of Congress? Yes. So this is the mid-1990s. It was Microsoft in front of Congress. And the Microsoft executive sat there and Congress, the congressman said, you have got to start lobbying. Indeed, this Republican, actually a Republican who was in the hearings with Zuckerberg, he's that old, from Utah, right? A senator from Utah, he stood up. He's a Republican. He stood up and he yelled at them. You guys have to start lobbying. You need to have a presence in Washington D.C. You need to build a building in Washington D.C. You need to have a presence here. In other words, he said, you've got to start bribing me. He didn't say it that way, right? You don't do that in America. And Microsoft said, we're not interested. You leave us alone. We will leave you alone. We're not interested in lobbying. We're doing fine. We're the biggest company in the world. What do we need you for? All right, they went home to Seattle. Six months later, knock on the door. We're from the Justice Department and we're here to sue you for antitrust violations. Now, what was the crime that Microsoft committed that violated anti-fragulations? They were doing what? What were they doing? Anybody old enough to remember? They were giving away internet explorer for free. For free? How dare they? Because I was using Netscape. You don't know even who Netscape is, right? Netscape was a browser. You downloaded and you had to pay $79 to use a browser. It was the only browser. There were a few browsers. You had to pay for all of them.  But before, you paid for everything on the internet. So I used to pay $79 for my browser. Microsoft said, we're going to give it to you for free. So Netscape went to the government and said, this isn't fair. Unfair trade practices to give away something for free. It's called dumping, right? And that's why they were sued. But really, the reason they were sued was what? They weren't lobbying. Guess how much Microsoft spends today? They spent 20 years on the government supervision. That was the agreement. They had a government bureaucrat at Microsoft supervising every decision they made for 20 years. A few years ago, finally, the guy left, right? They left. So how much do you think they lobbied today? They spent tens of millions of dollars. Tens of millions of dollars. They built this wonderful, beautiful building in the middle of Washington, D.C. about equal distance from the White House and from Congress. And everybody can see it's the Microsoft building. We are here. And now, they leave them alone. Google, from day one, has been spending the money around because they saw what happened to Microsoft. And they know. This is how you stay clean. Facebook hasn't bribed the right people. I mean, they don't call it bribes in America. They call it lobbying and they call all kinds of things. But it's the same thing. And all of this because the government has control. It starts with politicians. It starts with statism, regulations and controls. So if you want to get rid of cronyism, you have to get rid of government regulations, government controls. The smaller government is, the less cronyism there is. So cronyism is a feature of the big state. Cronyism is a feature of statism. It's not a feature of capitalism. Capitalism is the elimination of cronyism. So I would argue, I would never use the term crony capitalism. It's crony statism. Or cronyism. I use cronyism. It's the system where the state and business are hand in hand. But that's because the state has guns and business has not much choice. And then it figures out that it can manipulate the system. It figures out that it can benefit from having an ally with guns. But that's not how businessmen make their money, you know, in a free market originally. That's a consequence of government power.


From around the globe, it's theCUBE with digital coverage of AWS re-invent Executive Summit 2020 sponsored by Accenture and AWS. Welcome everyone to theCUBE's coverage of Accenture Executive Summit here at AWS re-invent. I'm your host, Rebecca Knight. For this segment, we have two guests. First, we have Helen Davis. She is the Senior Director Cloud Platform Services Assistant Director for IT and Digital for the West Midlands Police. Thanks so much for coming on the show, Helen. Welcome. And we also have Matthew Pound. He is Accenture Health and Public Service Associate Director and West Midlands Police Account Lead. Thanks so much for coming on the show, Matthew. Thank you for having us. So we are going to be talking about delivering data-driven insights to the West Midlands Police Force. Helen, I want to start with you. Can you tell us a little bit about the West Midlands Police Force? How big is the force? And also, what were some of the challenges that you were grappling with prior to this initiative? Yes, certainly. So West Midlands Police is the second largest police force in the UK outside of the Metropolitan Police in London. We have in excess of 11,000 people work at West Midlands Police serving communities through across the Midlands region. So geographically, we're quite a big area as well as well as being population density having that at a high level. So the reason we sort of embarked on the data-driven insights platform, which was a huge change for us, was for a number of reasons. Namely, we had a lot of disparate data which was spread across a range of legacy systems that were many, many years old with some duplication of what was being captured and no single view for officers or support staff. Some of the access was limited. You had to be in an actual police building on a desktop computer to access it. Other information could only reach officers on the front line through a telephone call back to one of our enabling services where they would do a manual checkup and look at the information, then call the officers back and tell them what they needed to know. So it was a very long, laborious process and not very efficient. And we certainly weren't exploiting the data that we had in a very productive way. So it sounds like, as you're describing, an old, clunky system that needed a technological re-imagination. So what was the main motivation for making this shift? It was really about making us more efficient and more effective in how we do business. So certainly as an IT leader and some of our operational colleagues, we recognize the benefits that data analytics could bring in a policing environment, not something that was really done in the UK at the time. We have a lot of data. So we're very data-rich in the information that we have, but we needed to turn it into information that was actionable. So that's where we started looking for technology partners and suppliers to help us and sort of help us really with what's the art of the possible. This hasn't been done before. So what could we do in this space that's appropriate for policing? Helen, I love that idea. What is the art of the possible? Can you tell us a little bit about why you chose AWS? I think really, as with all things, when we're procuring a partner in the public sector, there are many rules and regulations quite rightly as you would expect there to be because we're spending public money. So we have to be very, very careful and it's a long process and we have to be open to public scrutiny. So we sort of looked at everything that was available as part of that process, but we recognize the benefits that cloud would provide in this space because without moving to a cloud environment, we would literally be replacing something that was legacy with something that was a bit more modern. That's not what we wanted to do. Our ambition was far greater than that. So I think in terms of AWS, really it was around scalability, interoperability, things like the disaster recovery service, the fact that we can scale up and down quickly. We call it dialing up and dialing back. It's page go. So it just sort of ticked all the boxes for us and then we went through the full procurement process. Fortunately, it came out on top for us. So we were able to move forward, but it just sort of had everything that we were looking for in that space. Matthew, I want to bring you into the conversation a little bit here. How are you working with the West Midlands Police? Sorry, and helping them implement this cloud-first journey. Yeah, so I guess our journey with West Midlands Police started over five years ago now. So we set up a partnership with the force and wanted to operate in a way that was very different to a traditional supplier relationship. So through that, the Data Difference Insights Program is one of many that we've been working with West Midlands on over the last five years. As Helen said already, cloud gave a number of advantages, certainly from a big data perspective and the things that that enabled us to be. And from an extensive perspective, that allowed us to bring in a number of the different teams that we have, so cloud teams, security teams, interactive from a design perspective, as well as the more traditional services that people would associate with Accenture. I mean, so much of this is about embracing comprehensive change to experiment and innovate and try different things. Matthew, how do you help an entity like West Midlands Police think differently when there are these ways of doing things that people are used to? How do you help them think about what is the art of the possible, as Helen said? There's a few things for that. What's been critical is trying to code create solutions together. There's no point just turning up with what we think is the right answer, trying to collectively work through the issues that the force are seeing and the outcomes they're looking to achieve rather than simply focusing on a long list of requirements that think was critical. And then being really open to working together to create the right solution, rather than just trying to pick something off the shelf that maybe doesn't fit the force's requirements in the way that it should do. Right, it's not always a one-size-fits-all. Absolutely not. What was critical is making sure that we're creating something that met the force's needs in terms of the outcomes they're looking to achieve, the financial envelopes that were available and how we can deliver those in a iterative agile way rather than spending years and years working towards an outcome that has gone out of date before you even get there. So Helen, how are things different? What kinds of business functions and processes have been reimagined in light of this change and this shift? It's actually unrecognizable now in certain areas of the business as it was before. So just to give you a little bit of context, when we started working with Accenture and AWS on the Data and Insights program, it was very much around providing what was called locally a whizzy tool for our intelligence analysts to interrogate data, look at data, decide whether they could do anything predictive with it. And it was very much sort of a back office function to sort of tidy things up for us and make us a bit better in that area or a lot better in that area. And it was rolled out to a number of offices, a small number on the front line. And really it was in line with the mobility strategy that we had where officers were getting new smartphones for the first time to do sort of a lot of things on policing apps and things like that to again to avoid them having to keep driving back to police stations, et cetera. And the pilot was so successful, every officer now has access to this data on their mobile devices. So it literally went from a handful of people in an office somewhere using it to do sort of clever whiz bang things to every officer in the force being able to access that level of data at their fingertips literally. So what they would have to have done before is if they needed to check an address or check details of an individual, just as one example, they would either have to, in many cases, go back to a police station to look it up themselves on a desktop computer, or they would have to make a call back to a centralized function and speak to an operator, relay the questions, either wait for the answer or wait for a callback with the answer when those people are doing the data interrogation manually. So the biggest change for us is the self-service nature of the data we now have available. So officers can do it themselves on their phone wherever they might be. So the efficiency savings from that point of view are immense. And I think just parallel to that is the quality of our data because we had a lot of data, but just because you've got a lot of data and a lot of information doesn't mean it's big data and it's valuable necessarily. So again, it was having the single source of truth as we call it. So you know that when you are completing those searches and getting the responses back, that it is the most accurate information we hold. And also you're getting it back within minutes as opposed to half an hour, an hour or a drive back to a station. So it's making officers more efficient and it's also making them safer. The more efficient they are, the more time they have to spend out with the public doing what they, you know, we all should be doing. And have you seen that kind of return on investment? Because what you were just describing with all the steps that would needed to be taken in prior to this to verify and address say, and those are precious seconds when someone's life is on the line in sort of in the course of everyday police work. Absolutely, yeah, absolutely. It's difficult to put a price on it. It's difficult to quantify, but all the, you know, the minutes here and there certainly add up to a significant amount of efficiencies savings. And we've certainly been able to demonstrate the officers are spending less time at police stations as a result and more time out on the frontline. Also they're safer because they can get information about what may or may not be out and address, what may or may not have occurred in an area before very, very quickly without having to wait. Matthew, I want to hear your observations of working so closely with this West Midlands Police. Have you noticed anything about changes in its culture, in its operating model, in how police officers interact with one another? Have you seen any changes since this technology change? I think what's unique about the West Midlands Police is the buy-in from the top. You know, from the chief and his exact team and Helen is the leader from an IT perspective. The entire force is brought in to what is a significant change program and that trickles through everyone in the organization. Change is difficult and there's an awful lot of time effort that's been put into both the technical delivery and the business change and adoption aspects around each of the projects. But you can see the step change that it's making in each aspect of the organization and where that's putting West Midlands Police as a leader in the technology I'm policing in the UK and I think likely. And this is a question for both of you because Matthew, as you said, change is difficult and there is always a certain intransigence in workplaces about this is just the way we've always done things and we're used to this and don't try to get us to do anything new here. It works. How do you get the buy-in that you need to do this kind of digital transformation? I think it would be wrong to say it was easy. And we also have to bear in mind that this was one program in a five-year program. So there was a lot of change going on, both internally for some of our back office functions as well as frontline officers. So with DDI in particular, I think that the step change occurred when people could see what it could do for them. You know, we had lots of workshops and seminars where we all talk about, you know, it's big data and it's going to be great and it's data analytics and it's transformational, you know, and quite rightly people that are very busy doing a day job that not necessarily are technologists in the main and, you know, particularly interesting, quite rightly so in what we are not doing with the cloud, you know, and it was like, yeah, okay, it's one more thing. And then when they started to see it on their phones and what teams could do, that's when it started to sell itself. And I think that's when we started to see, you know, to see the step change. You know, and if we have any issues now, it's literally, you know, our help desks in meltdown because everyone's like, well, we can't manage without this anymore. And I think that speaks for itself. So it doesn't happen overnight. It's sort of incremental changes and then that's the step-changing attitude. And when they see it working and they see the benefits, they want to use it more. And that's how it's become fundamental to our policing by itself really without much selling. Matthew, Helen just made a compelling case for how to get buy-in. Have you discovered any other best practices when you are trying to get everyone on board for this kind of thing? We've used a lot of the traditional techniques, things around comms and engagement. We've also used things like 30 day challenge and nudge theory around how can we gradually encourage people to use things. I think there's a point with all of this around how do we just keep it simple and keep it user-centric from an end user perspective? I think DDI is a great example of where the technology is incredibly complex. The solution itself is extremely large and has been very difficult to get delivered. But at the heart of it is a very simple front-end for the user to encourage it and take that complexity away from them. I think that's been critical through the whole piece of DDI. One final word from Helen. I want to hear, where do you go from here? What is the long-term vision? I know that this has made productivity savings equivalent to 154 full-time officers. What's next? I think really it's around exploiting what we've got. And I use the phrase quite a lot, dialing it up, which drives my technical architects crazy. But because it's apparently not that simple. But we've been through significant change in the last five years and we are still continuing to bed all of those changes into everyday operational policing. But what we need to see now is we need to exploit and build on the investments that we've made. In terms of data and cloud specifically, the next step really is about expanding our pool of data and our functions so that we keep getting better and better at this. And the more we do, the more data we have, the more refined we can be, the more precise we are with all of our actions. We're always being expected to, again, look after the public purse and do more for less. And I think this is certainly, and our cloud journey and cloud-first by design, which is where we are now, is helping us to be future-proofed. So for us, it's very much an investment. And I see now that we have got it embedded in operational policing. For me, this is the start of our journey, not the end. So it's really exciting to see where we can go from here. Exciting times indeed. Thank you so much, Lily, Helen and Matthew for joining us. I really appreciate it. Thank you. And you are watching theCUBE. Stay tuned for more of theCUBE's coverage of the AWS re-invent Accenture Executive Summit. I'm Rebecca Knight.


Wow.  Wow. Thank you for stroking my ego. So I am Render Man, if you don't know me, for the three of you. I've been going to Def Con for ten years, asked to see my liver, and I'll show you the scars. I love this place. These guys are family. This is the only place I feel normal, and that should say something. So this talk is where I started getting a little more literal with my titles rather than going for, you know, analysis of yadda yadda yadda or anything like that. It's like, how do I poem the Let Me Count the Ways? This is actually a talk born out of a conversation I had at Sektor in Toronto. Good convention, everybody should go. Workforces are becoming increasingly mobile and depending on wireless devices. You know, you've all seen them in airports, hotel lobbies, you know, you probably see a whole swack of them out here in Vegas. You know, the guys that are always typing or on the phone or something like that, well, this is common now. Everybody, every company's got somebody like this.   As much as you'd like to, they're, you know, halfway around the world. You're investing vulnerabilities into your networks.  You run into these situations. This is by no means an exhaustive list and in the interest of full disclosure, there isn't really anything brand new in terms of wireless information if you're expecting some zero day or anything like that. It isn't, but it's, there's a lot of stuff in here you probably haven't thought of or would probably just wish you could club into your mobile workforces. So this is Bob.  Works for Widgets International, selling widgets, amazingly. Bob travels to customer sites all over the world, and he's your worst IT nightmareer. He likes to think himself tech-savvy, you know, you've all got this kind of guy in the office. He just knows enough to get access to porn. That's it.  You've all run into people like this. Don't deny it. He's also the worst case scenario in this situation. He's not, he is the dumbest, the worst offender possible, and we're going to spend the better part of the next hour making his life miserable. Amen. So let's poem Bob. We're not going to touch Bob. We're not going to goon rush him or something like that and take his wallet and beat this nut out of him or anything like that. We're just going to look at the various, you know, wireless and things at a distance that a business traveler or, you know, traveling person would probably have on them. I.e. Boardhacker on a trip to Norway for a wonderful conference called Sektor, or HackCon. If you spend any amount of time in airports, you always see these people and when I get bored, I don't know what you, but when I get bored, bad things tend to happen.  He is sitting there in the departure lounge and there's all these other laptops and Bluetooth devices around. It's just, it's enticing. So we'll start off with Wi-Fi, something I'm assuming you all have probably used this weekend. Bob has a laptop with tons of private info for the company. He connects to hotspots at airports, you know, hotels, Starbucks, wherever. You know, the kind of guy that just likes to get on the internet, find out the latest headlines from CNN, stock quotes, whatever, check his email. He'll connect to anything and especially things like the Glinksys Global Network that I'm sure you're all very familiar with and have possibly used on your trips here. I will remind you that is a felony theft of service, but that's not why we're here. In general, hotspots found in public places do not encrypt. Clear text is clear to read and also clear to inject in a lot of cases. So things like replacing images, scripts, oh look, I see a DNS request. What can we do with that? If you're not using a VPN or some sort of layer of encryption between yourself and the access point, you are vulnerable. Be it through just plain text leakage of information, you know, somebody's reading your email as it's being sent to, you know, malicious injection and alteration of your data both ways. If you don't have a firewall, think about this. You're connected to an access point. How many of you have access points in your home? You know, wireless networks in your home. Cool. You all know that you're on the local LAN. You are connected to every other computer on that local LAN. Now, if I'm connected to the hotspot in a hotel, oh gee, I'm on the local LAN.   And it is absolutely terrifying. You just connect to the network and it's like, oh, look, I've seen SMB shares. Man in the middle attacks, password snarfs are all possible. If you don't use any level of encryption last year. I think there was something like 20 access points for the hotel network here. During the con, there was like 150. This is a gambling town, but I don't know what those odds are for gambling is to connecting to the real one, do the fake one, real one, the fake one. Which is it that you're connecting to? And like I said, the attacker's on the local network. So you've just removed an awful lot of barriers.  I'm sitting right next to you. Driver vulnerabilities are especially entertaining because, well, think about it. You don't even need to have your computer connected to something in order to be vulnerable. I could just send a magic packet, your driver explodes, it gives me root access. It's kind of a terrifying thought. So when I was in Norway, I wanted to see, back in February, I wanted to see, I was bored, had the most horrible flight over, had just been up for like 48 hours straight, and I'm like, okay, I'm at a security conference. Let's see what kind of security knowledge we have here. Do people actually take what we've been ranting about for years to heart? Do they actually do it?  So what I did was I set up AirPone, and for those of you who don't know, AirPone is a wonderful little tool that, let's see, you're getting Google's homepage. You get the HTTP request for the main image, and when the remote server gets added, it sends back the image. Well, if I'm in the room, I can respond as the access point faster than Google's request would come back. So the client gets this image really quick, and oh, these other packets coming in, I'll just drop those. So I'm able to substitute my own image in place of Google's main logo. This debuted at DEF CON 10, 11, somewhere around there, basically where people were replacing images on the fly with GoatSea, and you can imagine that one. They've actually got some really good pictures and video up on their site. So I'm at this conference, a couple hundred people. They've got three wireless networks, an open one, a web-encrypted one, and a WPA one. So I started playing with the open network. So I started injecting an image to see if people were observant. I don't know if you can tell, but that's a 10-foot kayak and a 12-foot shark.  I hope that's not a rental because I don't think he's getting his deposit back. So I started injecting this image, and looking over people's shoulders, and they'd fire up a new site and all the little thumbnails would be this image. But it was small. You couldn't really tell what it was. Some people were like, somebody got a shark fetish or something. What's going on here? There was some general confusion, but nobody was really getting it. So kind of I had to up the ante here. When I'm bored, tired, and partially drunk, I get creative. So I started injecting this to see if anybody would actually figure it out. They got it. They figured it out. And it didn't get punched. I did have a few people running up to me with their corporate webmail with my face in the corner. They're like, what did you do to my laptop? What did I do to you? Why are you hacking me? Why are you giving me such grief? I didn't do anything to you. And I'm like, come to my talk. I'll explain then. And it was just highly entertaining because there's a room with vendors and they're going to show their stuff and they go to the company site and oh, what the hell? It's just highly entertaining. And I do have to thank the staff for hack on, for not kicking me out or getting overly pissed off with me because I was making a point and I think most people got it. Other threats to Wi-Fi that you're going to run into more and more at airports and public spaces, things like karma, hot spotter, Metasphalite now has a karma module. Karma is the program that listens for beacon requests from clients and says, oh, that's me. And literally makes your laptop the universal access point. It responds to any request to connect, no matter what the SSID is. Simple Nomad a few years ago gave a talk at Shmoon Chronicle hacking the friendly skies, which was highly entertaining. And he basically found a way that he could have an access point with beaconing when it couldn't find the network it was looking for or any network it was looking for. It dropped into an ad hoc mode that you could then see the packets coming from an ad hoc. And I don't know if you know I think about unencrypted ad hoc networks, but it's like, what authentication? Oh, that's me here. We're now connected to each other. So he'd sit back in cattle class on these long haul flights and you'd have all these guys in first class they'd be typing away in their ultra expensive laptops and everything like that and be able to look through the curtain and see these guys. So he would connect to them, go through their shares, see what sort of things were there. Oh, look, company memos, company directories. Oh, that's an interesting report. But he upped the ante and would go up, you know, walk up to these guys. And I don't know if you've seen Simple Nomad. Is he here at all? I've not seen him this year. Okay. You know, kind of an old scruffy guy, usually using like a trench coat. Just not your usual typical clean cut businessman. Would go up to these typical clean cut businessman and go, hey Bob, how you doing? How's Julian accounting? Did you get that memo about that new project and these guys would go, what the hell? How do you know all of this? It's just absolutely terrifying these people. Cafe Latte came out, I believe it was last year, which you don't actually have to be connected to a network, or your company network. If you're just sitting in a cafe, I can throw packets at your laptop and tease out the web key without you even being connected to the corporate network.  You know, he probably doesn't have an intrusion detection system on his laptop. So, you know, pull the web key out of there, go attack the company. No web can be broken in like 60 seconds. But still, think about this, that this is something that was, I don't type that fast.  Do that in a coffee shop, in a Starbucks, and people just go, huh, it says I'm connected, now I'm not, yes I am, now I'm not, what?  If your wireless is on, you're probably ponible in one way or another, so if you don't need it, turn it off, and if you do, take some measure. Disable it when not in use. Use a VPN, there's a lot of good free products out there. Assume you're going to be attacked. Don't just think it's all one happy wireless world out there.  Don't trust customer networks either. If Bob's going out to a customer network, and he's connecting in to get internet access to check his email, get, you know, pricing or availability or anything like that, I've heard stories from people that they'll see, oh, new host on the network, oh, go poke it, oh, it's the sales guys, oh, look, open shares, oh, private information about margins on all the widgets and everything and all the term models, that would be interesting to know so we know exactly how much we can, you know, talk him down in his price. Being able to renegotiate contracts based on knowing exactly how much margin is in there. Like, you're connecting to somebody who really, from a business perspective, is hostile to you. They want the cheapest price you want the most. Don't trust them. Turn on the firewall. Even if it's the windows one, just do something. Be cautious. And Roger Dingledein, is he in the room? Nope. Tor Project actually pointed out that in a pinch, Tor can actually be a real life saver because when you're sitting in an airport or something like that, Tor will encrypt from your from your laptop through the access point to have your connection pop out in like, Hungary or someplace like that. But it provides that level of encryption that you can't see what the data going through and inject. So Roger was actually at the same con in Norway when I was doing this experiment and he was hearing all these things like, what the hell's going on with the network? There's all these images in that. He actually had to turn Tor off before he noticed.  Let's go after his phone. How many of you guys have got the sales guys in your office that are always on their cell phone that's like welded to the side of their head?  Bob also loves his Bluetooth headset.  I have a Bluetooth headset. I use it in the car. It's handy.  How pretentious do you have to be? I'm sorry. I have a major hate on for these things. How pretentious do you have to be to be able to answer a phone call in two seconds? He also uses his phone for SMS one-time pads with his bank. This is actually something I'm not familiar with in North America, but I know in South Africa when you do a transaction before, you know, online or through like an ATM or something like that, it will SMS you like a one-time key that you have to enter to complete the transaction. And this is a really sensible system because it's a back channel that an attacker theoretically if they read your card off the ATM are not going to be able to get at your SMSes as well. Bob also keeps company directories in his phone. If it's on and discoverable with Bluetooth, anyone can probably connect or at least see it and initiate trying to connect. Most devices have a pin, however, how many of you have changed your default pin on your Bluetooth phone? How many of you haven't?  Give me time. Paired devices, if you can either force pairing, trick pairing or just have a legitimate reason to pair, oftentimes you can get access to the whole device, not just the little segment you're trying to, oh, okay, I'm trying to exchange a business card. Well, why do I have to, why can I do AT commands now to your phone? It should be just OBEX file transfer one way done. Why is there still a trust relationship here? This isn't a specification. It's kind of dumb. You can rewrite SMS messages, change phone book entries, notes, images. There's actually a very interesting case going on with the mayor of Detroit who got caught with several hundred SMS messages between him and his mistress on company time and the newspaper published all of them. His lawyer is actually saying right now that, oh, it was an attacker. Somebody inserted these messages in and, well, that's really possible. It'd be very interesting to correlate the date times that those were sent and if you was sitting in the same place at every time because I get the feeling if he's traveling around you'd have to have somebody following around within a few feet to insert these at different date times. Yeah, I don't think that case is going to hold any water. Some models of phone are actually really interesting in that you can connect to them and you get a, or the ability to connect an RFCOM channel 17 and you can issue AT commands like old style modem, AT commands, you know, ATDT and then a phone number. This is actually not something I've heard of in North America but in Europe there are some issues with premium rate calls like our version is like 1,900 numbers. You can basically connect to somebody's phone and make them dial a number of your choosing. This gets interesting because if I make Bob dial a phone number that shows up on the bill as something that he would never, ever want to have to explain he's just going to pay the bill and make it go away but imagine just putting into his expense report, you know, a nine hour phone call to Madam Whips a lot fun time party line. You could just, even just from a malicious standpoint, you could make somebody's life miserable or something like that. Audio. Have we got audio? Let me try some. And one of the later closest friends, the mobile phone. This is either UK. Audio. Experts have discovered a security flaw in some of the most popular Bluetooth enabled phones that allows others to hack into your handset and take control of it. Our hustlers are going to demonstrate for our hidden cameras just how vulnerable your phone can be to a silent attack. This is the Bluetooth scam. The problem with some Bluetooth phones is you can make a major connection to the phone if you take over the phone. Read the contents out to the phone book. SMS messages even actually make calls with it. So, how does the Bluetooth hack work? Our hustlers are going to demonstrate to here. Jess and Paul's phones have their Bluetooth turned on. Alex uses his mobile pocket PC. Because of a security flaw in Jess's phone, he is able to locate the Bluetooth signal, hack directly into the handset and take control of it. Before he can make a call from Jess's phone to Paul's phone. This technique is known as bluejacking. The Bluetooth hustle works by using the Bluetooth feature on mobile phones. It's a very modern, very useful, very easy piece of technology. They can connect to your phone without you knowing it. Your phone will not even show that it's connected to something. They're now ready to test it on the public. Alex has set his pocket computer to search for Bluetooth signals. Now he's pinpointed a mark with an enabled handset. He can go for the hack. Now he's in and making a call using the Mark's phone. But where's the actual scam here? It's in the number that Alex is calling from the Mark's phone. Our hustlers have set up their own premium rate line where calls are charged at £1.50 a minute. So this guy's having the money taken from his pocket but he's totally oblivious. And if the Mark doesn't realise his phone is calling the hustler's premium line during his journey between Manchester and Liverpool, he will lose £75 in call charges. Ouch! So if Alex spends some time here, he can really take home the vacant. In an hour he manages to target 20 phones with an average call time of 15 minutes, making him a grand total of £500. Our hustlers have proved for those with the ability and the know-how this is a really money-spilling scam. The simplest and best way to protect yourself against this scam is just simply to keep your Bluetooth turned off unless you're actually using it. If you do not switch off the Bluetooth feature on your phone when you're not using it, you're in danger of somebody finding a way in. With apologies to Major Malfunction. I wish you guys could see it from up here but every time I play this video or give this talk there's always Bluetooth threats. Bob's headset uses a default pin. Most headsets don't have a keypad so you can't change the pin number. It's usually like 0000 or 1234 something simple. There's the indexes out there of different manufacturers they know exactly what these are. You can do a blue bump and repair. You basically break the connection between the headset and the phone. You connect your device to it and oh look you can now turn it on. You can also inject audio.  But think about this I go to a bunch of meetings and standard procedures take the phone out turn it off or put it to vibrate or whatever but they'll take it out usually just turn it off but they'll leave the headset on and it's still blinking. Wait a minute you just turn off the phone in a favorable mode. Hmm we now have a bug in the board room. If I can connect with a long range antenna I can now start pulling audio directly from that room conversations you know Lord knows well I really like the idea of injecting audio particularly if you're sneaky enough to lower volumes in that and just screw with people's minds you can make this guy think he's crazy.  I'll start that I'll start that again So what are you doing tonight? Uh Nothing Do you want to go to a party? With me? Yeah We could just go to my place before and hang out Yeah that sounds awesome Hold on How many of you have had a moment like that where you're not sure if there's schizophrenic or on a phone call especially line candid seal and the taste Keystone light is always smooth even when you're not I'm really sorry but what just happened That's cool I was feeling embarrassed Just a second We've all been there that's why I hate these things Okay This this is a new section I'm trying to stay serious here so follow with me Alright Well rummaging through Bob's phone we make a discovery There's just something that should not be wirelessly enabled and I'm sure you will agree with me on this Special SMS messages in his phone with special five or six-digit headers to them You start looking at these things and oh these are all to his Bob's wife We now know the codes We know the phone number of his wife This device in question is not discoverable but who the hell thought this was a good idea Have you figured it out yet? A Bluetooth-enabled vibrator I've tried to stay serious with me here Basically this device pairs with a cell phone and you send it special text messages with a set of digits at the beginning to identify that it's meant to be passed onto the device Each character that you can type in on the phone has a different motion So there's like 700 or hundreds of different motions on this thing You can imagine The toy.co.uk if you really want to check this out but Let's get in your window out of the way I want this to be serious I know you're thinking it I really want to find one on a site survey You don't wait Really? Teledonics is an actual field of research It is But Every example you come across when you start Googling into this stuff There is no security Nobody can tell me on devices like this or any of the other stuff what level of security they've done Nobody's done reviews on this This is something that Hello New field It's an extreme example of a generally bad idea of wirelessly enabling stupid things I mean I'm sorry There are serious implications here I'll give you a moment You okay? Now if I start sending messages to Bob's wife through his phone he didn't send those Is that rape? Seriously Oh honey Thanks for sending me that message It's 7.30 I didn't send you a message Think about it There probably will be a court case sometime in the near future where this will happen this will become a question What about non-principle characters? Has anybody checked that out? You know start sending control characters or something like that to the device You know, does it like misbehave? Like does it just jam on? Or like what? You know It makes me wonder like Has anybody actually tried this? It seems funny but this is a serious thing because I mean this is rather intimate and you could theoretically hurt somebody Now you could like hurt somebody with an SMS message This is terrifying actually What's that? Actually I don't own one My wife helps me with a lot of different projects This isn't one she was wanting to But if anybody's willing to pony these things are like 300 bucks if anybody's willing to pony went up to bring to like DEF CON next year so that you can just put on a table and just let people go shit on this thing Well that is you come out at the end of this thing with a white paper Does stop Bluetooth threats She asked What do you mean by on this thing? Poor choice of words I apologize But Does stop Bluetooth threats just turn off your Bluetooth It saves battery life too I mean my phone goes maybe eight hours if I'm lucky if I keep the Bluetooth off or keep the Bluetooth on turn it off I get a couple of days change the default pin if you can some of the new headsets that don't have keypads you can actually connect to you over USB and through a little program change what the pin is you need to connect to this thing always if your cell phone allows for it prompt for pairing don't just automatically accept any connection that wants to turn off headsets when not in use it saves battery as well plus you can get that stupid blue light blinking in my eye from across the board table and just consider the security implications of devices you have with you and if you're building something it gets really really bored and wants to you know start hacking away at whatever device you're creating think about it let's go after his keys Bob has one of his RFID proximity cards for his office you know you've all seen him Johnny Long has a really good talk in his no tech talking hack he talks about how he goes around and takes pictures of people's IDs that are on their land years and is be able to make up his own based on those because often times they're on their land years or whatever I don't know if you guys have seen this but Jonathan West used a website called cq.cx pointing out most of these cards are legacy systems some of the newer ones are encrypted but a lot that you see out there are not it's just reader queries give me a number it says here's my number oh you're on my list I'll let you in Jonathan created this wonderful little device this is actually one he's actually gone to version three now which is even better with this one it's basically you push one button wave in front of the card it reads the information push the other button it spits it back out so I just go up to the CEO or the other person who has access to the entire building the janitor and go hey how are you doing you're doing a really good job you know copy their card now the you know now the or the CEO has to explain why he's in the research and development you know rooting through you know sensitive files at three o'clock in the morning because all the audit trails will point back to his key which is still in his pocket traditional keys are vulnerable too they're photos of keys so if you're one of these guys that carries their keys on an exterior belt it's all jingling away a lot of janitors or maintenance people do this too the photo can reveal the bidding of the key which is what you need in a lock D-BOLD who is now premier election systems got caught doing this with the keys for their election machines you go to their web store to order more keys for your devices and oh there's this handy little photo of what a key looks like that's a key of the actual the photo of the actual key yes that's the actual key and it works for all the machines across that the company makes it's actually a common as dirt office supply and like mini bar key can we have the video up? there we go basically what happened was some guys were auditing these units for one of the universities and one of the guys was like hmm that key looks familiar that's the key from his mini bar at home we basically went home came back the next day it's like click oh hell there goes democracy passports passports I know major malfunctions been talking about these for a while but still it seems that people are just not quite getting it new RFID passports how many of you have one of these new passports with the little boxy thing at the bottom that says it's RFID enabled? you do? alright the encryption on these is generally not very strong the way it works is each country does have the ability to tweak how they're doing their keying but typically what happens is you walk up to border control hand on the passport they have to swipe the machine readable zone the text at the bottom underneath your picture on that page swipe that that gives them the information they need to generate the key to then unlock and download the contents of the RFID chip well there's only so much information to have in that machine readable zone the things like birth date very fairly limited key space especially if you know your target and roughly when they were born sequence number of the passports well if you know how long passports have been around or with the RFID well you can make some easy guesstimations as to how they do their sequencing you know there's other information that's fairly easily guesstible and they've actually been not US ones but other countries they've been able to hack them literally through the envelope that it arrives in the mail with just based on stuff that's on the outside envelope and Google you know they're able to get into these things even the US like the US okay when they put out when they were looking at doing this they basically put out a request for comment and the results from that were rather interesting like 2,000 comments against saying this was a bad idea to saying it was good a little lopsided their solution to appease people was okay we'll put it literally put a tinfoil hat on this the front and back cover have a liner of foil that is supposed to keep the keep the signal from being the RFID reader from being able to interrogate the chip until it's actually open and presented well again apologies to major malfunction this is his hand I don't know if you can see this but you push the thing closed the data stops it naturally springs open and they start streaming by the passport the tinfoil hat is self defeating I'll show you that again basically what happens is the rigidity of the material they used makes the cover want to pop open about an inch and that's just enough to get a reading now this passport is sitting right on top of the reader yeah paperclip an elastic band something you know it just if this thing is sitting in a purse or a loose pocket or something like that it can be read it's false sense security oh we put the tinfoil hat on it well passport condom he says no actually I'll just let that run swipe this from major malfunction actually in the UK just on Wednesday they were basically able to clone the chip out of a UK passport and put it into another one and the software used at border control says yep this is authentic and they were also able to go in and alter that chip and replace a baby's photo with bin Laden now and it still checks us out like this is like still going on right now we've been going at this for like a couple of years now we've been talking about this people aren't getting this and I think you really need to educate your salespeople who do travel abroad now one of the common fears is that oh you know if you're an American traveling abroad sniff your passport and figure you're American so they can kidnap you or hijack you or whatever I'm sorry most anybody from the western hemisphere traveling to Europe or other countries we stick out like a sore thumb you know it's like a bunch of hackers in Vegas in the middle of August you know we all dressed in black and we're pasty white we stick out but still think about this if you go somewhere if somebody clones your passport because they kind of look like you and you re-enter and they re-enter the country before you suddenly you're at the border and you're re-entering they say well you're already here come with me and you know the usual rubber glove treatment and you have to explain that no I how they won't probably believe that somebody clones your passport even though it is possible alright Bob has other duties occasionally works trade shows Bob's sales depend on having you know all those widgets in order being able to show off how wonderful they are you know how neat and shiny and everything that they are well what happens when you have widgets that have unauthenticated unencrypted commands and I'm sure most of you have probably spent a great deal of time because I was so excited when I found out that the badges were TV be gone that I was I'm sure there was a few people around Vegas this weekend that were kind of hating that this is an extreme example and please don't try this at home most of all this focus is about making most of all this focus is about making sure that we deliver devices for the consumer and you can just wait a moment to make some more screens and just go down and get some dancing girls to keep an eye on you for a few minutes while we just get things together let's take a look at what he go up to on his trip to Vegas that was Gizmodo at CES 2008 this past January Make magazine gave them a bunch of TV be gone and this was like the year of the TV at CES how much does a little square of tape cost I mean you have to feel sorry for the Motorola guy though I don't encourage people to do this because if you're trying to do a presentation this guy's livelihood and they're sitting there making his life miserable not cool now the wall of screens that was cool so I mean you know don't be a complete jerk when you're pointing things out to people but you know make the point now a couple of years ago at Shmucon I found myself in an interesting position there was a vendor who was giving away little USB hubs that lit up with an LED when they were plugged in and you know they had a whole bunch of them daisy chained together and they wanted to show off that they lit up so they had to plug them into something namely the demo laptop well I again you know you get a hacker right place right time right tool gets kind of bored waiting for registration and I'm standing next to these guys and I fish around in my pocket and I find oh look I have my USB switchblade with me go up to them have a friend distract them go click click I don't have all passwords out of memory I'm not one to make a small you know to make a small point I like making big like clue-bat side to the side of the head kind of level so scrub the passwords because I'm not complete it you know complete jerk scrub the passwords just to the first three characters did up a little script scrolled them across the screen borrowed a projector set up on the table next to them projected on the poster in behind them and then stood on a chair the announcement that okay ladies and gentlemen you're at a security conference that means to secure your ports not just your network ports but your USB ports and went on to explain what I had done the best part was that the the woman who was working the booth at the time was fairly new to the job and there was another more senior guy who was off on lunch he'd spent the entire day previously secure out of HackerCon don't connect anything with that you need a password for you know to be very careful these guys are listening who's doing this and who's doing that so she was being ultra paranoid with her laptop like not checking email not doing anything that transmits a password it was his laptop so when he came back from lunch he had a rather rude awakening it was very cordial I didn't give away all of his passwords but I proved what I had done and basically had to go through a little bit of mandatory retraining when I got back to the office they have a ritual in the office that if anybody does a real bone head pulls a homer they have to wear this big foam cowboy hat for a whole day so so what have we done to Bob here pwned his laptop his data his cell phone his wife his keys his demos and his ability to work you know has anybody got any suggestions I know Blackberry is going to come up Blackberry has actually done a fairly decent job of keeping things relative to secure Bluetooth on the device notwithstanding but through their own network they do do a decent job but just think about this how many of you have seen or have been or have seen people that during one of the big outages or something like that they can't get their email off their Blackberry they become like simpering piles of goo on the floor clicking like hoping that it will come back up imagine if you could cause that on demand like people are way too dependent on technology I'm also curious what are you guys doing to teach your staff and those under you that you're responsible for the security what are you doing to teach them to keep their stuff secure you know so other things that you might want to consider wireless presentation mics you notice me using the cord wideband jamming the ever so wonderful Lady Aida I gave it as a Christmas gift to everybody I can't remember the name of it right now wideband scanner does GSM Wi-Fi and Bluetooth wave bubble that's what it is just the ability to stand there and go click and watch people like hello, hello where'd you go hello just to be able to do that on demand and enforce some privacy is her logic imagine doing this next to something a little more interesting like I said people are putting wireless and stuff that should not be medical devices that's a little terrifying if you have any questions there's my email a huge thanks to Rob T. Firefly out in New York who did the artwork for Bob because I can't draw with crap and basically I have for sale the WPA tables the 33 gig addition 9 DVDs for sale in the wireless village basically I have to pay you back DEF CON for the ability to sell these things as well as this is how I'm eating so if you really like what you hear and you want to be able to crack WPA faster please buy a set they're 50 bucks they have lots of them I don't want to take them home thank you