All right, I'm Zaz and it's a great pleasure to be back here at Defcon China 1.0. My talk is going to be on hacking driverless vehicles and I'm also going to be including connected vehicles in that because basically unmanned vehicles are just another Internet of Things device except they're an Internet of Things device that's covered in sensors and that's able to move around in the real world and cause all kinds of mayhem. So I'll be looking at all of that stuff and one important distinction that I want to make right from the beginning is that I'm not going to be looking at attacks that include a network foothold on the platform. If you have a network foothold on the platform you can do anything you want that's well known. I'm going to be looking at ways that you can attack these vehicles from outside the system. A little bit of background about myself. I have an autonomous robot's background. I have a PhD in robotics and I am best known perhaps for a TV show that I did with another speaker that spoke at 10 a.m. this morning, Joe Grand. The show was called Prototype This and in the course of that show we actually prototyped a number of autonomous systems. For example, we did an unmanned aerial vehicle that delivered life preservers out to a swimmer in the ocean and we learned a lot about the various failures of aerial platforms and how to iterate on those platforms as part of that. We also turned our hands to ground vehicles looking at one of the principal problems in the autonomous ground vehicle space, the efficiency of pizza delivery. So we did two vehicles there, a local vehicle for delivering pizzas on the streets and a long range vehicle for long range pizza delivery which included the first ever autonomous crossing of a highway bridge in the United States. So this was pretty serious work. Although autonomous vehicles seem like a new and shiny thing, I want to take you back to the sort of origins of autonomous vehicles which weren't in fact in the United States. The first stuff was done in Germany. In the 1980s, 1986, a vehicle called Vamors that was done by a pioneer in autonomous vehicles, Ernst Dickmans. And in 1995, they drove an autonomous vehicle from Munich to Copenhagen at up to 175 kilometers per hour on the Autobahn using nothing but vision, no GPS, no LiDAR, just using cameras. So this stuff is pretty old, right? So this is kind of like, what's this talk about? It's about hacking Windows 3.1. No, a lot of new stuff being done in the space. So I'm just going to do a quick summary of what's happening in the autonomous vehicle area in Asia because America is getting all the press, but a lot of interesting stuff is happening here in Asia. So Singapore, they're looking to be a regional leader. They have autonomous bus services that they're planning to schedule by 2022. But China right here is the place that's positioning itself to be a technology leader in unmanned systems. So they're doing autonomous bus testing in Shanghai right now, in Guangzhou, public road testing since last year here in Beijing. Baidu is testing on 105 kilometers of suburban roads, and they have a partnership with Volvo. So they're looking to deploy level four autonomous taxis, fully driverless taxis by 2021. Pretty exciting. So there's a revolution that's going to happen in driverless systems because of so many advantages, right? The energy efficiency of not having to haul a human driver everywhere you go, the time efficiency of not having to deal with the needs of that driver, and the new applications that it opens up. So lots of areas in the civil space, obviously transportation, also oceanography, right? The ocean is a dangerous and boring place for the most part, it's a great place to send robots, mapping of course, filmmaking already, autonomous drones have taken off there, engineering tasks like power line and structural inspections, and logistics delivery, right? That's something that has to happen everywhere. So in particular in terms of logistics, don't just think about the air and the ground. The sea is a very important part of logistics chain, and unmanned cargo shipping is the future, because the accidents are mostly caused by the humans. 75% of maritime accidents caused by human error, and a major technical challenge that has to be overcome though before we can deploy our unmanned systems is dealing with these long voyages, months at a time, and having hardware failure. A lot of reason you have a human crew is to fix things that break. But here's an example that's just been announced, the Kongsburg company has announced they're going to deploy this boat called the Yara Birkeland, which is going to be an electric zero emissions vehicle with autonomous capability. And so this will just give you an idea about how these systems are likely to be rolled out. It's going to replace 40,000 truck trips a year, so they're going to start in 2020 as a manned vehicle. They're going to drive this boat back and forth delivering its things. 2021, they're going to start down crewing and letting the robotic systems take over, preparing for fully autonomous operation in 2022. So that's the kind of thing that you'll see out there in the real world. So the priorities of the industry in the civil space are agriculture and self-driving cars. And the things that are in the way why we don't have our roads filled with driverless cars right now when Dickman's was doing this stuff in the 80s and 90s is the fact that we have shared infrastructure. We have to share the roads and the airspace and the oceans with human piloted vehicles and the public acceptance, the perception of safety and robustness. So on the topic of safety and robustness, let's talk about how these systems fail. So I'm going to look at two case studies of classic failures. Number one, there was an event held in 2004 called the DARPA Grand Challenge. That was an autonomous robot road race from Los Angeles to Las Vegas, the home of the original DEF CON. And this was a big deal. The prize was one million dollars and a lot of teams put a lot of work and a lot of money into trying to be the first to win this road race. And the favorite for this race was a vehicle called Sandstorm from Carnegie Mellon University. They developed all kinds of new sensors. They loaded it up with everything that it had to offer. The vehicle took off and everyone was super excited. This was the one that everyone thought was going to win. Well what happened? This vehicle got a few miles along the course and then it veered off the road, crashed, rolled over and caught fire.  What went wrong? Everyone was very disappointed. Well what went wrong was the fact that they knew the course a couple of days in advance and they manually walked the whole course. They had a perfect GPS map. But when they actually went to run the race, they had to set the waiting to trust the sensors versus the GPS and they trusted the sensors too much. The sensors got a wrong reading, drove the car off the road. So it s a constant battle to know what the robot actually knows. And the correct estimation of that state is the key to decision making. And so a successful exploit of an autonomous vehicle will most likely subvert the state estimation process. Here s another more recent big failure. This was a Tesla in autopilot mode. This is from last year, March 2018, a huge accident that unfortunately was fatal for the driver. What went wrong? So the Tesla was in autopilot mode and so it had two types of autonomous behaviors that it was doing. Number one, dynamic cruise control. It was following the vehicle in front. Number two, automatic steering with lane following. It was following the lanes and staying in its lane. It got to a fork in the highway and it could have just kept following the vehicle in front and it would have been fine. Instead it followed bad lane markings and it crashed right into the barrier between the two roads. At 120 km per hour, the crash attenuator that was there on the highway was supposed to stop that from being a terrible crash but it was previously damaged. It wasn t fully functional and so it was a bad accident and as I said, the driver unfortunately died. And this all came down to one decision. The vehicle selected those poor lane markings instead of following the lead vehicle. So autonomous vehicles is an area that s full of fragile decision making and edge cases that have to be decided in a split second. So let s take a look at the logic structures inside an autonomous vehicle. There s a hierarchy of behaviors that operate at different speeds and at different priority levels. So at the bottom you have your control loops, the maintenance of stability and operation of the vehicle. Above that, collision avoidance. So making sure once you have control of the vehicle that you can control it not to run into things. Then you ve got your navigation and localization, getting where you want to go. And at the top level, your task planners and your reasoners. If you re a taxi, getting to the right place and then billing the customer, delivering pizza, so on, which I ll get to in a sec. So if you re attacking one of these systems, then you might want to attack lower level in the stack because they defeat everything above that. However, because those lower level things are more mission critical, then more engineering effort is spent on guaranteed robustness. So they may be a harder target, right? You ve got your juicier but harder targets. Just like attacking many types of infrastructure. So two quick examples from the very beginning of this talk. The lifesaving drone first and its control loop level, it has its PID loops on its autopilot tuned to be out of cope with wind gusts and warm rising air, cold descending air and so on. Then the collision avoidance level, absolutely nothing. This is a big problem for aerial vehicles is that there s very little sense and avoid in place. Navigation and localization, that was simple waypoint circuit going from waypoint to waypoint. And then the mission reasoner was to set up this dynamic bombing run to drop the life preserver to the person in trouble. So this is a system that s extremely vulnerable to collision because there s no collision avoidance. And its high level logic depends on one single sensor, GPS. So if there s any problems with that sensor, the system is going to have problems. Pizza delivery. I ll look at the local pizza delivery because it s kind of more interesting. The control loop level, this is a two wheeled vehicle so it has to do balancing and dynamic weight shifting. It s got to deal with the fact that the weight on top changes when a pizza is removed.  Navigation and localization, route planning using a map that the system builds itself with a process called SLAM, simultaneous localization and mapping. And then the final high level reasoner is to dispense the pizza to the correct person using the credit card as an authentication mechanism. So this is a system because it s doing its dynamic obstacle avoidance and it s building its own map. It s vulnerable to all types of redirection, trapping and map confusion attacks. So let s take a look at the suite of sensors that you might encounter in autonomous system. There s a basic difference in sensor types. You have your active sensors that send information out and read a return and the passive sensors that just read in information from the environment. So commonly, you ll find GPS and they usually mounted on the roof so they have an uninterrupted view of the sky. LiDAR, also often mounted on the roof, that s a laser rangefinder which we ll get to later. You ve got cameras for doing vision, often a millimeter wave radar, usually on the front and back of the vehicle. Ultrasonic transducer is a common for parking. Usually, you have a digital compass. It s always nice to have an absolute direction reference. An inertial measurement unit so that you can measure acceleration and rotation.  And then for other types of vehicles, for example, under the water, you might have sort of more esoteric sensors that are designed to operate in that domain. So a Doppler velocity logger, for example, under the water, scanning sonar and pressure transducers often used both under the water and in the air to give you, for example, a barometric altimeter. So sensors have a number of sources of uncertainty. They constantly have noise that they have to deal with. They drift. So that s where they over time get a reading that s further and further away from the true reading. And then you have latency problems. So you have some time delay between when you read the sensor and what the actual value is. So the real world may have changed during that latency period. So you have to model that noise. You have to model that uncertainty under a series of assumptions. When you have more than one sensor, that s great because then you can do sensor fusion. So fused or registered data can be more useful than separate individual data from one sensor alone. However, what do you do when those sensors disagree? This is very topical right now because of an automation failure that s been in the news, the Boeing 737 MAX, which had two angle of attack sensors, but they were only using one of them. They chose not to do sensor fusion for whatever reason. Boeing has never explained publicly why they chose not to do sensor fusion. But it may have been what do you do when the sensors disagree? And what they re going to do now with the new 737 MAX fix is to display a cockpit light that tells you tells the pilot when the sensors are disagreeing. So again, your robustness of your system may come down to how good is your system at discounting one malfunctioning or one intentionally spoofed or hacked sensor? That was Boeing s problem. So let s take a look at our general classes of sensor attacks. Two basic kinds of attacks. We ve got denial, just like denial of service, that s preventing the sensor from getting any good data or spoofing. That s when you are deliberately causing the sensor to retrieve specifically bad or incorrect data. So you ve got a basic attack mode choice as well from the attacker s perspective. Do you attack sensors instantaneously, give them immediately bad data, or do you do a more subtle attack where you give it aggregated bad data and you build up a poor model over time to cause it to make bad long-term inference? So I m going to take a look at all these various sensors and how we do both kinds of attacks on them. So GPS. Denial is very easy with GPS. We can jam it. You can buy online a GPS jammer. Check your local area for legality. But China is a place that manufactures many of these GPS jammers. You can also build your own. Schematics are available online. And they just throw a bucket of radio noise out and they just prevent the receiver from receiving valid satellite signals because the actual signals that come from the satellite are very weak. Spoofing is also possible. So basically you re sending out valid GPS signals with a radio at a higher power than the genuine signals from the satellites. So here s an example of that from the UT Austin Radio Navigation Laboratory, which is one of the first places to really popularize the possibility of active spoofing of GPS. So they re demonstrating here that to take over the GPS of a vehicle, you send out a valid signal first.  And by doing that, you can affect the controller on board the vehicle. So for example, an aerial platform, if you send your fake GPS signal, send out a signal that makes it look like the vehicle is going up, then the controller on board will drive it down to compensate. So here s an example on a real aerial vehicle.  And so the controller s going to drive it down. And if the safety pilot didn t take over, this vehicle would drive itself right into the ground. This is something that the Iranians said that they did when they captured this American military drone. They captured this RQ-170 drone and they claimed that they used GPS spoofing to bring it down. That s almost certainly not the case because everybody knows that GPS is very spoofable. The military knows that. They have their own encrypted GPS signal, but they don t use GPS as a primary sensor because they know that it s jamable and spoofable. What happens though in a real system when a GPS system causes a problem? So here s an example from the second DARPA Grand Challenge. This is a vehicle called Sandstorm, and we can see a straight road, but it s pulling it off the road. That s what will normally happen when you have a vehicle that s got multiple sensors and it s getting bad GPS data, is that it won t immediately go off where you want it to go. It ll sort of be trying to pull it back onto the track. Here s another example from the DARPA Grand Challenge of the perils of GPS-only navigation. This vehicle completely fails to do collision avoidance because it s getting bad GPS data and that s pulling it off the road. It s not just cars that use GPS. In particular, for ships, GPS is often the primary navigation sensor because out in the middle of the ocean you have no other navigation references, especially during the day when you can t use the stars for navigation reference. So that same lab, the UT Austin Radio Navigation Lab, decided, well, let s see if we can take over a super yacht. So they found a rich person and they said, hey, can we borrow your yacht to see if we could potentially steer this thing off course? And sure enough, they had their attack system on board the yacht and instead of sending a drift signal up or down like you would send an aerial vehicle, they changed the angle of the direction of GPS travel by just three degrees because three degrees over the thousands of kilometers that a boat might travel across an ocean can put that boat in a very, very different location. So they were able to use their spoofed GPS to send a different navigation track and cause that ship to go off course. There s the radar image from that ship to prove that that was possible. So originally, this might have been an expensive attack, but now SDRs have made this attack very, very inexpensive. The first one to be presented at a hacking conference was this one using the Blade RF. So for about $1,000, this was the QUIHU 360 Unicorn team at Defcon 23. They showed that for about $1,000 using a Blade RF, they could spoof GPS. And that has become even easier since then as SDRs have come down in price. Also, they went to car showrooms and they showed that the in-car navigation system could believe that the car was in the middle of a lake somewhere. Now, we can do this with a Hacker F1 for about $400. You do have to modify it. So you need to get that the built-in oscillator in a Hacker F1 is not load drift enough to do this attack reliably. You need a 10 megahertz TCXO load drift oscillator and you make a little daughter board for that. So here s the Osh Park images for that daughter board and that shows it installed on the Hacker F.  They provide the GPS ephemera data, all of the details of where the satellites are for the previous day. So you can spoof for any day in the past by doing that and then you can also fake the time to spoof in the future. Then you use the open source project GPS SDR sim to convert that ephemera data into active GPS signals that are then transmitted out through the Hacker F. So I m actually doing that in this auditorium right now. Unfortunately, it just stopped. I m just started again. It may take a little while to get the signal but I ve got the maps here. We can see here my GPS device, if we can switch to the camera. Can we grab the camera view over here? Hopefully you can see that I m getting full satellite signals on this device. It hasn t quite picked it up yet because I just restarted the system but it s getting them. You can see there it s got quite a few satellite views already. Then this iPhone has already got the signal and I don t know if you can see it there, if there s any way to zoom in on that. But right now, and you can check this on your own phones, I recommend turning Wi-Fi off before doing this so it s only relying on GPS but hopefully we can see here that I m broadcasting GPS right now that s putting everyone in this room in the middle of the White House in the United States. Here s my rig here with a Hacker F1 connected to an Raspberry Pi connected to my computer. You can take down UAVs with this technique very easily if they are a little commercial drone that has no fly zone capability. Here s a video that I did of forcing a Mavic, DJI Mavic drone to land. You can see here I m simulating the GPS of the middle of an airport and that s going to force it to land. It ll take a few seconds but you ll see that the no fly zone capability has been activated. Well, I don t know why it keeps transitioning to the next one. Anyway, the drone lands because it thinks it s in an airport which is a no fly zone. Moving on to LiDAR which is a laser range finder. These were originally industrial monitoring sensors. They were used for factories and so on to keep people out of places they weren t supposed to be. They re a mechanically scanned laser and they re primarily used in robotics for collision avoidance and map building. So in order to deny LiDAR access, you can actively overpower them with a strong laser which or you can prevent the return signal because it needs that signal to bounce off something and come back. You can make it so that signal doesn t come back. For spoofing them, you can manipulate the absorptions of the reflectivity of items in the environment so they look like something else or you can actively spoof them and I ll discuss all of those methods. Here you can see the kind of point cloud map that the LiDAR builds. So as a 2D sensor, there s this laser that s being scanned backwards and forwards or left one way to the other, they re highly orientation dependent. So if you have a platform that tilts like this two-wheeled vehicle, the kind of thing that the LiDAR sees depends a lot on the angle that the vehicle is at. So that means inclines can look like obstacles and it means it may miss low obstacles and discontinuities. It s an active emission sensor. It s sending out information so it can only see something that bounces a signal back. So if it doesn t see something, it thinks nothing is there. It has no information about it. So think about it. Everything over the horizon returns nothing. It s just empty space. So most of the world returns no data to the LiDAR. You actually, even though you get a dense point cloud of things that are there, looking 360 degrees in every direction, most of it s got nothing. So also absorbent things look like nothing and transparent things. The laser goes through them. So that means we can do some physical attacks. So we can paint absorbent material on a wall just like the old Wiley Coyote gag and the vehicle might try and run right into it. We can also use obstacles that let light through and the LiDAR won t see them. Reflective things can cause problems for the LiDAR. For example, puddles of water on the ground. This is something that causes all kinds of difficulties for real world vehicles. So that means that things that are far away, the laser can bounce off, reflect back and they look like they re close. So a tree in the distance can look like a tree coming out of this puddle. And it means if you don t get a return, it s indistinguishable from a hole in the ground. You see a big gap in the road. Is that just a puddle or is it a huge hole? You can t really know. These techniques, by the way, are known by insurgent groups that are trying to deal with military applications of this technology.  And it mentions GPS jamming with a Russian-made Raqqal system. And it mentions using reflective materials to thwart laser designators. So these are valid adversarial techniques. Reflectance, though, in the case of unmanned vehicles is also a feature because one thing that reflects is road line markings. It s actually a very robust method to take those holes in your LiDAR point cloud and do shape matching on them and use them to find road markings. So that means that s a way that we can do an attack that fakes road markings in a way that the robot sees them, but a human safety pilot doesn t see them. So we can paint black on black reflective material that a human can t see, but we can cause unpredictable behavior from the vehicle. Or we could just send some jokes in for the people back at the mapping zone. I m sorry, I went too fast on that slide. But to send things back to home base. Solid looking objects are indistinguishable from actual solid objects because the laser just knows when it bounces off something so we can have lightweight obstacles. And in the case of denial, we can use a strong laser source to overpower the LiDAR in a certain area. So here s some research from KAIST in Korea showing that if we point a strong laser at a multi-beam velodyne so we can see here all those horizontal stripes are the individual beams of this multi-beam LiDAR and by pointing a strong source at it we can blank out part of the area of its scan. So we could hide an obstacle by overpowering the LiDAR selectively in a spot. But it gets more interesting than this. We can actually spoof returns by using a property of the way that most of the LiDAR is built. So here s an example of two very popular LiDARs, the velodyne pucks. And both of those things have curved glass on them because that way they can scan 360 degrees. Because we don t want to have sharp edges to the glass which are going to be difficult to transmit the laser through. But we can use that refraction to move our false points by modulating the power of our spoofing laser source. So that means if we were attacking a vehicle from the side we could make it think there was something in front of it. So we don t have to put our fake returns in line between us and the target. We can make them show up somewhere else. So this is using exploiting the fact that the curved glass refracts the light inwards to a different location. And then we can modulate that with source strength. So here s an example of using a weak source and we can see the false returns between the spoofer and the actual LiDAR. But then increasing the strength of our source from the same location places our fake returns off to the side. Now we can also do active spoofing. We can do kind of a relay attack. But this is a much more complicated attack. This involves characterizing the behavior of the remote laser. So we have to watch for a while, see what types of pulses it puts out, what the frequency of pulse is, and then we can start sending predictive returns from the spoofing source. This also gives us great control over where our fake returns are located. But this means the timing is critical. So you have to very, very accurately characterize the pulse rate of the source and its location. But we can see here the spoofer is located to the left of the target LiDAR, but we're inducing fake dots further to the left. Or changing the timing, now we can put the dots between us and it. So that gives us control, not of the full area, of the full xy plane, but it does give us control of the distance from the target of the fake returns. Now a system that's out there in a large number of vehicles is the Tesla autopilot because they've put that into a lot of their vehicles and they're able to update that as an over-the-air update. And Tesla's philosophy here is to only use cameras. They do use a little bit of millimeter wave radar for long-distance collision avoidance, but mostly their theory is we do everything with our eyes when we drive, so our cars should be able to do everything with cameras. We already saw one example of where that's caused a fatal accident. But so let's talk a little bit about cameras. So cameras are used for specialized object detection, including signs with traffic control features and lane markings. Sometimes we'll use stereo cameras to get a depth map, so to get a noisy version of what the LiDAR gives us. And sometimes we'll use it to fuse with a LiDAR in a method called colorizing the LiDAR. That's getting color information from the camera and melding that with the distance information from the LiDAR. This is useful for this reason. This is a video from the second DARPA Grand Challenge. This is the vehicle that won that one. The second one actually did have a winner. It was a vehicle called Stanley from Stanford, and they won because they were able to drive extremely fast. And the way they did that was to use the LiDAR to give a sense of drivable versus non-drivable, and they matched color to drivability. So that was a way they were able to look further out than the LiDAR could look by saying, okay, this is what color the drivable road is. The rest is non-drivable and being able to drive at speed. Cameras are very easily denial of service. You can dazzle them with a bright light. And then spoofing is all the ways that we know about from the ways that our own vision is fooled. Camouflage techniques, assumptions about what color things are, and then in the case of stereo, repeating patterns really confuse the stereo system. I want to say very briefly about something about all the research that's come out lately about spoofing deep learning camera recognition models. This stuff right now is kind of overblown, but I just mentioned it for completeness. So there's a very famous paper on adversarial examples for stop signs.  Of course, feature-based sign detection is pretty robust, and you probably wouldn't train a deep learning model to find stop signs in the first place. There have been other examples where by adding adversarial noise to the incoming camera image, you're able to get it to make false recognition. So in this case, the pedestrians are missed when you've added some very fine, invisible to the human adversarial noise. Of course, if you have the ability to add adversarial noise to the camera signal post acquisition, that means you have a foothold on the platform, there's lots of other things you could do. Generally, these techniques are white box techniques. So these are researchers that have trained their own model. They know how the model works, and so then they attack it based on things that they develop to fool their own model, and they don't tend to work reliably in the face of parametric distortions. So when you have different angles, different amounts of glare, and things like that, different lighting conditions, they don't work so well. Here's an example of something that was pretty robust. It was robust enough that they were able to 3D print objects, and then they could show it to the camera in any orientation, and it would still work. These are the adversarial turtles from MIT. So this, for example, is a turtle that's obviously a turtle, but it has patterns printed on it so that the vision system thinks it's a rifle. But again, these techniques are not robust, and they often involve the injection of noise that requires a foothold on the platform. I mentioned it for completeness, as I said. Cameras in general involve fragile discriminators. So here's an example from just this year, from the Tencent Keen Lab. They were looking at Tesla Autopilot. They found some results that are not entirely surprising, but it's proof of what I'm saying. So for example, they added white tape to the road here to blur a lane marking, and they showed that the system successfully didn't pick up a lane marking. So this is real world blurring to prevent the Autopilot from seeing a lane, and then the converse is also true. So they were able to add just a few dots to the road. You can see them there in the upper right, and just those three fake markers were able to cause the vehicle to make an unplanned lane change. So just like that fatal accident, this is something that requires a small amount of real world preparation, but can cause a very bad accident. Moving on to the millimeter wave radar. This is the system that you find at the airport often to scan you and see under your clothes. Primarily use the collision avoidance. They're a lower resolution than the light are, and most things are very, very reflective to the radar. So it returns a lot of noisy data. So you can deny them with a radar jammer or with a chaff, which are little pieces of metal that are spat out. That's what the military uses to fend off radar guided missiles. Very flat, reflective in the RF domain, things like overhead signs cause problems for the radar. So this is an example of a Bosch millimeter wave radar sensor that's used in the Tesla, and this was shown to be jammed well at DEF CON 24. So if you had an oscilloscope and a signal analyzer, a signal generator, and a harmonic mixer and frequency multiplier, it's about $100,000 worth of equipment, then you could jam the Tesla autopilot and cause it not to see a vehicle that was in front of it. So a very expensive attack, but it's possible. They theorized spoofing and relay attacks but did not perform them. Moving on to the inertial measurement unit and the compass, here's an example of three such systems at various price points. You think there's a wide, wide range of IMUs available from very cheap, hobbyist level ones for a few dollars all the way up to many, many thousands of dollars. These are very, very robust, or they can be very, very robust. So they are used as the primary navigation sensor for some systems, such as certain underwater vehicles or such as some military systems because of the high fidelity. If you're going to shell out for, say, an IMU from a Boeing 777, you can expect 0.1% of the distance traveled as cumulative error. So that means if you go 300 kilometers, let's say under the ice, when you pop up to the surface, you're 300 meters away from where you think you would be or within a 300 meter radius. So pretty accurate. Not much you can do in the way of attacks. Very, very difficult to interfere with. You can do physical attacks with magnetic fields and exploiting thermal drift if you have physical access to the platform. One particular interesting physical attack on a particular type of readily available IMU is the acoustic attack. This is pretty exciting. This is because those cheap MEMS IMUs actually have physical components in there that move. There's a vibrating element in the MEMS gyroscope, and it has a resonant frequency. So this is what a MEMS microelectromechanical system gyro looks like, and you can disturb that with an external acoustic source. So this is similar to attacks that have been publicized recently on hard disks, what they call the screaming in the data center attack. So you blast some noise at a hard disk and it vibrates the read head and you can cause the hard disk to not be able to read or even cause physical failure of the drive. Similarly, you can do this to a MEMS gyro. So here's an example from KICE in 2015 where they took a real multi-rotor platform and they blasted acoustic noise at it, and you can see here this is the control outputs showing that when the noise was blasted, the control went nuts because it was trying to deal with the fact that these MEMS gyros were vibrating at their resonant frequency from the noise. And so they were able to successfully crash at whatever distance you can blast that level of sound pressure, a multi-rotor craft. Next sensor, wheel odometry. These are physical encoders that measure the rotation of the wheel. So here are some that you might see on a commercial driverless vehicle at present. These are good to know what your real speed is and when the vehicle has actually stopped. And here's an example from our show, from prototype this, when we were doing our bridge crossing on one of our attempts, the vehicle took this turn too closely and it actually scraped into the barrier and it knocked its wheel encoder off. You can see it there knocked off. At that point, the car got very confused. It wasn't able to get ground truth odometry and it wasn't able to proceed. So a physical attack that knocks that off can cause stopping of the vehicle or other unpredictable behavior. So ways you could attack the wheel odometry is to change the diameter of the wheel, to make the surface slippery and to scrape them off, to force the car to go close to something and physically remove them. Another sensor that I'm including really just for completeness here is the ultrasonic sensors. These are most commonly used for automated parking. So this is stuff that happens at a very slow speed. You can attack them in all the same ways as the other remote sensors. You can jam them. So here from Defcon 24 sending out ultrasonic noise to prevent them from getting a reading. You can send fake pulses back to make it look like something is close when it's not really there. And you can also do an interesting thing because it's in the acoustic domain. You can do cancellation. So if you characterize the sensor, you can send an inverse pulse and cancel out the real returns that are coming from the sensor. Again, it's a low impact attack quite literally because this is only happening at parking speed. So you know if you were going to be James Bond and produce a vehicle that was going to be able to have various attacks involved against autonomous vehicles, you would have a GPS jammer of course. You'd have smoke and dust and vapor to interfere with the LiDAR beams, lightweight decoy obstacles, radar chaff, transparent things to drop down and puncture the tires and of course no bond vehicle is complete without an oil slick and it works a little differently in this case. You're following the wheel odometry.  So I'm just going to do a quick time check here and see who I am on time. All right, so I'm going to move pretty quickly through these next things just to try and get through to the last stuff. But the map is very, very important. There's a great emphasis these days on pre-acquired map data and that's primarily because some of the big developers of autonomous vehicles like Google, they already had a lot of mapping data. And so here you can see a LiDAR-acquired point cloud. You can see how detailed your map can be. So detailed in fact that it's often considered by designers to be the truth. And that's because it's very, very useful. If you know where everything is in the world you don't have to do as much work on recognition. Things that are hard to spot like traffic lights like vegetation. You can just know where they are, speed bumps for example. You don't need to see them if you know exactly where they are and if you know exactly when you are. Here's a quick example on traffic lights. You can have a camera that's programmed to just look where you know the traffic lights are and then you can check the status of the lights easily. You don't need to detect lights, you just need to look in the right place and detect state. But that means fake traffic lights won't be detected by your system but they will be obeyed by human drivers. These are shared roads. So they create a difference between stuff that human drivers on the road are going to pay attention to and that robots won't. So if you put a fake green light, a person is going to think it's okay to go or a fake stop light, the robot is not going to see it, you can cause an accident. Vegetation is an interesting one because vegetation changes. So you might often use say a colorized light art and a transmission classified to determine if a piece of vegetation is crossable or not. But things that overhang and change and grow like vegetation, if they're not on your map and it hasn't been updated, they can cause problems. So dependence on the map may exacerbate the brittleness of your sensor discrimination algorithms. Here's an interesting little video that shows what you can do if you have a perfect map. If you have a perfect map you don't have to have any kinds of traffic lights. You can just drive straight through an intersection. If you have perfect knowledge of the world then you can make these like split second decisions that are always right but you don't always have perfect knowledge. The map has to be constantly updated. So a local map is vulnerable to unexpected real world features and a remote map is vulnerable to denial of the update process. So jamming, spoofing through man in the middle of tax and so on. So the map is an interesting vulnerability and an attacker is going to try and exploit the logic structure that your vehicle has. So the goal is to maximize the uncertainty of the system. For example, a blind right hand turn is a very brittle traffic maneuver and this is an area where you can force the vehicle to require manual assistance. Confuse and annoy the occupants. Inconvenience the other road users. If you can prevent the robot from successfully navigating a fragile maneuver. And of course the best thing to do, if you are an attacker, you have access to the map as well. You can find exactly where these fragile decision points are and cause bad behavior and maybe even cause a bad accident like this. Trapping and redirecting are a general class of attacks that attack at the collision avoidance and navigation layers, forcing the robot to postpone its high level tasks so you can use moving obstacles. You could perhaps even use robot swarms, obstacle swarms. You can use artificial traffic control features to force the robot to go somewhere that's under the attacker's control. And you can make things that are subtle. There are ways that a human driver would just feel free to ignore them. But the robot has very, very fixed rules and it can't ignore the attacking features that you're putting in its way. The other big attacks area is clobbering. So forcing the robot to crash into something, subverting the collision avoidance layer. So the goal here is to incapacitate the vehicle to strip off sensors or to damage them physically. You can do this with subtle map deviations if the robot is very dependent on the map. You can imitate vegetation that it thinks is crossable but is not. You can simulate high impact obstacles while the vehicle is at speed that will cause it to make defensive maneuvers that will cause an accident. You could disguise entrances like overhangs or gateways that are within the GPS noise level so the vehicle might scrape its side removing those encoders for wheel odometry or so on and then placing dynamic obstacles underneath large signs to fool the radar. So all of that said, there's some debate about whether we'll see level four and five autonomous vehicles in widespread use on the road anytime soon. This is a funny quote from 2014 talking about how 99% of the country is areas where the autonomous vehicles cannot drive. So another area that I just wanted to cover very briefly at the end here is the recent announcement of the rollout of standards for vehicle to vehicle and vehicle to infrastructure communications. This is from the US Department of Transportation. These are little radio messages that cause transmit between each other and between infrastructure that warn them of certain safety hazard things. So the early V2V components are going to be rear end collision warning, lane change warning and intersection warning scenarios. Right now, just warnings. No active control of the vehicle is proposed at this stage. So it uses these radio messages both on board and to roadside communicators. The protocol is called DSRC, Dedicated Short Range Communications. These are omnidirectional 200 to 500 byte packets and the protocol is called the BSM, Basic Safety Message. These packets are not encrypted but they are authenticated via certificates. So standard PKI issues apply. These are what the packets look like. Part one is the core that's transmitted with every packet. Part two is appended whenever it's changed and there's a lot of vehicle specific information in part two. GPS is transmitted unencrypted. So if you can receive these messages from a vehicle, you can tell where it thinks that it is. So if you're sending out your GPS spoofing, you can actually get feedback to make sure that the vehicle actually is being fooled. Here's a breakdown of the security model. I want you to pay special attention to the stuff that's initial deployment versus full deployment. So those dashed lines are the initial deployment. Most of this is slated for later on. The initial deployment is very small. A lot of functions are split for security and privacy reasons. Privacy has been a big component of the development here. The certificates come pre-installed so your car will come with thousands of certificates and they're valid for five minutes each. There's a location obscura. So privacy again, very important here. The bottom line on V2V, the rollout is very careful. It was 11 years in development. They're estimating 37 years to the full fleet. Okay, so that's a long time. Compare that with all the hype about autonomous vehicles and like, oh, we're going to have robot taxis by 2022, maybe in some limited cases. But just for this short-range communications protocol, 37 years to full fleet rollout and tracking and privacy are more immediate concerns than other malicious attacks. The standard PKI concerns apply and remember, certificates are going to be very, very easy to get. You're going to have thousands of them in every vehicle, but five minutes is a long time on the road and there's no direct control even proposed. So actually the driverless vehicles may be with us before we have V2V direct control if it's going to take us 37 years just for warnings. But hopefully they won't make the same mistakes that were made in other traffic and infrastructure sensors. So here's some stuff from DEF CON 22 from IOActive. They showed that these traffic sensors that were sending data into traffic control systems had no encryption or authentication. So the messages were clear text wireless transmissions that were fully spoofable and firmware updates were neither encrypted nor signed. So hopefully they won't make these particular specific mistakes, but no doubt they will make others and we as hackers have a duty to find out what they are and notify so that things can be changed and made safe. So here's just a quick example, very recent research. This was released by Georgia Tech in March, I think. They just looked at what are the results if we had widespread driverless vehicles that were hackable. And so this is looking at Manhattan Island, New York and just looking at what happened in terms of access to the grid network. If we had a lot of vehicles because we need emergency vehicles to get through. We need ambulances and fire and so on. We want our roads to have full access. But hacking just 10% of the vehicles at rush hour, you can see they used a percolation theory to develop the simulation. And you can see a large chunk of the island is now completely inaccessible to any vehicle just by being able to do denial of service attacks on about 10% of the rush hour vehicles. 10 hacked vehicles per kilometer per lane. And then doubling that to 20% and we can see that the island is basically unnavigatable. So that's really, really serious. So this is why we need to take this problem seriously and why we need to do research here and make sure that we're putting these IoT systems out into the world in a way that's going to keep people safe. So that's it. In summary, I will just say I want you to remember that driverless vehicles are cool. We're in favor of driverless vehicles. We want all the advantages that they offer. We're not trying to stop this in any way. Don't do any of these things in the real world. Only do it in a research context. Don't hassle the half or rather don't hacksaw the bots. And hopefully we can bring the world together into the place where we're living the driverless dream where we can be doing our romance in the backseat of the car while the robot does the driving. Thank you very much.


Software development is risky, but 25 years ago, building a program was like gambling at a casino. In 1996, 31% of U.S. development projects failed entirely. 53% were over budget, schedule, or missing functionality, and only 16% met initial expectations. The problem was in the way IT projects were planned. In the 1990s, software development usually looked like this. First, the team of user representatives gathered requirements, development was scheduled and budgeted, then thousands of pages of documentation was handed off to a tech team to begin design and development. Eventually, the system was integrated into existing processes and tested. This was called the waterfall model, as steps were discreet and strictly followed each other. Today, the drawbacks of this approach seem obvious. Planning and documentation could take months. But what if you've made a mistake and the software solves non-existent problems? This would then be revealed only at the end of the cycle, and estimating new features that need deep research was essentially guesswork. An additional headache became evident in enterprises and complex projects. How to translate business requirements to technicians? Project managers and analysts that worked on analysis stages spoke a language completely different from the one engineer spoke. Programmers like solving tech puzzles, not business problems. And this made the gap between planning and coding even deeper. There was a need for someone who'd take the technology leadership, have a high-level view on the problem, and just a solution expressed in a language accessible to technicians. In 1999, engineers from Rational Software, led by Philipp Kruchin, suggested building software in short iterations, reevaluating assumptions and results at the end of each development phase. This technique was called the rational unified process, and this partly solved the first problem related to long planning and a feedback cycle. However, today most development teams use other iterative and agile methods like Scrum, Lean, and Kanban. But Rational, by bridging the gaps between requirements and technology, helped address the second problem, communication difference. Later, this practice was dubbed solution architecture, and the role of the person leading it, solution architect. The term architecture was borrowed from the construction industry to highlight the complexity of problems. These were enterprise systems after all. The term also depicts a vantage point. Architects oversee the big picture without diving into low-level technical details. Today, we know three main roles related to architecture. Enterprise, solution, and software architects. First, there were enterprise architects and software architects. Solution architects weren't initially recognized either in enterprise architecture frameworks like TOGAF or in software development ones like Rational. But sometime in the mid-2000s, the role emerged as a middle ground between a wide enterprise perspective and a narrow tech perspective. Ultimately, the roles differ in how high their vantage points are. Enterprise architects consider the big picture and see the entire corporate tech ecosystem from the business perspective. Solution architects work with specific products and define how they answer the business questions with technology. Software architects specify the structure and behavior of a system described by solution architecture from a technical perspective only. Any software development starts with gathering requirements from a diverse group of people, stakeholders, user experience specialists, or even end users. Business analysts are in charge. They document requirements in user stories, short descriptions of product behavior from a customer perspective. These are functional requirements. There are also some non-functional ones, the general qualities of software, its speed, maintainability, security, and some others. Besides requirements, any project has its own constraints, a budget, manpower, time, licensing, and risks.  The architect is responsible for converting them into a high-level technical guide. But there's an additional level of complexity. Usually, a product never exists alone. It must fit into the current enterprise architecture, other systems a company has been using before. For instance, if you use an old legacy monster, your shiny new admin panel must be able to tame it and communicate with it. So a solution architect considers existing enterprise architecture to make sure that a new product fits well in the existing ecosystem. If you look at the Amazon Web Services Certification Exam Guide for solution architects, you'll get the idea of the problem domains that this person faces, designing resilient architectures, defining performant architectures, specifying secure architectures, and so on. Too vague, right? OK, for example, an architect building an AWS-based system may need to decide which Amazon cloud service to use and how if the client's salespeople upload their sales figures daily. They need a durable storage solution for these documents that also protects against users accidentally deleting important documents. The selection of specific technologies isn't limited to cloud services only. An architect decides technology stack, databases, architectural patterns, standards, and more. But where's the architecture? What is that exactly? Construction architects have concepts and plans. Solution architects have diagrams. And remember rational software. In 1995, they suggested Unified Modeling Language, or UML. UML is a language for visualizing how software works. This is Hello World in Java programming language. And this is Hello World in UML. UML suggested three main building blocks. Things, boxes, circles, and other shapes to show functional objects. Relationships, lines and arrows to connect objects and explain how they interact. And diagrams, combinations of these to capture the big picture. Using this simple logic, you can describe very complex systems with modules, services, databases, flows, or even movies. Here's a recent Netflix Bandersnatch. In 1997, Object Management Group standardized UML as a general purpose modeling language. It also became ISO Standard in 2005. Now the language is widely used to describe architectures. UML isn't the only way to describe architectures. With the growth of cloud services, their providers also suggest their individual visualization methods. And that's how business needs are first translated into user stories and then into the format that engineers can work with. The ability to win the hearts and minds of our customers while explaining abstract technical concepts as they relate to business value is a must. And this is a quote from the job posting for a solution architect in one of the tech companies. Solution architects do lots of talking. Diagrams look static, but making them reiterating and tweaking entails a lot of communication with different parties. Stakeholders and project managers to capture the business problem and pitch a solution. Business analysts to work over requirements. Enterprise architects to fit a solution into the corporate ecosystem. Engineering team and software architects to break down tech details. Many people, besides that one guy that's always at the coffee machine. But who's qualified for this kind of job? Usually solution architects have engineering backgrounds and a pedigree of complex projects. Framing intricate requirements into an architecture takes top-to-bottom understanding of a product from the client side from images, operating systems and virtual machines to an infrastructure. Most likely a solution architect would understand a domain. There are architects who know the ins and outs of financial systems or healthcare software or travel, you name it. They may also have tech domains. Cloud providers like Amazon Web Services and Microsoft Azure offer architect certifications to illustrate their proficiency with these ecosystems. So look for that if you plan to hire one. And obviously you want the soft skills that keep all that communication humming. But does it work? In 2009, experts from Camp Gemini and the University of Amsterdam surveyed people from 49 software projects to understand the business value of solution architecture as a practice. They found that the use of solution architecture has decreased budget overrun by about 19%. It improves customer satisfaction and finds a better technical fit for project results. This doesn't mean, however, that all software projects need solution architects. If you build landing pages, integrate small modules, or make minor iterative improvements, most likely you won't need dedicated solution architecture. As one of the building blocks of enterprise planning, solution architecture remains in the realm of complex projects.


Let's talk about perennials. Perennial flowers, they are their prize for their natural beauty. And tonight, Harleen is going to introduce to us some of the more hard-eared types that require minimal care. So welcome again, Harleen, to the forum. Great. Well, I'm going to start off. Why include perennials instead of maybe something like angels? Because yes, of course, angels probably do have more flower color for you because they put all their energy into their flowers because they know they're going to die. Even the perennials are going to be there year after year. So if you don't want to plant every year, go with the perennials. You still can go and get color, but you might have to do a little bit more in design so that you have season on color when one finishes another, fills in, and takes over. And so your landscape may change a little bit with that. Of course, West Moline and therapeutic, I could probably talk another whole day about all the therapeutic aspects of just plants, period. But you can see those right there for benefits. And here are some more benefits, especially with youth. You can teach responsibility. That's what I'm going to do after this. I'm going to go over to Reed Johnson. We're going to go and plant up the dorm guys there. We're going to go and plant up flowers for their parents, for their mom to take home for afterwards. And so I got already small seedlings that are growing because I figured out of sight, out of mind, they'd never, if we planted a seed, they probably never water it. They'll go and see this plant. It's alive. It's growing. And hopefully they'll take some responsibility. But anyway, open. So what are the essential elements for a low maintenance perennials? Or, you know, I'd love a low maintenance perennials, but I don't think there is something like that. So because a perennial is going to die back down to the ground. So you're going to have to clean it up in the springtime. I wait until the springtime instead of in the fall because it's just so much easier to go and clean things for me in the springtime and let Mother Nature mat everything together. So first of all, it has to be hard enough for your zone. So I'll talk about that. And the ones I have tonight are hardy for all of North Dakota. They need to tolerate local summer conditions without supplemental care. So what that means is now if we have a little bit of a drought, they're not just going to die on you. They don't require any special soil type. And if they do better in a certain, you know, as far as moisture, you know, some will do much better in a more moist condition. So they're put, you know, as far as shade because they have more of a requirement for moist soil. So they grow well in the shade. Light requirements are appropriate for the location. So sometimes it's putting the plant in the correct location that makes it low maintenance. Because if you don't put them in the right location, you're going to have to do a lot of work to go and make them look okay. Some other things that you might want to have is that they're long lived.  So, and also infrequent division. A lot of times we don't divide enough. We could really rejuvenate these perennials if we go on a three to five year division.  They're going, oh, no, here she comes again.   And if you don't go and rejuvenate that, they're going to start to go downhill or they're not going to flower as much. And so you just need to go and do some division on those. But generally, other care, you shouldn't have to stake them. You shouldn't have to go. And DH means deadheading. I mean, who has the time to do that? Not me.  We'll stand summer heat, occasional drought, slow growing. A long bloom period would be good. But even if they did have that long bloom period, they got to have some attractiveness to their leaves. And I'll have a few examples of some plants that really don't have the longest bloom period. But they have an attractive leaf. Or maybe that's the only thing they really have going for them is that. So we're going to get into the spring ones. And, you know, just the pulp, it may not be on everyone's list because first of all, it does kind of need that moist soil. But I have a nice area. And OK, I should set these up first. The P.S. needs part shade and or shade. You'll see a or if you're, you know, it's the pessimist versus the optimist. If you're the optimist, it needs part sun. If you're the pessimist, it's part shade. And so, and maybe you might be an FS, which is full sun. So this is just the light requirement. And then the flowering in the zone probably enough. And then your general height. And a lot of times if they're going to be one to two foot tall, they generally get about one to two foot wide. Some of them may be a little bit more clunger than others. But Jack and the pulp, it's really, it has a spath that is considered with a flower. And that then changes into this red fruit cluster, I guess. And so here you see it coming out of the ground in the spring. Those are that typical leaf and your spath. And then that changes with time into this cluster of red berries. And really you see these growing a lot in shade. And now I have someone like I said on the north side and they just kind of come up amongst some of my other stuff. They're just kind of there. I never do anything to them. And so we'll maintenance that way for sure. Blue star flowers in the middle. And I kind of, I like because everybody's flowering. You see a lot of flowers. And you really, it doesn't have any kind of pest problems. It tolerates drought. And it probably is best and most so. But I've seen it in just, I mean, we have one on the east side of Lofscar out in full sun. And it does fine. And they don't do any watering around on that area. And it's just looking great. And so where I have less flowers under part sun, you might not see as many flowers.  And here's some of those, some cultivars that you might find in the available. So wild ginger, Canadian wild ginger. There's also European wild ginger. And really if you need something as a ground cover, I think these work quite well. Really doesn't have anything for a flower, makes though a good thick ground cover. And something that gets that nice and thick as a ground cover, you don't have to worry about pulling weeds in between. And so does quite well as a ground cover and, and hardy. So there it's coming up in the spring and you can see that little flower very small. And then it, you know, disappears and leaves open up more and you don't even see it close up of that flower, not too attractive. So this is more for those heart shaped leaves that just, you know, really do a wonderful job of making a ground cover and filling in an area. Now a lot of these that you're going to see are really native prairie plants, forbs. And so false indigo is one of those. And, you know, it gets rather tall, but it's one of those plants that you can just, you've got a spot for it. You don't have to have a rather large spot for it, because you can see three to four foot tall, three to four foot wide. But it'll be as contentive as it can be and it doesn't spread.  They want to take over that spot, that spot, that spot and just keep on going. False indigo, just happy, cool pea-like flower that then becomes a seed capsule that is used. A lot of times you'll see it used in dried arrangements. The leaves, the flower, you can see a little bit there. It was 2010 prairie plants of the year. You can see some of the clusters there. There's the seed capsules. And then in the fall, the dried ones. So now, bleeding hearts. So fringe bleeding heart is much better than any other, you know, your common bleeding heart.   Middle summer. What was there?  You don't have to worry about that with the fringe bleeding heart. And so much more petite, smaller. And it doesn't go dormant when you have any kind of a little drought come. And so there's also the western bleeding heart. I don't know anyone who can really tell them apart unless some botanist really wants to. But the western bleeding heart, you'll see a little bit different colors. Now, here, the Siberian iris. This is one of those things that is, you know, they don't flower a long time. But then when they're done, they're very easy to clean up and you don't have to do anything. I got this clump in this area that gets a little bit more moist. And so I've gone through a number of other perennials. Boom, this one's just happy there. If we get those rings and water collect, it doesn't die. And they've got the flowers, then you have these sword-like leaves. And so that's what I think it's really nice about. But then you get all different kinds of colors and there's like a thousand different colors of that. And last for the spring flowers, so then would be South Dakota State Flower, the past flower. Comes up very early in the spring in flowers. Again, then as it's, when it's done, its seed capsule is kind of like your climatic. A little wispy. And hopefully, you know, maybe I don't have an example of that, but you can see how early, this gives you some color really, really early. And there's the seed capsule in the corner there. And so moving on, if there's one year old that I think has, now it would be the firmly feral because again, only that thing, it only comes in yellow. But unlike the other year old, there's no soil which creeps all over. This is very content. So, you know, it'll stay into this clump. It won't take over. There's been some on campus probably for 20 years staying clump. And I'll make it a little bit bigger, but it isn't trying to go all over the place. See deer and rabbit resistant. They're really good. So let it yellow. Pretty yellow though, bright yellow. Ladies mantel. Another one that, you know, I think ladies mantel is more for the leaves. You get these fuzzy leaves that you can see on this picture while they're holding on those little beads of water because of all the little hairs on the leaf. I don't think so much of the flower cluster, but the flowers actually are long-lived. And I've seen them used. They'll put them into little vases and they'll last for a long time. And so there you see that close-up of the leaf. That's really, I think, what is great about this plant. I have had it go and actually seed occasionally, but usually you don't have it self-seeding. Prairie grasses, of course, prairie grasses are all going to be low maintenance in which basically you just let them grow in the springtime. They'll go and chop off down to, you know, give about four to six inches and remove the rest above and it'll go and sprout out. And so big blue stem is one of those. Another rather tall plant is goat's beard. And, you know, this one, well, this one's kind of marginal as far as, you know, you kind of have to work a little bit more with it if you want to go and keep it from getting some leaf margin burn. And so, but it has male and female flowers. You can see the one up on the above. That's male flowers. That's showier. They've got to strut their stuff better than the female flowers. Another grass. Now, this one, if you don't see this one on every block in the neighborhood, then they're not into grasses too much because feathery grass is probably the most common ornamental grass I've ever seen. Because, especially the Carl Forester, it's gorgeous. And it's so, it just has, it's self-care free, easy to take care of, and has an attractive flower or seed head to it. There's the Carl Forester. Flower's a little bit two weeks earlier. It's a little bit shorter. So, it works into a lot more of your landscape. And it has then, as it ripens, you get that nice yellowish gold color. There are some variegated ones that probably, they aren't near as hard. But if you continue to get variegation into a species, you give that something. And a lot of times, it's winter hard enough. So, with a lot of these, if you see variegated, think about it's not going to be as hardy. Of course, there's another native forb, prairie forb, butterfly weed, in which it's a milk weed and has those orange flowers. We're getting a little bit more of the breeding to get some other colors. But, you know, as far as the monarch butterfly loves the milk weed and so, one of those. And very slow growing, never have to worry about it. Doesn't transplant those. So, don't try to transplant it after you've gotten it established. It just doesn't do well.  I love to say some of the suffuga, but blacks may correct. Because it really does require a lot of, it's best in moist soil. And if you don't have moist soil and don't do a lot of mulching to keep it moist, you're going to see a lot of leaf marchin burn. It's not going to look so attractive. But, otherwise, you wouldn't have to do much with it. Now, they have a lot of these purples that are really attractive. So, of course, another prairie, forb, echinacea, purple cone flower, probably is seen in everyone's ornamental planting. But, you know, recently they've come out with a lot of these double flowers and like that. They're white.  You get something like that, rasmutaz. But, they've also come out with other colors. Now, a lot of these aren't as hard to either. So, you have to be careful with that. Another grass, a blue fescue. Now, I cut off the seed heads because you just like those little, narrow, bluish-green, whitish-blue leaves that you see there. Hostas, of course.  Hostas are great. They come with all kinds of variegated colors. So, not that you need this flower that you see here. The plantagenae is supposed to be the fragrant one with the flowers. But, they don't have as much color, variegation in colors as some of these other ones. You get the wavy leaf, hostas, and like that, the blue ones. You know, they've got that giant one that's supposed to have size. It doesn't have leaves like, you know, something else. Of course, peonies, hybrid peonies. Very traditional. Been around for a long time. I don't think I need to go. And I think everyone knows, well, they're pretty darn hard to even kill.  My mom's neighbor, every summer, after they were done flowering, she'd mow them down. And just weekly mow them down. And 30-some years later, they were still there. They weren't flourishing, but they were still there. So, pretty amazing. You know, the doubles aren't as clean as the singles. The singles, you don't have to worry about doing anything with. Then you have the doubles for the flowers. And then you get a rain. That's when it gets a little ugly. So, if you really want low maintenance, go with the singles. I think they're gorgeous. That huge cup, you know. But everyone's like, oh, we can get all the extra petals with the doubles. Selvia, please. The last few years, Selvia, perennial. Selvia has just been amazing.  And sure enough, it's perennial Selvia. I don't know what happened these last two summers, but it's just been gorgeous. The most robust purple. And again, very easy to take care of low maintenance type of plants. I really like Selvia, perennial Selvia. Not a lot of different colors, though. Okay, just three more, I think. Blazing Star, unique. Again, it's a parade for it. But most of your flowers will go and they'll send up this flower stock. And from the bottom, they'll keep elongating. And so your most mature flower is at the bottom. And your least mature one is at the top. Liatras, they're gave the other Blazing Star just the opposite. It starts from the top down. And so that and legularity, I think, are the only two that do this that I can remember. But there's a lot of different species, but I see these used. The flower lasts a really long time. I see them even used for cut flowers and some of the different colors. But usually, you'll see that in the purple. And there's a whole bunch of different fords and species that have just difference in their flowering characteristics. Sedum, or Shoei Sedum. Again, this is one of those things that if you can't grow a Sedum, don't try anything. And now you can get some of those that like the purple one that tends to get over too bushy and then it kind of flattens out. But a lot of these, if you divide every three to five years, you shouldn't have any problem with them starting to lose their mound and split open. But, I mean, if the dog runs through and knocks off a few stems, stick them in the ground, they'll run, and you'll have more plants. They're just so easy to take care of. The only bad thing is they flower really late, and so you get that freeze and cut up three days of nice color, and then that was it. So that's the worst part about Shoei Sedum.  So, but I really like the, well, Indian chief, I like it because it's a deeper red. I don't like the blue-screened leaves and then something like starburst white. It's just I don't see the kind of contrast with that. Some of these other ones. So, not too bad. Not too bad. Where did you train? Okay, let's get it going though. Okay, first of all, a question. Do you have a recommendation for a crinial that would do well in a moist area? Well, no, low-spotty. Low-spotty? How about a low-spotty? You can go raspberries on it, so wet. Well, you know, I do think there would be a number of irises that do really, really well. And so you can get into your Japanese irises. You yellow irises, they actually plant them right along the margin or in the water. So irises, I think, would be not your bearded irises. You know, I wouldn't go with a bearded iris. How about, do you have a professional opinion about Lillie of the Valley? As I did in my Colorado moist area. Be careful. It will take over the world. I started with a cloth like this. Now it's bigger than that table and I keep digging it up and I think that I'm just spreading it more. And so... Well, I didn't know that was a bad thing. I don't see something spread here in North Dakota. How about a... When's the best time to divide a P&E? Spring or fall? Or do you have some general recommendations? We get this question a lot about when should you divide a perennial? I like to go and do it in the springtime because then you're not taxing... Okay, get it established and then it goes into winter. I wait and do it in the spring so I know... I don't have to worry about it trying to survive the winter. And that, you know, did I do it early enough? So, I like to do it in the springtime. I know what I'm working with.  They start to have their little combs poking up. Good time to go and divide it. And then I don't have to... You know, when you do it in the fall, you got leaves and then you're damaging leaves. You're damaging a lot of roots. You got transpiration. And so you're losing moisture and you may not have enough of a root system there. And so you're really stressing it. And then it goes into winter. You do it in the springtime, you avoid those kinds of things. Okay, thank you. How about as a general rule, never divide a perennial when it's blooming? What can you do with thinking about that? Well, why would you want to divide it? Because I don't know anyone who could go and divide something and not make a mess of things. At least I, you know, I try to divide something when it's flowering. I break off this stock, that stock. And so enjoy the flowers while you can. Okay. How about what's the best way to tell the difference between salvia and lavender? Well, lavender is not going to, isn't going to last very long. Because the snow, yeah, different. And salvia, just like that crinkly leaf that I don't see necessarily. What's the difference between salvia and veronica? Oh, veronica. Okay. The question was, what is the difference between salvia and veronica? Now, those might look a little bit more similar. But veronica stays, a lot of them stay smaller and send up a single spike. And so salvia, again, veronica doesn't have that kind of crinkly whole leaf on it. And what veronica, yeah, spicata, it stays small. And then you have some other veronicas like blue, blue, something. And those are hybrids.  But they have different flowers. They have more of a cone type flower. And your salvia is more of a spike. Okay. Thank you. And we're going to have to cut the questions now. I know there's some more questions to be asked. But we do, we want to stay on schedule as best as we can. And we will do our best to address these extra questions during our overtime session at 8, or approximately 8.30 tonight. So let's thank Arlene for the perennial presentation. Thank you, Arlene.


Autolight and its 96,000 dealers present Suspense tonight Autolight brings you Miss Claire Trevor in angel face a suspense play produced and edited by William Spears The flowers are blooming this spring Hey, hey, hey, hey, you better save that for the bathtub or consult a voice expert seems to be an expert for everything these days Everything from voices to spark plugs take auto light spark plugs for instance They're designed by auto light ignition engineers the men who engineer spark plugs just as they engineer coils Distributors wire and all the other important parts of the ignition system to work together as a perfect team Why the skill of these ignition engineers has made auto light spark plugs world famous for their quality and dependability Ignition engineered auto light spark plugs you might say yes, sir And these same auto light engineers developed the famous auto light resistor spark plug one of the greatest advances in spark plug design for Automotive use in the past 20 years real experts. Hey, welcome, right you are hap so friends See your friendly auto light spark plug dealer tomorrow have him replace worn out spark plugs with world famous ignition engineered Auto light spark plugs and whether you choose the resistor type or the regular type You can't buy up better spark plug for your car because you're always right with auto light And now with angel face and the performance of Claire Trevor auto light hopes once again to keep you in Suspend did you wish to see I see her already if you don't mind out of my way Really you're supposed to get yourself announced honey. The name is Wheeler. Does it mean anything to you? I had caught Ruby Rose Redding the noted burlesque queen at breakfast time hers not mine Quarter to three in the afternoon Breakfast with a tomato and lettuce untouched and a glass of bromo seltzer She banged her cigarette to death against an ashtray and looked me up and down Like auditioning me for her current chorus line Wheeler, what's it supposed to mean? Are you his mother? I'm a little young for the part But I'm the only one he's ever had.  I don't mind coming right out with it in front of you made if you don't yeah Take a look around the block a couple of times Well, take him again, maybe you can get him tomorrow already I'll give it to you short I Want you to lay off my brother your brother now. Let's see. What am I supposed to have done to him? He's been spending money like wild money that doesn't come out of his salary and another thing He started wearing a gun about two weeks after you started wearing him Did I teach him to shoot it too? No, but I think some of your friends may have how you've been reading the gossip columns Well, I read them too some big gun from Philly is supposed to be paying my rent Forget it, honey.  Would you? Pick on someone your own size well of all Hey, I've heard of wives pulling this bit and even mothers and once in a picture. It was the old man now It's a sister. Well, send grandma around the mall Beat it. I walked out past her if she touched me. I I think I'd have murdered her Oh, I went for a walk. Why aren't you at work? I Quit my job. You did Chick You aren't going to Chicago with that dame. Are you why Chicago? Well, it's in variety Ruby roses opening a club there It also says a big shot gangster from Philly is backing it. Oh, what? Well, does he know about you? What are you trying to do scare me off?  That wouldn't work all that about that guy That's nothing but publicity Ruby told me all about. Oh, yeah, sure Look sis, you don't know her like I know chick.  No cracks Jerry Look Ruby rose ready makes good money. All right, but the rent on that plush line Wraptrap she lives in if you can call it living is around 3,000 a month not somebody's paying those bills and nuts What do you really know about more than you ever will leave? I'm in show business to you lay off. Will you? Chick wait listen. I heard enough chick.  You're heading straight for the eight-ball chick Chick don't go to her Disalve to girl sitting at kitchen table playing very solitary solitaire About four that morning. I was looking for an aspirin When the doorbell buzzed chick I Yo cherry Wheeler, aren't you? He was nice.  I really couldn't say I've My crocs out of order miss Wheeler your brother was going to Chicago with Ruby Rose Redding the dancer you knew that didn't you oh Now why would he go anywhere with anyone with a name like that your brother went to the Alcazar apartments at 815 tonight and beat up This wedding thing what then he put his two big thumbs on her throat and throttled her until she was dead You're just saying that He's just saying that isn't me Well, he didn't do it please please he didn't do it All right, I I didn't know about him and Ruby Rose, but I've been on the squad eight years Angel face and we never in all that time caught a guy as dead to rights as your brother He showed up with his release in the foyer of the Alcazar at exactly 12 minutes past eight tonight He said to the dorm and what time is it that miss Redding center baggage down yet? We've got to make it train Well, she had sent her baggage down and then she changed her mind. She had it all taken back upstairs again There's your motive right there. Well, that doesn't prove anything She's the dormant rang her apartment and said through the intercom mr. Wheeler's here And she gave a dirty laugh and sang out I can hardly wait I don't see that so what so she was alive at 13 minutes past eight the dormant went out for coffee at 815 At 820 Ruby Rose asked the operator to give her the police. She was shrieking with fear at 832 I arrived your brother was crouched over shaking her and she was dead There were two thumbprints on her neck as well as the marks of a big signet ring where she'd been pummeled the initial W for Wheeler Is that a case or isn't it? What was chick wearing that ring when you arrested him? But there's a ring mark on his right third finger where he got rid of a ring He'd been wearing a long time. He pawned it. He needed money Lost it. Well, even if he did hit her How do you know somebody else wasn't in that apartment just before chick showed up? Where was that that French maid of hers discharged got her notice in two weeks pay and left around six story checks Did chick confess? Oh, no. No, he was crouched over shaking her trying to restore or he said all right All right, I'll tell you everything Write it down. Yeah, that's more like it. I went there this afternoon and told her if she didn't lay off my brother I'd kill her I Came back home found chick packed up ready to leave. I I tried to stop him. He hit Asked the neighbors. They heard his rowing. He struck me and I couldn't stand it I beat it up to her place in a taxi got there first went in the back way and and Gave her one last chance to leave him alone She wouldn't take it She was all soft and squashy and I just grabbed her by the neck and pushed hard I've got big thumbs, too Got all that color Yep, well that about puts a lid on it. Let's go. Yeah, let's wait a minute Yeah, ain't you afraid aren't you going to take me who wants you? Oh, you think this is one of those phony confessions Well, maybe the newspapers won't think so and they'll be right. Maybe you did do it at that. Maybe I'm underestimating you Oh, she's wearing pajamas. You're right. That was a good guess See you later, angel face The way he said it. I couldn't tell whether it was a threat or a promise. I Never saw him again until after the jury had found my brother guilty of murder in the first degree Auto light is bringing you miss Claire Trevor in angel face tonight's production in radio's outstanding theater of thrills Suspense Speaking of expert experts. Did you say that reminds me of auto light ignition engineers the men who design and build complete ignition systems Used as original factory equipment on many makes of America's finest cars. They're ignition experts So naturally they know how to build auto light spark plugs So they'll work as a perfect team with all the other important parts of the ignition system That means they know how to build into spark plugs the best in quick-starting smooth the performance and gas Marley J Harlow righto and say it's the skill of these same auto light engineers that made possible the development of the auto light Resistor spark plug one of the greatest advances in spark plug design for automotive use in the past 20 years edge-wise How's that again? Have just trying to get a word in edge wise Well, here's the last word hap see your friendly auto light spark plug dealer tomorrow Have him replace worn-out spark plugs with ignition engineered auto light spark plugs Whether you choose the resistor type or the regular type you'll be right because you're always right with auto light Now auto light brings back to our Hollywood soundstage our star Claire Trevor in angel face a tale Well calculated to keep you in Suspense I Was coming out of the courtroom in a wet hen frame of mind and Guess who had an umbrella? Hello angel face get away from me you put your hat a girl feeling better already Just keep out of my way wait a minute angel Let go of me you need help baby. I'm trying to give it to you All right Tell me the name of the little man who wasn't there wasn't where in that courtroom Standing trial for her murder in place of chick.  Why did you wait till now to say that we didn't give me anything to go on nothing But that phony confession of yours when it was better than the purge of testimony of that doorman and that phony French maid And you you set up there with your face hanging out and put in your two cents worth look angel face I was sent by my superiors and answer to the patrolman's call that night question check and put him under arrest Don't hold that against me. What do you care whether I hold it against you? You really don't know no Look in the mirror sometime and find out Right then I decided to stop holding it against him that he was a detective. I kind of like the guy and maybe Just maybe he could find a way to help clear my brother and Over a couple of Manhattan's in a cheery little bar. I told him so Uh-huh. Well, you got a plant angel face here. Are you going into a blind? I think I'll start with that French maid. Why her she left two hours before the murder. That's love Yeah, maybe so but she was grease plenty to soft petal of one right name that belongs in this case She may not have been there, but she knew who to expect around She may even have tipped him that Ruby Rose was throwing him over for chick But if she's been paid off enough to commit perjury, what makes you think she'll tell you anything? I'll club it out of her if I have to wait a second here Try this first use a little intimidation when it may work 150 bucks. Hey, aren't you married or anything? Not yet ain't no faith. I has her address Suzette Leblanc real name Susie White 435 West 54 The street had been roped off and there was a block party or something going on Anyway, it was noisy. I elbowed my way up the steps of 435 and found Susie White's name on the bell panel I broke my best finger nail on the button No response. I pushed into the hall past the drunk falling out And fought my way through the fumes to the stairway Her door was standing open lock was broken and the molding was a mess of splinters This was the point where the private eye always finds the dead body, but I didn't find anything Somebody had taken the room apart. I wondered if they had found what they were looking for When I hit the street again the ugly face front I had passed coming in was standing by the front steps And there was a fresh wood splinter clinging to the elbow of his coat I remembered the splinters on the broken door upstairs I stopped next to him and made like straightening my seams. I look good that way Then I started walking fast So did he I Pause and some of some steps leading down to a basement apartment and waited for him Hello, beautiful. Well, how are you? Just fine. Is this your house? Yes, indeed I'll buy a little drink, baby. Uh-huh. You'll fight me in. Sure. You go first He hit the line railing with his head on the way down and the paving stones at the bottom to the rest It was right there in the first pocket I looked in behind his wallet a letter addressed to Ruby Rose Redding and post mark Philadelphia I'm over here. Hello angel face. I know I know you didn't find her. I may as well tell you why she doesn't live there anymore What they fished her body out of the Hudson about a half hour ago. Oh What are you smart detectives calling it suicide? I might as well Listen, she was being paid off to keep quiet about a certain friend of Ruby roses He must have decided he didn't dare leave any bets uncovered. What's certain friend for instance this certain friend for instance I Watched his face as he read it. I knew every word of it by heart. It said dear Ruby I hear you've been running around with some punk. I don't believe a word of it But you better get rid of them before I hit town will be a new York Friday night milk PS the Chicago dealers off and giving you featured spot at the Calcutta instead About that punk drop him or I drop you milk So that's a missing name milk Miletus Lay's rough too runs a Calcutta club and Friday night Nick the night. She was killed Miletus the initial on that ring could have been M as well as W I never thought of that But Miletus spells real dough, baby. Your brother wasn't even carfair to ruby ruby. Oh, maybe she was scared of Miletus And maybe she wanted out could be you could sell me, but I convince easy from you, baby You're not selling it to the grand jury She had something on Miletus and that's why he couldn't afford to lose control of her Well, what ruby rose redding could squeeze out of a man I can yeah what he squeezed out of her He could also squeeze out of you namely your breath. Oh, yes, but I have a friend on the force Oh, maybe really rude ruby rose had a friend on the force too. Why do you say that? She didn't it would be the only department she missed Oh, what now come and catch my act at the Calcutta club I'm looking a little higher. Don't be coy. Look. I sing and dance and that's all you got any numbers with you numbers Look, what are we wasting time for? Let's meet the guy who does the hiring. Hey, wait a minute. Mr. Miletus wants you Where she says she wants to meet you mr. Miletus so why not? Well Hello Hello yourself. Oh, mr. Miletus. You know, I've always wanted to work for you Okay We start the Calcutta tomorrow night.  Hello Angel face, huh? Yeah. Mm-hmm. My name is say angel.  Good tag Sit down angel face I gave him my warmest high voltage smile and took out my compact I saw my eyes in the mirror and in each iris. There was a little electric chair with chicks strapped in it That made it a lot easier to keep the high voltage running through milt Miletus He was good-looking in a swarthy loose lip sort of way Nothing to tag them as a gangster except his name and a habit of keeping his right hand in his coat pocket all the time As if he were holding a gun We talked through the bottle of champagne and he ordered another I kept smiling because somehow I had to get into his apartment We went back to what we've been talking about and I said Well Some night I might just feel like changing the scenery around me Dissolve to girl entering the Miletus apartment on park avenue It was a duplex with a two-story living room and a bedroom opening Opening off a balcony I decided to start at the top and work down in case I needed to get out of there fast I gave his bedroom the suspicious wife treatment In the night table, I found the nine letters to ruby rose and a bunch of treasury certificates The kind doctors have to make out to get narcotics In a box containing studs cuff links and 132 caliber bullet I found what I was really looking for I never really expected to find A signet ring with the initial M on it And I got the bell I thought fast If Miletus was out of town everybody who knew his private number would know better than to ring it If he was checking up on me, I had nothing to lose so Hello Oh, Nick Oh, what a scare you gave me. Listen. I've got all I need some dope certificates The rest of the letters and best of all a signet ring with his initial How am I doing? Not so good. Miletus just got back sores the devil over a phony wire Somebody sent him a decoy amount of time Well, that was me only I thought he'd stay overnight at least How did you know I was here angel face? Did you really think I'd let you go into this alone? I'm downstairs now. I've been watching every minute. Oh, I wish I'd known I wouldn't have been so scared He ought to be scared. He left the club five minutes ago. It must be all right. Nick. Now look angel face Nick Someone just came in take it easy, baby sarge and colin.  I had kicked down the flight of stairs Miletus turned to him. Good offer. I'll call. Can't you see I have company? Wait a minute. That's her the dame I was telling you about Don't be calling my lady friends dames. I'm trying to tell you Come here angel face. Come on down. Don't be scared. Oh, sure mil. Why should I be afraid of you?   It's your party. Come here angel face. No, no, no. Wait a minute. What for? I'll a girl like some soft lights and a little sweet music, you know Okay, I'll go mix us a drink He came back from the bar on tiptoe with a drink looking like a cat stalking a bowl of cream I threw up my first line of defense.  I'm just putting on my face. Oh, you don't need it My pearl leave it there. Oh, I will not those are my things You haven't any right. Hey, what's that just things of mine some letters. Let me see though. No What else did you lift out of my room?  Stay where you are. What's up, Rocco? I try to tell you about the game She's trying to frame you. She's wheeler sister. You weren't at the trial. You weren't filled.  There's cops all around the building two detectives coming up. Okay, Rocco My cheap little I'll fix you even if they get me. I'll fix you you're wrong milt I took those things because I knew they were coming here. I did it for you. Oh for me. She's saying I looked up at miletus There was blood running down the part of his hair He dropped down to his knees beside me like he wanted to apologize But he didn't say anything then he fell over on his side and didn't get up anymore His right hand the one he always held in his coat pocket was grabbing at the rug There wasn't any thumb on it And there'd been two big thumb marks on ruby's throat That wasn't necessary nick You didn't have to kill him He let go of her when we came in I I know I I shouldn't have but he had that right hand in his pocket always I thought he had a gun in there All right, man Anyway, I saved the state some money. You got that stuff angel face. It's over there in my purse. Good Nick Ruby rose was throttled by a man with big hands two big thumb marks on her neck Look at miletus No, right thumb. Yeah That's bad angel face. I make What? This is ring here Isn't this your fraternity ring Don't move nick Yeah Yeah, I guess it is Do we have to mention all this coli? Yeah, nick I think we'd better So that's about it captain As you know sir nick met ruby rose while he was doing undercover work on milk miletus Yeah, but I didn't know it'd gone on seeing her after he left the assignment I guess nobody did except me Yes, that and sergeant coli clinched the whole thing with that ring. Well miss wheeler the Ring clinched the case all right, but not against miletus An m upside down see is a w But an m sideways Is a greek letter and nick planted it in miletus's apartment for me to find, huh? It was nick's fraternity ring. Yeah, that's right He was probably afraid you were digging too deep to find the real killer So he killed miletus to make you drop the case. Oh his plan might have worked without coli here Well miss wheeler, this isn't exactly good news, but thank you for filling in some of the blanks. Oh captain. Yeah When will chick be free? Uh, it takes a little time, but he'll be home I guess he learned about women the hard way, huh? Oh, he hasn't learned anything yet That kid brother of mine. Just wait till I get him home Suspense presented by auto light tonight star miss claire trevor Hello, the expert will clocks the bright never will desert the cause of auto light Because folks auto light makes more than 400 products for cars trucks planes and boats in 28 plants coast to coast These include complete electrical systems used as original equipment on money makes of america's finest cars Generators coils voltage regulators wire and cable starting motors electric windshield wipers All engineer to fit together perfectly work together perfectly because they're a perfect team So friends if your auto light equipped car needs replacement parts Ask for and insist on auto light original factory parts at your neighborhood service station car dealer garage or repair shop Remember you're always right with auto light Next thursday for suspense our star will be denis o' keith the play is called very much like a nightmare And it is as we say a tale well calculated to keep you in suspense Tonight suspense play was produced and edited by william spear and directed by norman mcdonnell Music for suspense is composed by luchin morroweck and conducted by lud bluskin Angel face is an original play written for radio by cornell woolrich Claire trevor may currently be seen co-starred with fred mcmurray in borderline In the coming weeks you will hear such stars as charles boy a edward g robinson and jack carson Don't forget next thursday same time auto light will present suspense starring denis o' keith Auto light resistor or regular spark plugs auto light stafel batteries auto light electrical parts at your neighborhood auto light dealers Switch to auto light. Good night This is cbs the columbia broadcasting system