1 00:00:00,110 --> 00:00:03,889 ZOZ: I'm Zoz. This is "Hacking Driverless Vehicles." 2 00:00:03,889 --> 00:00:08,490 My academic background is autonomous robots, but I'm probably best known for a little cable 3 00:00:08,490 --> 00:00:14,980 TV show I did with Kingpin called "Prototype This!" We did some autonomous robot stuff. 4 00:00:14,980 --> 00:00:20,980 Tried to hack together cool demos. For example we did a UAV that would protect 5 00:00:20,980 --> 00:00:24,410 unguarded beaches. So press a button if you are out in the ocean in trouble and it would 6 00:00:24,410 --> 00:00:31,410 fly out and drop of you a life preserver. We learned a lot about unmanned air systems 7 00:00:31,660 --> 00:00:36,510 on that episode and a lot about it racing in the design process. 8 00:00:36,510 --> 00:00:41,350 We also turned our hand to driverless ground vehicles for one of the most pressing technological 9 00:00:41,350 --> 00:00:47,120 challenges in the ground space. Which is maintaining the American lead in high‑speed pizza delivery. 10 00:00:47,120 --> 00:00:53,130 Here is the solution for local delivery. Sharing routing space with pedestrians. Here is the 11 00:00:53,130 --> 00:00:58,980 long distance method. Operating within the shared automobile network. 12 00:00:58,980 --> 00:01:03,790 This was the first ever autonomous crossing of a U.S. highway bridge. Can be hard to convince 13 00:01:03,790 --> 00:01:10,530 human delivery drivers to make this trip. So screw them. Let's let the robots do it. 14 00:01:10,530 --> 00:01:15,420 One of the things I've 'been doing is hosting autonomous vehicle competitions for robo nation. 15 00:01:15,420 --> 00:01:20,540 These are student competitions, university level and below. Ground vehicle competition 16 00:01:20,540 --> 00:01:23,080 two air vehicle competitions ones for boats and one for submarines. 17 00:01:23,080 --> 00:01:30,080 I want to say a few words at the start about the motivation for this talk. I am a huge 18 00:01:31,570 --> 00:01:36,520 fan of unmanned vehicles. I love robots. I think they're the future. And they are definitely 19 00:01:36,520 --> 00:01:41,140 coming. Because there are so many advantages. Energy efficiency. Not having to carry a human 20 00:01:41,140 --> 00:01:45,280 driver. Not having to carry food and water, bathrooms or go out of its way to acquire 21 00:01:45,280 --> 00:01:51,119 them. Same with time efficiency. Not having to deal with fatigue, boredom, taking rest 22 00:01:51,119 --> 00:01:55,340 stops, operator changeover and so on. And all the new applications that are going to 23 00:01:55,340 --> 00:01:59,659 be enabled where it wasn't practical with a human driver. 24 00:01:59,659 --> 00:02:06,520 The revolution is coming. You can't stop it. Even if you want to, it's here. 25 00:02:06,520 --> 00:02:10,399 (Laughter) But like everything else that humans have 26 00:02:10,399 --> 00:02:15,370 ever made, these systems are going to be hacked. So I want to start that conversation now. 27 00:02:15,370 --> 00:02:19,489 I want to start talking about it before the systems are too entrenched for us to go back 28 00:02:19,489 --> 00:02:23,340 on decisions. So what I am definitely not trying to do here 29 00:02:23,340 --> 00:02:27,620 is spread FUD ‑‑ fear, uncertainty and doubt. It's not going to be some kind of alarmist 30 00:02:27,620 --> 00:02:33,450 anti‑robot propaganda. You like that one? (Laughter) 31 00:02:33,450 --> 00:02:40,450 How about this one? That's not it. It's not every presentation where I get to make two 32 00:02:43,620 --> 00:02:50,620 Hitler jokes on the same slide. I couldn't resist. Hope it's not too soon. 33 00:02:51,690 --> 00:02:56,819 But I think that the, you know, this revolution is coming. And really got a sense of this 34 00:02:56,819 --> 00:03:02,760 recently ‑‑ this is footage I shot a year ago at SAUC‑E. This is fire scout. 35 00:03:02,760 --> 00:03:08,970 It's autonomous, unmanned robot helicopter. And it's doing a takeoff run there. And it 36 00:03:08,970 --> 00:03:13,750 just looks like it's sitting still on the ground right. It's so stable. You look at 37 00:03:13,750 --> 00:03:18,580 that and think, yeah, that's how to do it. Let's let the robot drive. I'm not trying 38 00:03:18,580 --> 00:03:23,959 to interfere with this process. This is my point. DEF CON is not a security conference 39 00:03:23,959 --> 00:03:28,370 per se, it is a hacking conference, a hacking convention. To me that's hacking in the old 40 00:03:28,370 --> 00:03:32,870 school sense, right. Figuring out how things work. What is wrong with them if there's stuff 41 00:03:32,870 --> 00:03:36,370 wrong with them and how to improve on them. This talk is about getting people excited 42 00:03:36,370 --> 00:03:43,349 about contributing to that conversation. The analysis, discussion, design and acceptance 43 00:03:43,349 --> 00:03:47,750 of driverless vehicles. So you can think of this as a recruitment talk. Sort of like General 44 00:03:47,750 --> 00:03:53,379 Alexander's NSA keynote last year. The big difference being of course I don't want to 45 00:03:53,379 --> 00:03:59,440 see what you fab to on the internet. You can keep that to yourself. 46 00:03:59,440 --> 00:04:03,420 We're going to talk about vulnerabilities and interfering with driverless vehicles. 47 00:04:03,420 --> 00:04:07,970 But it's out of love. It's kind of tough love, right. Kind of like when you teach people 48 00:04:07,970 --> 00:04:12,989 to swim the Australian way. By putting them in a pool and throwing in a crocodile. 49 00:04:12,989 --> 00:04:17,939 (Laughter) This is true. This is how we all learn. 50 00:04:17,939 --> 00:04:21,609 Speaking of managing vulnerabilities, right, there's the fire scout, it's very stable, 51 00:04:21,609 --> 00:04:27,599 but notice the people operating that robot. They're hiding behind the start CART. Just 52 00:04:27,599 --> 00:04:32,949 in case anything goes wrong. That's a security mindset. So when I talk about exploits and 53 00:04:32,949 --> 00:04:37,349 counter measures I want you guys to think about counter counter‑measures, right. Stuff 54 00:04:37,349 --> 00:04:41,610 that I say, well okay here's how we design around that. Here is how we fix that problem. 55 00:04:41,610 --> 00:04:47,379 Increase the robustness of the system. So unmanned system space is kind of wide. 56 00:04:47,379 --> 00:04:51,409 Means basically no human driver or pilot on board but doesn't necessarily mean you don't 57 00:04:51,409 --> 00:04:58,409 have someone off board controlling or supervising it for an unmanned system. Not actually autonomous. 58 00:04:58,699 --> 00:05:01,649 You might actually have a safety pilot on board or might be carrying passengers. There 59 00:05:01,649 --> 00:05:06,689 might be people on there but they don't have a direct control role. 60 00:05:06,689 --> 00:05:10,050 Of course the military are early adopters. Most of this field has been adopted by military 61 00:05:10,050 --> 00:05:16,649 spending and applications for a number of reasons I'm sure you can guess many of them. 62 00:05:16,649 --> 00:05:20,930 A pretty significant amount of uptake has been the airspace. For example this is global 63 00:05:20,930 --> 00:05:27,539 hawk. If you look at global hawk flight hours, it's looking pretty much like an exponential 64 00:05:27,539 --> 00:05:32,869 curve, right. That's because it works well. There's a lot of simplifications that apply 65 00:05:32,869 --> 00:05:37,860 to the airspace but they really want to push the changes down to other domains. 12 years 66 00:05:37,860 --> 00:05:44,860 ago congress insisted one‑third of all operational ground vehicles were to be unmanned by 2015. 67 00:05:46,159 --> 00:05:50,139 Now congress can put up the money and say whatever the hell it likes. Doesn't mean they're 68 00:05:50,139 --> 00:05:53,800 going to get it right? We're clearly not going to make the 2015 deadline but the will is 69 00:05:53,800 --> 00:05:59,319 there. I'm not going talk a lot about military vehicles in this talk but the capabilities 70 00:05:59,319 --> 00:06:03,949 are largely classified so it's very speculative. They already have a really active interest 71 00:06:03,949 --> 00:06:08,979 in resistance to adversarial engagements. So they already think a lot about the security 72 00:06:08,979 --> 00:06:15,439 stuff. They have the highest quality sensors and, of course, hopefully most of us will 73 00:06:15,439 --> 00:06:20,589 never encounter one. Unless Edward Snowden is in the audience. Hopefully we'll not encounter 74 00:06:20,589 --> 00:06:24,819 one of these. Here's just a quick example of a military 75 00:06:24,819 --> 00:06:29,949 UGV. A driverless ground vehicle specifically designed with threats in mind. You can see 76 00:06:29,949 --> 00:06:35,449 a lot of sensors. Designed looking for people. It's got weapons on board. You just have to 77 00:06:35,449 --> 00:06:40,889 get close enough to this thing to press one of these kill switches. 78 00:06:40,889 --> 00:06:45,339 (Laughter) It's kind of like putting an unshielded reactor 79 00:06:45,339 --> 00:06:52,339 exhaust port on a death star right. Presumably they'll remember to remove those in the final 80 00:06:53,419 --> 00:06:56,619 version. Let's start thinking where these things are 81 00:06:56,619 --> 00:07:02,659 going to show up in our backyards. So we've got transportation. Nothing bad about these 82 00:07:02,659 --> 00:07:07,539 guys by the way. We love these guys. They're friends. Not singling them out but they're 83 00:07:07,539 --> 00:07:14,539 pioneering stuff in transportation. Oceanography. Film making is big. If you see the film making 84 00:07:14,919 --> 00:07:19,969 conferences you see UAV's everywhere. Some of the weird esoteric ones you wouldn't think 85 00:07:19,969 --> 00:07:25,539 of like power line inspection with UAV's and, of course, logistics like the pizza delivery. 86 00:07:25,539 --> 00:07:31,029 Lots and lots more. And there are two main priorities that industry 87 00:07:31,029 --> 00:07:38,029 advocacy group has for unmanned systems in the civil sphere. Precision agriculture. A 88 00:07:38,999 --> 00:07:45,999 lot of combine harvesters now are practically mobile robots. And secondly self‑driving 89 00:07:46,239 --> 00:07:51,869 cars. This is big right. Wide applicability over the entire country. 90 00:07:51,869 --> 00:07:56,929 Roadblocks to up take the civil domain. Shared infrastructure. They have to share the stuff 91 00:07:56,929 --> 00:08:01,029 with humans. It's much tougher if the robots have to interoperate with humans. Hand in 92 00:08:01,029 --> 00:08:08,029 hand with that goes acceptance. Do they trust the safety. You got to convince the public. 93 00:08:10,349 --> 00:08:13,959 Also privacy is important. So we're talking about safety and robustness, 94 00:08:13,959 --> 00:08:19,300 the fun stuff here is failure. Let's take a look at that. Here is a couple of classic 95 00:08:19,300 --> 00:08:26,300 failures. First of all a UAV failure. This is the OCU 3 star. There's a surviving example 96 00:08:27,679 --> 00:08:34,679 in the Smithsonian. It was supposed to cost $10 million a unit ultimately. The first 97 00:08:36,129 --> 00:08:41,149 couple were very expensive. The first prototype failed on its second flight test. On it's 98 00:08:41,149 --> 00:08:45,440 second takeoff actually. I've seen the video of this crash with my 99 00:08:45,440 --> 00:08:50,829 own eyes but was not able to obtain it for you guys. So you just have to imagine. Here 100 00:08:50,829 --> 00:08:56,779 is a non‑crash takeoff. Just extrapolate from that and imagine if it didn't take off 101 00:08:56,779 --> 00:09:01,750 like that and its keeping down the runway and starts to wobble up and down and oscillate. 102 00:09:01,750 --> 00:09:05,629 And the oscillations get more and more and it pitched its nose up and comes down hard 103 00:09:05,629 --> 00:09:12,110 and it's a huge fire ball and is millions of dollars down the trade. This is a quote 104 00:09:12,110 --> 00:09:18,870 about what happened. Here's what as it was explained to me by researchers involved this 105 00:09:18,870 --> 00:09:24,040 is what happened. They had modeled the takeoff run with a flight control system on an asphalt 106 00:09:24,040 --> 00:09:30,279 runway. The second flight takeoff was on a pre‑fabricated concrete runway. The cracks 107 00:09:30,279 --> 00:09:37,019 between the concrete panels were under damped and they set up this oscillation that eventually 108 00:09:37,019 --> 00:09:42,230 caused the failure of the vehicle. Right. So just small impulses. 109 00:09:42,230 --> 00:09:48,160 The moral of that story is the expectations of the designers are critical. Even a seemingly 110 00:09:48,160 --> 00:09:52,449 trivial detail like runway composition can mean the difference between success and failure 111 00:09:52,449 --> 00:09:58,829 of these systems. If there's going to be exploitation there's a good chance it's going to happen 112 00:09:58,829 --> 00:10:03,290 at these cracks between the boundaries of the designers expectations. 113 00:10:03,290 --> 00:10:09,800 Here's a second example. This was the favorite vehicle to win the first challenge in 2004. 114 00:10:09,800 --> 00:10:15,550 This is the desert race from Los Angeles to Las Vegas fully autonomous. This vehicle 115 00:10:15,550 --> 00:10:22,550 was done by a CMU off‑shoot called red team racing. And it got a few miles. Seven miles 116 00:10:24,269 --> 00:10:30,439 or something like that, before it tack a hairpin turn wrong and ran off the side of the road. 117 00:10:30,439 --> 00:10:35,779 Its engine caught fire and it was all over. While we wait for that video to catch up, 118 00:10:35,779 --> 00:10:40,639 let's see if we can play both of them at the same time. Hey look at that. There it is. 119 00:10:40,639 --> 00:10:43,550 Failed. What went wrong? Apart from the fact that 120 00:10:43,550 --> 00:10:49,750 diesel engines don't like running on weird angles? They had a huge team. They had extensively 121 00:10:49,750 --> 00:10:54,600 mapped the course beforehand. Even though they only got the final route two hours beforehand 122 00:10:54,600 --> 00:10:59,519 they knew every road in the area. Very precise. They had people walking that course with GPS 123 00:10:59,519 --> 00:11:04,629 receivers. One of the people on that team told me their map was so good they could have 124 00:11:04,629 --> 00:11:08,680 just about made it on map data alone. The big problem was they paid too much attention 125 00:11:08,680 --> 00:11:13,040 to the other sensors. If they just ignored the laser range finder they would have gotten 126 00:11:13,040 --> 00:11:17,290 through just fine. The moral of the story is the robot faces 127 00:11:17,290 --> 00:11:21,870 a constant battle of deciding what it knows best. What information is reliable and what 128 00:11:21,870 --> 00:11:28,189 isn't. Correctly estimating the state you can't observe is the key to all decision making 129 00:11:28,189 --> 00:11:32,779 on uncertainty. So hacks and exploits have some of their best chances of succeeding if 130 00:11:32,779 --> 00:11:36,420 they subvert or undermine that state of summation process. 131 00:11:36,420 --> 00:11:43,420 Let's take a look at some of these logic structures at a higher level. Just like with humans we 132 00:11:44,740 --> 00:11:50,860 can think about behavioral logic of a robot in hierarchal fashion. At the bot we have 133 00:11:50,860 --> 00:11:55,060 control loops, stability maintenance. This stuff typically runs independently at a high 134 00:11:55,060 --> 00:11:59,279 cycle rate. Might run on a completely different computer from other stuff. Sometimes I'll 135 00:11:59,279 --> 00:12:02,959 see a robot that's completely crashed. It's not doing a damn thing but maintaining perfect 136 00:12:02,959 --> 00:12:08,970 stability in the water and air. Just hanging out there. Above that you might see some of 137 00:12:08,970 --> 00:12:15,300 that collision avoidance. That's preservation of the robot. Kind of like (inaudible) third 138 00:12:15,300 --> 00:12:18,300 law. Taking precedence over everything except low 139 00:12:18,300 --> 00:12:25,300 level control. Above that navigation and localization. That might be all of the mission in it's just 140 00:12:25,709 --> 00:12:32,649 navigation. Then above that high level mission task plan, stuff like that. 141 00:12:32,649 --> 00:12:37,660 So what can we take away from that arrangement? First of all, there's an implicit dependency. 142 00:12:37,660 --> 00:12:42,540 If you attack at lower level in the hierarchy you can defeat everything above it. If the 143 00:12:42,540 --> 00:12:47,199 robot can't maintain functionality there it can't at the higher levels. I like to think 144 00:12:47,199 --> 00:12:52,699 of it for those people who have office jobs no one thinks about filling out their TPS 145 00:12:52,699 --> 00:12:57,680 report while they're actually being kicked in the balls. You can go back to work and 146 00:12:57,680 --> 00:13:03,350 try that and see if it's true. But, secondly, more engineering effort might 147 00:13:03,350 --> 00:13:09,240 have been spent programming ‑‑ guaranteeing robustness at lower levels. Stability is super 148 00:13:09,240 --> 00:13:13,519 important but getting lost once in awhile you might be able to recover from that. So 149 00:13:13,519 --> 00:13:17,540 the lower layers might be juicier attack targets but might be better defended and harder to 150 00:13:17,540 --> 00:13:23,620 find bugs in. So a couple of examples that I mentioned before 151 00:13:23,620 --> 00:13:27,529 from prototype this just looking at the way things were arranged, the life saving drone 152 00:13:27,529 --> 00:13:32,930 had an auto pilot that did all the stability maintenance, has low level control loops for 153 00:13:32,930 --> 00:13:38,540 all the basic air worthiness for the different environmental conditions you might expect 154 00:13:38,540 --> 00:13:44,360 to encounter. Nothing in the way of collision avoidance; this is missing from just about 155 00:13:44,360 --> 00:13:49,779 all UAVs. That's one of the things that needs to implemented and designed in order for the 156 00:13:49,779 --> 00:13:56,779 shared space arrangement to happen. Navigational localization is GPS based. That has way points. 157 00:13:59,980 --> 00:14:04,800 And this involves also control loops. PID loops controlling all the aircraft approaches 158 00:14:04,800 --> 00:14:09,220 those way points and when it discovers ‑‑ when it decides whether or not its hit one 159 00:14:09,220 --> 00:14:11,519 or not or whether it should go back around for another try. 160 00:14:11,519 --> 00:14:18,519 Then at the top we had our bumming run planner that would set up a temporary weigh point. 161 00:14:22,610 --> 00:14:27,689 So the system of course fully vulnerable to collision. Because there was no effort to 162 00:14:27,689 --> 00:14:32,519 not be. And the high level logic depends on one single sensor ‑‑ the GPS. That single 163 00:14:32,519 --> 00:14:38,100 point of failure is a big vulnerability. And they're really common in the robot fear. 164 00:14:38,100 --> 00:14:44,660 Local pizza delivery has to have all kinds of control for stability maintenance because 165 00:14:44,660 --> 00:14:48,069 it's balancing on two wheels. It's got to do weight shifting for when the pizza gets 166 00:14:48,069 --> 00:14:53,980 removed and the center of gravity changes. Lots and lots of collision avoidance. Pretty 167 00:14:53,980 --> 00:14:59,129 much almost everything that the system does. The main strength is dealing with those dynamic 168 00:14:59,129 --> 00:15:05,470 obstacles that aren't on the map. At the high level, navigation by route planning from a 169 00:15:05,470 --> 00:15:09,899 map that's pre‑generated using simultaneous localization and mapping. That's where the 170 00:15:09,899 --> 00:15:14,240 discrimination between static and dynamic obstacles happens. Then the high level task 171 00:15:14,240 --> 00:15:19,259 is a simple one. Just to dispense the correct pizza when the correct credit card gets given 172 00:15:19,259 --> 00:15:23,730 to it. So this kind of system is vulnerable to redirection, 173 00:15:23,730 --> 00:15:28,509 trapping and map confusion attacks. All those things that attack where the robot thinks 174 00:15:28,509 --> 00:15:34,930 it is. And, of course, you can always try to get the pizza out if you didn't pay for 175 00:15:34,930 --> 00:15:40,319 it. Now that we're thinking about that logic hierarchy 176 00:15:40,319 --> 00:15:44,069 let's look at the operation of the hierarchy and what kind of logic is going on there. 177 00:15:44,069 --> 00:15:49,360 There's usually some form of state machine. Represents the mission and what the designers 178 00:15:49,360 --> 00:15:55,189 envisioned. So the robot considers itself to be in a state and can stay in this state 179 00:15:55,189 --> 00:16:02,189 or transaction to a new state. They define what logic the robot runs at any 180 00:16:03,569 --> 00:16:08,519 given time. These states may correspond to tasks and the transitions may be task completion 181 00:16:08,519 --> 00:16:13,139 or contact switches caused by things like priority shifts. Might be simple time‑outs, 182 00:16:13,139 --> 00:16:20,139 for example, anything like that. They may contain subordinate states. 183 00:16:23,300 --> 00:16:30,040 For math people watching, if this looks like a Mark off chain, there's a good reason for 184 00:16:30,040 --> 00:16:36,680 that. These feature a lot in robot control systems. The thing to be aware of with these 185 00:16:36,680 --> 00:16:40,610 machines is the machine is ‑‑ the state machine is not necessarily deterministic. 186 00:16:40,610 --> 00:16:46,319 Right? Just because we think we're in a state doesn't mean we're actually there. And so 187 00:16:46,319 --> 00:16:51,100 there's this hidden state that the robot has ‑‑ that it can't necessarily observe and has 188 00:16:51,100 --> 00:16:58,100 to figure out. That's where things get tricky. So to put some labels on this stuff and not 189 00:16:58,759 --> 00:17:03,129 have it be totally abstract here is the robosub mission. I've chosen this because it's kind 190 00:17:03,129 --> 00:17:08,730 of linear and easily broken down into various mission states. First the sub has to navigate 191 00:17:08,730 --> 00:17:12,539 through a start gate. This is often done open loop. Just point it in the right direction 192 00:17:12,539 --> 00:17:18,039 and drive it for a certain amount of time. But then you have to start making decisions. 193 00:17:18,039 --> 00:17:22,159 The sub's got to start looking for a buoy then trying to touch it. It's got to decide 194 00:17:22,159 --> 00:17:28,199 can it see the buoy? Has it touched it yet? Should we try again? You got some choices. 195 00:17:28,199 --> 00:17:31,850 And then you've got to start looking for a path on the bot of the pool. And you've got 196 00:17:31,850 --> 00:17:36,299 more choices once you find it. All the different sub tasks you might want to do. The obstacle 197 00:17:36,299 --> 00:17:42,159 course. Identifying the targets and dropping markers on them. Finding the torpedo targets 198 00:17:42,159 --> 00:17:47,480 and firing torpedoes through them. Then underwater manipulation task. You've got to determine 199 00:17:47,480 --> 00:17:51,510 the state of that in terms of finding it and whether you have managed to complete that 200 00:17:51,510 --> 00:17:57,520 task or make progress on it. You can time‑out from all those. A lot of these are vision 201 00:17:57,520 --> 00:18:04,520 guided but a certain point you can transition from anywhere in the graph. It uses a different 202 00:18:04,570 --> 00:18:09,840 sensor. So you've got to find the pinger and go and retrieve a package. 203 00:18:09,840 --> 00:18:14,480 So looking at this from the point of view of second guessing the designers, where are 204 00:18:14,480 --> 00:18:21,480 the vulnerabilities and potential exploits? There's the package. They may be in the state 205 00:18:21,929 --> 00:18:27,779 estimation. What does the robot think it's trying to do versus where it is. The transition 206 00:18:27,779 --> 00:18:32,289 between states. Can we spoof them or prevent them from occurring? Or are there bugs in 207 00:18:32,289 --> 00:18:38,690 the states themselves? Just unexpected conditions or results. Here is a key thing. When designers 208 00:18:38,690 --> 00:18:42,240 watched the robot in action they don't necessarily know even though they programmed the whole 209 00:18:42,240 --> 00:18:49,200 thing why it's doing what it's doing. They can only guess from that output. So until 210 00:18:49,200 --> 00:18:53,649 you see the logs you don't necessarily know. So the would‑be exploiter also has to put 211 00:18:53,649 --> 00:18:57,559 themselves in the mind of the designer and think about what might they have been thinking 212 00:18:57,559 --> 00:19:02,600 about and what might they have got wrong? But I don't want to talk about attacks that 213 00:19:02,600 --> 00:19:07,580 would work on any vehicle, even human driven vehicles, because there's no point in that. 214 00:19:07,580 --> 00:19:12,149 Like digging a big pit and camouflaging it I. Want to talk about relevant physical attacks. 215 00:19:12,149 --> 00:19:19,149 The input mechanisms on the sensors. They can be active or passive. Which is a important 216 00:19:19,720 --> 00:19:26,720 distinction that we'll cover as we go. Some common examples are GPS of course. Laser range 217 00:19:26,809 --> 00:19:33,809 finder, cameras, millimeter wave radar, visual compass and measurement unit, wheel encoders. 218 00:19:35,399 --> 00:19:40,169 Then for the specialty vehicles like subs we have things like Doppler velocity bloggers, 219 00:19:40,169 --> 00:19:47,169 scanning sonars, pressure fuses for the air. In addition to all this there's the map. We'll 220 00:19:48,940 --> 00:19:53,480 definitely talk about that, too. So sensors don't give a perfect picture of 221 00:19:53,480 --> 00:19:57,059 the world. Just the best guess. And you've got plenty of sources of uncertainty. They 222 00:19:57,059 --> 00:20:04,059 all know about noise of course. A constant battle associated with noise is drift. Latency 223 00:20:04,090 --> 00:20:10,179 and update rate come into play. So when we were doing the life saving drone, we had one 224 00:20:10,179 --> 00:20:16,110 update on the GPS. That meant with unknown time stamp on that. That meant that the vehicle 225 00:20:16,110 --> 00:20:20,399 could be anywhere within, like, a 70‑meter distance when we got that position estimation. 226 00:20:20,399 --> 00:20:25,809 Getting a position estimation that could is not valid to the tune of up to 70 meters 227 00:20:25,809 --> 00:20:29,279 that's hard to do a five‑meter precision bombing run. 228 00:20:29,279 --> 00:20:34,100 You have to model these uncertainties under various assumptions. You need to know what 229 00:20:34,100 --> 00:20:40,269 the underlying noise models are. A lot of people don't use GPS because they don't ‑‑ 230 00:20:40,269 --> 00:20:44,140 they can't get from individual units what the noise model of the GPS is so they can't 231 00:20:44,140 --> 00:20:50,000 develop a noise model that's super reliable. You might think fusing sensors together might 232 00:20:50,000 --> 00:20:54,580 be more useful than a single sensor. That's true in many cases. Fusing and registering 233 00:20:54,580 --> 00:20:58,529 together can be more useful than taking separate sensors. 234 00:20:58,529 --> 00:21:05,529 But what do you do when a sensor disagrees? Which one do you trust and how much? The robustness 235 00:21:07,230 --> 00:21:12,370 of the robot in the end may come down to how smart is it at discounting one single bad 236 00:21:12,370 --> 00:21:17,510 or spoofed sensor? Even though it might have a whole suite of sensors on board. 237 00:21:17,510 --> 00:21:23,140 So let's look at sensor attacks. Two basic kinds ‑‑ denial, basically preventing 238 00:21:23,140 --> 00:21:28,990 the sensor from recovering useful data. Then spoofing. Causing the sensor to retrieve information 239 00:21:28,990 --> 00:21:34,110 that is specifically incorrect that the attacker wants it to retrieve. Then you've got a basic 240 00:21:34,110 --> 00:21:40,450 attack mode choice. You can directly attack the sensors. Give them instantaneous bad data 241 00:21:40,450 --> 00:21:47,450 or try to mess with the aggregated data they're accumulating over time. I'm going quickly 242 00:21:47,649 --> 00:21:53,950 go through most of the common sensors and leave the specialized for another time. 243 00:21:53,950 --> 00:21:58,740 GPS we know is a major reference for vehicles that have access to it in the atmosphere. 244 00:21:58,740 --> 00:22:02,399 Denial is straight up jamming. You can buy a jammer from a number of sketchy Chinese 245 00:22:02,399 --> 00:22:08,070 websites. You can also find plans online if you want to build your own. It's blow throwing 246 00:22:08,070 --> 00:22:15,070 a big bucket of noise at the frequencies. The transmissions are weak from the satellite 247 00:22:15,120 --> 00:22:20,960 so you just overpower them. You can also spoof them. You need to generate fake GPS signals 248 00:22:20,960 --> 00:22:25,919 at a higher power than the satellites themselves to override the receiver. This has been demonstrated 249 00:22:25,919 --> 00:22:32,919 with UAVs. This is a group from UT Austin demonstrating taking over a UAV using GPS. 250 00:22:35,929 --> 00:22:41,100 So the attacker on the left is broadcasting a GPS signal that is at a higher power and 251 00:22:41,100 --> 00:22:47,110 first is aligning to the GPS signal that the UAV is really receiving. Those three dots 252 00:22:47,110 --> 00:22:54,039 are match filter trackers on that output that's finding that peak. And then you start to move 253 00:22:54,039 --> 00:22:58,169 your signal off and you take the tracking points with you. So now convincing the UAV 254 00:22:58,169 --> 00:23:01,870 it's in a position it's not really in and its flight control system is trying to correct 255 00:23:01,870 --> 00:23:07,700 that and moving it somewhere else. Here is some video of doing it to the real 256 00:23:07,700 --> 00:23:14,140 UAV. They're going to take control over it and convince it via GPS that it is moving 257 00:23:14,140 --> 00:23:19,570 upward at a certain speed. And the flight control system is going to try to account 258 00:23:19,570 --> 00:23:26,570 for that and move the helicopter down. You can see if the safety pilot doesn't take over 259 00:23:31,309 --> 00:23:35,539 here, you could drive it straight into the ground. 260 00:23:35,539 --> 00:23:39,090 This technique was claimed to be used by the Iranians when they brought down an RQ seven 261 00:23:39,090 --> 00:23:46,090 surveillance. They said they spoofed night the ground. This is widely believed to be 262 00:23:46,710 --> 00:23:51,799 a model. Because the original crashed. But seems unlikely to me. Because the military 263 00:23:51,799 --> 00:23:58,799 systems use encrypted GPS and the military don't rely on GPS for UAVs because there are 264 00:23:59,019 --> 00:24:05,789 jamming and less likely spoofing for military GPS. But we do know for civilian systems it's 265 00:24:05,789 --> 00:24:10,720 easily jammed due to the weak amplitude. Easily spoofed, I mean. 266 00:24:10,720 --> 00:24:17,649 In the civilian realm the GPS is used as a primary sensor. Take the GPS and maybe resolve 267 00:24:17,649 --> 00:24:23,350 the last couple of meters with the laser range finders. It's important to point out those 268 00:24:23,350 --> 00:24:28,309 sensors all have filters on them. The GPS will drag the vehicle slowly from incorrect 269 00:24:28,309 --> 00:24:33,399 trajectory rather than just snapping it off its path. 270 00:24:33,399 --> 00:24:37,340 Here is from 2005. Again without access to the logs it's hard to know exactly what's 271 00:24:37,340 --> 00:24:41,070 happening. But looks like because there's nothing on the road looks like we're seeing 272 00:24:41,070 --> 00:24:48,070 GPS drift off the road that is then being corrected by laser range finders. So that's 273 00:24:49,679 --> 00:24:52,710 typical of what you might see if something's relying on GPS. 274 00:24:52,710 --> 00:24:59,169 Here's another challenge example. Looks like a GPS run that is drifting off the road here. 275 00:24:59,169 --> 00:25:05,240 Who knew that a van could drive over a Josey barrier? 276 00:25:05,240 --> 00:25:08,870 (Laughter) Something maybe for the next DEF CON canon 277 00:25:08,870 --> 00:25:15,870 ball run to keep in mind. Next up laser range finder. Originally a sensor 278 00:25:17,200 --> 00:25:20,870 for industrial automation but then the robot guys got hold of them and were like this is 279 00:25:20,870 --> 00:25:26,649 awesome so we're going on use these outside. Mechanically scanned via rotating mirror. 280 00:25:26,649 --> 00:25:32,049 They measure time of flight. It is an active sensor. Depends on a return signal coming 281 00:25:32,049 --> 00:25:37,539 back. These are primarily used for collision avoidance and map making. I don't know why 282 00:25:37,539 --> 00:25:42,399 I do these bullets because I always forget to advance them. They return a point cloud 283 00:25:42,399 --> 00:25:49,299 of reflected distances within the laser range. So you can do denial on them by actively overpowering 284 00:25:49,299 --> 00:25:56,299 then or by preventing a return signal with dust, smoke, mist. You can also spoof them 285 00:25:56,580 --> 00:26:02,929 by manipulating the surface absorbance of the things they're looking for. So, basically, 286 00:26:02,929 --> 00:26:07,250 manipulating absorbance and reflectivity to give the receiver incorrect information about 287 00:26:07,250 --> 00:26:14,250 what it's looking at. After a 2D sensor. It's highly orientation 288 00:26:19,730 --> 00:26:26,669 dependent. It's often mounted in a push broom configuration. What that means is you are 289 00:26:26,669 --> 00:26:31,419 looking for obstacles nicely but if the ground slopes up it can look like a brick wall. You 290 00:26:31,419 --> 00:26:38,419 can see ‑‑ this is from pizza delivery. On the left is the Light R. You can see that 291 00:26:40,289 --> 00:26:45,210 you get these ranges stopping when you see an obstacle. If you look over there to the, 292 00:26:45,210 --> 00:26:49,320 you know ‑‑ as it sort of turns off to the right where the street slopes up, you 293 00:26:49,320 --> 00:26:55,740 can see that range just drop away to nothing. Kind of looks obstacle like. In addition if 294 00:26:55,740 --> 00:27:02,169 it's shooting down over the low obstacle or like a curb it can miss it entirely. So it 295 00:27:02,169 --> 00:27:09,169 can fall in a ditch if it is orientated right. Active emission sensor. It only return that's 296 00:27:09,889 --> 00:27:16,889 active signal back to the receiver. So no return means that it assumes nothing is there. 297 00:27:16,889 --> 00:27:22,019 Think about that. Over the horizon and out of range returns no signal. So most of the 298 00:27:22,019 --> 00:27:28,360 world returns no data. What that means is thing that's absorb in the laser frequency 299 00:27:28,360 --> 00:27:33,460 look exactly like nothing. Things that are transparent in the laser frequencies also 300 00:27:33,460 --> 00:27:38,200 look like nothing. So if you were to paint an absorbent tunnel on a wall it's just‑like 301 00:27:38,200 --> 00:27:45,200 Wile E. Coyote. The robot would not see that. If you were to make obstacles out of glass, 302 00:27:46,799 --> 00:27:51,299 it sees right through them. Glass is transparent but also reflective. There's a limit to this. 303 00:27:51,299 --> 00:27:57,120 Might miss a bottle but probably going to see a (inaudible). Some also have this multi‑echo 304 00:27:57,120 --> 00:28:02,909 suppression so if you have glass close to them it's designed to ignore them. 305 00:28:02,909 --> 00:28:08,590 It's all about what gets returned to the sensor. Reflective things confuse it. For example 306 00:28:08,590 --> 00:28:15,009 a puddle on the road is very reflective. They can make far away things like obstacles look 307 00:28:15,009 --> 00:28:19,830 near if the angles match up. Right? So this is something that robot actually has to deal 308 00:28:19,830 --> 00:28:24,580 with. Or if there's nothing to be reflected the signal goes out into space and that lack 309 00:28:24,580 --> 00:28:30,630 of return makes it look like a big hole in the road. Not just water in puddles. Even 310 00:28:30,630 --> 00:28:34,850 fresh asphalt cannot give a return and look like a big hole in the road. 311 00:28:34,850 --> 00:28:41,850 One of the challenge vehicles I won't say which one ran into a brand‑new black SUV. 312 00:28:42,210 --> 00:28:46,120 Because it was so shiny and reflective and recently washed that it looked like nothing. 313 00:28:46,120 --> 00:28:51,389 So even a new car can be a problem. That's what the millimeter wave radar is for. You 314 00:28:51,389 --> 00:28:55,460 can use reflective surfaces to make things look like a switch the vehicle won't consider 315 00:28:55,460 --> 00:29:00,429 safe to drive over and will have to take a different route. People out there, bad guys 316 00:29:00,429 --> 00:29:03,429 out there have good reason to try and use these kind of techniques. 317 00:29:03,429 --> 00:29:07,880 This is just something for fun. I found this on my travels on the internet. These are documents 318 00:29:07,880 --> 00:29:14,880 from al Qaeda in the Arabia Peninsula that were captured in Timbuktu. You have to trust 319 00:29:15,309 --> 00:29:22,309 me what it says in Item 2 referring to a GPS jammer. This is a document on how to avoid 320 00:29:23,759 --> 00:29:29,769 UAVs and drone strikes. Item three advises them to place reflective plates on their vehicles 321 00:29:29,769 --> 00:29:35,049 to reflect off the laser designator. To make the missile miss slightly that. Could be the 322 00:29:35,049 --> 00:29:39,049 difference between life and death. Of course for this to work you would need a material 323 00:29:39,049 --> 00:29:45,460 reflective in the laser wavelength. But it's on the list of techniques. 324 00:29:45,460 --> 00:29:50,669 Laser reflectance is also a feature. The road mostly gives a decent return unless it's fresh 325 00:29:50,669 --> 00:29:57,669 asphalt. But the white lines are quite reflective. They look like gaps in the road. They use 326 00:29:58,120 --> 00:30:05,120 this to do road line detection. So a fun consequence of this is that you could make fake road markings 327 00:30:06,090 --> 00:30:10,159 in a way that's invisible to the human like black on black but the robot is going to see 328 00:30:10,159 --> 00:30:17,159 them perfectly. So you could paint some black on black swervy lines, for example. A human 329 00:30:17,330 --> 00:30:21,860 does not know why the robot is swerving all over the road. Or you could even try to be 330 00:30:21,860 --> 00:30:25,549 more kind about this and leave the lines as they are but do the black on black as hidden 331 00:30:25,549 --> 00:30:32,549 messages for the humans back at base when they go and look at the map. 332 00:30:32,850 --> 00:30:36,529 (Laughter) Cameras are used as well. But not as much 333 00:30:36,529 --> 00:30:41,669 as you might think. Vision is really hard. Specialized object detection sometimes, sometimes 334 00:30:41,669 --> 00:30:48,220 stereos used to get a def map but it's noisy. Often what people do is colorize the data 335 00:30:48,220 --> 00:30:55,220 with cameras. You are registering your data to this color information from your camera. 336 00:30:55,730 --> 00:31:00,940 Why isn't this video going? Come on. So this is stop a challenge from '05. This 337 00:31:00,940 --> 00:31:07,500 is Stanley. The way that Stanley drove so fast is it used lasers to get an idea of what 338 00:31:07,500 --> 00:31:12,450 was road and what wasn't road. Then used the camera information to match colors and say 339 00:31:12,450 --> 00:31:16,190 everything that looks like road in front of me I'm going extrapolate that based on color 340 00:31:16,190 --> 00:31:21,679 and that's where I can drive. I'm going compute my path based on that. It was a really nice 341 00:31:21,679 --> 00:31:28,679 technique. Of course, cameras are easily dazzled and subjected to blinding attacks just like 342 00:31:29,090 --> 00:31:35,929 we always talk about with anti‑surveillance stuff. You can also do spoofing with it. Right. 343 00:31:35,929 --> 00:31:42,929 Because like camouflage works for the Mach 1 human eyeball you can mess with the color 344 00:31:43,059 --> 00:31:48,549 reception. If something is saying there's stuff that looks like road everything that 345 00:31:48,549 --> 00:31:54,529 is that color is road just make obstacles out of road color stuff. Repeating patterns 346 00:31:54,529 --> 00:32:00,809 confuse the hell out of stereo cameras. They don't know what matches with what. 347 00:32:00,809 --> 00:32:05,509 Millimeter wave radar. This is used in vehicle applications to get through obstacle avoidance 348 00:32:05,509 --> 00:32:10,169 stuff. You have probably seen it in one form or another because this is the stuff at the 349 00:32:10,169 --> 00:32:16,879 airport that shows off your junk to the TSA. Primarily used for collision avoidance looking 350 00:32:16,879 --> 00:32:21,840 for things that will reflect the radar well like signs and other vehicles. Lower resolution 351 00:32:21,840 --> 00:32:27,059 than the Light R. Produces fuzzy images and lives in a weird world where everything is 352 00:32:27,059 --> 00:32:33,960 a mirror. So lots of stuff is reflective. You can't use it a lot to find decision making. 353 00:32:33,960 --> 00:32:40,409 Like any raider you can confuse it with chaffing. Spitting out thing that's reflect the signal. 354 00:32:40,409 --> 00:32:45,679 Also gets a big return from things like sign. So an overhead sign the robot might be programmed 355 00:32:45,679 --> 00:32:50,000 to ignore that as an obstacle because it's getting a huge return from it. If it happens 356 00:32:50,000 --> 00:32:56,549 to be a dynamic obstacle underneath that it might miss it. 357 00:32:56,549 --> 00:33:03,549 Compass stands for inertia integration unit. This is the primary navigation sensor for 358 00:33:05,649 --> 00:33:11,659 a lot of systems because they can be very, very robust and because they can be very resistance 359 00:33:11,659 --> 00:33:17,269 to any kind of spoofing or attack. You can get everything from hi‑fi dealt models to 360 00:33:17,269 --> 00:33:24,269 these hobbyist ones that are often pretty noisy. A commercial aircraft IMU or commercial 361 00:33:24,850 --> 00:33:31,850 robot IMU like a Boeing 777 IMU has a cumulative error of .1 percent of total distance traveled. 362 00:33:32,340 --> 00:33:38,100 So they're used on a lot of arctic UAVs. You travel 300 kilometers and when you get to 363 00:33:38,100 --> 00:33:43,730 the destination you pop up and get a GPS fix your cumulative distance error is about 300 meters. 364 00:33:43,730 --> 00:33:48,730 That's easy to deal with. Very difficult to interfere with. They're all on board. They're 365 00:33:48,730 --> 00:33:52,570 fully encapsulated. You are just recording what the robot feels. Which is why military 366 00:33:52,570 --> 00:33:59,570 depends on them. However, the compasses very susceptible to magnetic field. 367 00:34:02,360 --> 00:34:08,350 Another part of doing this dead reckoning is wheel odometery. Encoders on the wheels 368 00:34:08,350 --> 00:34:14,250 and giving rotation information that you can integrate out. They're really key component 369 00:34:14,250 --> 00:34:18,150 because they give you good speed information relative to the ground and let you know when 370 00:34:18,150 --> 00:34:22,430 you are stopped for sure. One of the only sensors that can let you know when you are 371 00:34:22,430 --> 00:34:28,740 stopped. Which you might not know how to do especially in a tunnel. So important what 372 00:34:28,740 --> 00:34:34,270 happened when we did our pizza delivery we had to get interrupted at the end because 373 00:34:34,270 --> 00:34:39,030 it took this turn tight and scraped off the wheel encoder. That was bad enough coming 374 00:34:39,030 --> 00:34:44,330 off the bridge we had to rescue the vehicle at this point. There it is hanging off the 375 00:34:44,330 --> 00:34:48,550 side. Very sad. So there are some things you can look at to increase the wheel odometery 376 00:34:48,550 --> 00:34:55,550 uncertainty or just trying to remove it. Odometry drift by changing the wheel diameter. Slippery 377 00:34:57,540 --> 00:35:04,540 surfaces might cause drift. And, of course, when you remove them potentially unpredictable 378 00:35:05,130 --> 00:35:11,030 behavior or stoppage. Now we've talked about all these physical 379 00:35:11,030 --> 00:35:15,320 attacks we can do. Sounding a bit like a James bond car package right. We've got our GPS 380 00:35:15,320 --> 00:35:21,050 jammer to knockout the absolute localization. We've got smoke, dust or vapor ejectors to 381 00:35:21,050 --> 00:35:27,510 confuse the Light R, the Light R lasers perform particularly bad in mist. We've got 382 00:35:27,510 --> 00:35:31,170 chaff dispensers for the millimeter wave radar. Could be sparse enough that a human wouldn't 383 00:35:31,170 --> 00:35:36,510 see anything but the fine metal particles that make the robot stopped suddenly. Glass 384 00:35:36,510 --> 00:35:40,820 Cal drops. If you are James bond you have to have Cal drops. Then, of course, an oil 385 00:35:40,820 --> 00:35:45,780 slick to prevent it from telling when the car's really stopped. Nice if you got a James 386 00:35:45,780 --> 00:35:52,620 bond budget and, of course, an Aston Martin to put them on. But there's another really 387 00:35:52,620 --> 00:35:57,310 important thing to talk about besides all these sensor attacks ‑‑ the map. The 388 00:35:57,310 --> 00:36:02,370 old school mobile robots went into the world pretty much knowing nothing about it and just 389 00:36:02,370 --> 00:36:06,260 using sensors. Lots of emphasis on doing slam and using sensors to build up the map as they 390 00:36:06,260 --> 00:36:12,680 went along. But that aggregated map data is so cheap and ubiquitous there's a huge emphasis 391 00:36:12,680 --> 00:36:19,680 on pre‑acquired map data. Think about for example what Google does as another huge part 392 00:36:20,020 --> 00:36:25,080 of their business, right, mapping. So the map is so comprehensive it's treated like 393 00:36:25,080 --> 00:36:31,640 the ground truth. Here's an example, a video of mapping with sensor data. 394 00:36:31,640 --> 00:36:37,050 This is the kind of map we created for the treasure island pizza delivery. The map is 395 00:36:37,050 --> 00:36:43,310 so comprehensive it's treated as the ground truth. It's really powerful because it reduces 396 00:36:43,310 --> 00:36:47,880 the recognition load on the robot in realtime. The robot can instantly map its sensor data 397 00:36:47,880 --> 00:36:53,880 to static features such as traffic lights, trees, vegetation, speed control and traffic 398 00:36:53,880 --> 00:37:00,880 signs, speed bumps. But reliance on one single thing even a big thing like this cab also 399 00:37:01,520 --> 00:37:07,030 be a weakness. A single point of failure. So there are potentially all kind of things 400 00:37:07,030 --> 00:37:13,810 that we can make use of if they're relying too much on this map. Traffic lights for example. 401 00:37:13,810 --> 00:37:20,810 This is how some vehicles locate traffic lights. 100 percent robustly locating a traffic light 402 00:37:22,110 --> 00:37:25,790 is hard right. They could be anywhere that you can see. You've got to do vision to find 403 00:37:25,790 --> 00:37:30,650 them. And you've got to have 100 percent. You can't just have your robot go around occasionally 404 00:37:30,650 --> 00:37:35,000 blowing through a red light. Because, you know, the robot, you know, it doesn't have 405 00:37:35,000 --> 00:37:40,780 to get ‑‑ it's got to do this. Because you are on the same road as humans. The human 406 00:37:40,780 --> 00:37:45,790 is expecting that if the light's red that guy is going to stop. 407 00:37:45,790 --> 00:37:50,140 But if you've got a map of every single traffic light and it's registered to GPS so from everywhere 408 00:37:50,140 --> 00:37:57,140 you see it you know exactly where to look to see that traffic light, then detecting 409 00:37:58,430 --> 00:38:01,320 the color of the traffic light is trivial. You know where it is. You just have to look 410 00:38:01,320 --> 00:38:08,320 for that blob. But now you've got a potential schism between the human and robot's assumptions. 411 00:38:08,930 --> 00:38:14,570 Because the human assumes the robot can see the light under any conditions. But the robot 412 00:38:14,570 --> 00:38:21,570 assumes the light is where the map says it is. So if it gets moved, shifted around, human 413 00:38:22,710 --> 00:38:28,390 drivers have no problem with that. But the robot isn't going to recognize the new state. 414 00:38:28,390 --> 00:38:32,460 Registration detection is another example. Let's say the robot has some kind of rules 415 00:38:32,460 --> 00:38:38,960 to determine what is vegetation and what kind it is. So you might have some blob of vegetation. 416 00:38:38,960 --> 00:38:45,080 You've got the colorized Light R looking for green stuff. Oar you might have some kind 417 00:38:45,080 --> 00:38:50,290 of transmission classifier. How much of the laser is coming back? The vegetation is not 418 00:38:50,290 --> 00:38:53,600 100 percent reflective. And you know that if it's on the ground it's grass and you can 419 00:38:53,600 --> 00:38:58,960 drive over it. And you've got all your trees mapped out. So you know where those are all 420 00:38:58,960 --> 00:39:03,860 and you treat them as static obstacles. So big deal. Since the last time you drove that 421 00:39:03,860 --> 00:39:09,890 route the trees have grown and the foliage is overhanging the road. And suddenly it's 422 00:39:09,890 --> 00:39:14,030 now spotted by the row about the as dynamic obstacles. Here is this vegetation and the 423 00:39:14,030 --> 00:39:17,880 robot is going crazy and stopping everywhere even though it's light vegetation it could 424 00:39:17,880 --> 00:39:24,880 drive right through. So the meta‑point here is that the rules 425 00:39:25,190 --> 00:39:28,900 for guessing what things are that the human designers have come up with are often brutal. 426 00:39:28,900 --> 00:39:33,530 They represent the best efforts that the designers have been able to do to design acceptable 427 00:39:33,530 --> 00:39:38,390 tests. But when you have this great thing that looks like the truth in the form of a 428 00:39:38,390 --> 00:39:43,210 map, you come to depend it on. And dependence on the map may exacerbate the brutalness of 429 00:39:43,210 --> 00:39:47,640 the rules in the way that opens the door for exploitation. 430 00:39:47,640 --> 00:39:53,880 You got to have constant updates. This video is an example of simulation of what robot 431 00:39:53,880 --> 00:39:58,080 vehicles could do at an intersection if they have completely reliable realtime local information. 432 00:39:58,080 --> 00:40:05,080 There's no need to stop they just go through the intersection. Totally terrifying right? 433 00:40:06,020 --> 00:40:11,330 (Laughter) So you could do that if you have that map 434 00:40:11,330 --> 00:40:14,940 update. But if you have a local map, then you can't do that kind of stuff and you're 435 00:40:14,940 --> 00:40:21,940 vulnerable to unexpected real world features. If you have a remote map, then you are vulnerable 436 00:40:22,880 --> 00:40:27,840 to all those attacks on the network. Right? So you can do denial. You can jam your 4G 437 00:40:27,840 --> 00:40:31,790 map updates and also spoofing. Man in the middle the map dater as it comes through. 438 00:40:31,790 --> 00:40:38,790 All of the techniques we're familiar with from other parts of DEF CON. 439 00:40:39,200 --> 00:40:44,540 So looking at some of the general vulnerabilities here let's talk about the logic structures 440 00:40:44,540 --> 00:40:50,210 and how we might craft and exploit. So we want to maximize the uncertainty facing the 441 00:40:50,210 --> 00:40:54,910 vehicle in order to cause mission failure. Some of the maneuvers that a vehicle needs 442 00:40:54,910 --> 00:40:59,000 to do when it can only do on‑board sensing or more uncertain and, therefore, more fragile 443 00:40:59,000 --> 00:41:04,770 than others because of the geometry. One example is a right turn on right. Upcoming 444 00:41:04,770 --> 00:41:09,870 traffic from the left and the view could be blocked from other vehicles. The same problem 445 00:41:09,870 --> 00:41:15,410 a human driver has. The robot is going to be necessarily more cautious here and this 446 00:41:15,410 --> 00:41:22,110 provides an opportunity to trick it. So we might want to force the robot to require manual 447 00:41:22,110 --> 00:41:26,940 assistance, right. To be unable to continue without supervision. We might want to confuse 448 00:41:26,940 --> 00:41:33,940 or annoy the occupants so they abandon the robot vehicle transportation. Even regularly 449 00:41:34,030 --> 00:41:37,370 dropping the vehicle back into manual mode might do that. 450 00:41:37,370 --> 00:41:41,450 Inconveniencing the other road users, right. If the robot stops and blocks traffic, you've 451 00:41:41,450 --> 00:41:48,450 got robot road rage. So getting back to these fragile maneuvers like the right turn on red, 452 00:41:49,690 --> 00:41:54,680 if you can make it too uncertain so it sits there and blocks traffic or if it ventures 453 00:41:54,680 --> 00:42:00,730 out at the wrong time and gets T‑boned, then that's the vulnerability to be exploited 454 00:42:00,730 --> 00:42:07,730 here. Now if you have physical access to the physical, you can do these kind of physical 455 00:42:08,000 --> 00:42:13,550 attacks on the logic. Kind of like a 21st Century version of slashing the tires. Obviously highly 456 00:42:13,550 --> 00:42:17,910 dependent on the configuration and mission. But if you have the abilities for example 457 00:42:17,910 --> 00:42:23,270 to get near the compass and stick a device only a millimeter away if it has a strong 458 00:42:23,270 --> 00:42:29,890 electronic magnet that's got a 4G (inaudible) you can figure out where it is and mess with 459 00:42:29,890 --> 00:42:36,890 its compass at just the right time. An obvious style of attack is re‑directing 460 00:42:41,030 --> 00:42:45,010 the robot away from where it's suppose to go. Or even trapping it in a spot it can't 461 00:42:45,010 --> 00:42:50,940 get out of. Attack on the collision avoidance and navigation layers forcing it to postpone 462 00:42:50,940 --> 00:42:56,010 its high level tasks. Have obstacle that's move and you can force the robot to stop. 463 00:42:56,010 --> 00:43:00,680 If you can put obstacles around it so it can't get out. Or you can have moving obstacles 464 00:43:00,680 --> 00:43:04,710 that guide the robot off the path somewhere you want to take it. 465 00:43:04,710 --> 00:43:07,930 (Laughter) You could even have obstacles formed of other 466 00:43:07,930 --> 00:43:11,760 robots. Artificial traffic lights is another one. The robot is depending on the map. You 467 00:43:11,760 --> 00:43:16,080 can't put up a fake light but if you can use the real light and modify it so the robot 468 00:43:16,080 --> 00:43:20,390 thinks nights a different state the human would figure it out right away but the robot 469 00:43:20,390 --> 00:43:27,390 has to stop. Another attack is clobbering. This is a term from the cruise missile world. 470 00:43:31,770 --> 00:43:35,630 Make the robot run into something. Like a piece of terrain or something. Subverting 471 00:43:35,630 --> 00:43:41,710 its collision avoidance ultimately to incapacitate the vehicle perhaps. So you might want to 472 00:43:41,710 --> 00:43:46,340 completely crash into something or might want to scrape off sensors. You can do this by 473 00:43:46,340 --> 00:43:50,520 doing subtle deviations from the map, changing things on the map, especially near fragile 474 00:43:50,520 --> 00:43:56,570 maneuvers. Or by changing things post mapping. You can do it by imitating light vegetation 475 00:43:56,570 --> 00:44:00,540 so it thinks it can go through it but it can't. Simulating obstacles at speed. So the robot 476 00:44:00,540 --> 00:44:07,540 has to stop suddenly or swerve. Disguising entrance walls. Like a fake tunnel. Putting 477 00:44:08,920 --> 00:44:13,820 materials within the localization noise so it goes too close to one side or overhanging 478 00:44:13,820 --> 00:44:20,510 piece that scrapes off the top sensors. Or obstacle as I mentioned underneath big 479 00:44:20,510 --> 00:44:26,560 radar reflectors so the robot is normally programmed to ignore. Like big overhead traffic 480 00:44:26,560 --> 00:44:31,460 signs. So now that I've said all of these things 481 00:44:31,460 --> 00:44:35,860 that you could potentially do to mess up a robot, mean and nasty things to driverless 482 00:44:35,860 --> 00:44:42,860 vehicles, I want to reiterate driverless vehicles are cool. Don't do any of these things. 483 00:44:42,940 --> 00:44:47,520 (Laughter) I'm saying this: Don't hassle the Hoff. I 484 00:44:47,520 --> 00:44:54,520 mean, don't ax saw the bot. But instead if you are into autonomous vehicles and getting 485 00:44:54,850 --> 00:44:59,600 involved in the future of transportation why not get involved in the hard challenge of 486 00:44:59,600 --> 00:45:05,530 actually making them work. Screwing them up is easy. Getting them right is the cool part. 487 00:45:05,530 --> 00:45:09,610 So for any students here I'd like to close out by just mentioning some stuff about the 488 00:45:09,610 --> 00:45:16,610 autonomous robot competitions. SAUC‑E and Robobot and Robosub. I want more DEF CON 489 00:45:19,140 --> 00:45:24,650 people involved in these competitions. Because DEF CON people like to push the envelope. 490 00:45:24,650 --> 00:45:29,360 So here's just a quick run through of the tasks that are done and what you guys might 491 00:45:29,360 --> 00:45:34,790 be interested in. SAUC‑E wave point navigation, search for 492 00:45:34,790 --> 00:45:39,830 and identifying secret symbols on the ground and connecting to an area Wi-Fi network and 493 00:45:39,830 --> 00:45:45,660 downloading secret codes. This is DEF CON right here ‑‑ secret codes. Also coming 494 00:45:45,660 --> 00:45:52,660 soon, hopefully, package dropping. Legit excuses to write bombing runs for UAVs. Cool challenges 495 00:45:53,090 --> 00:46:00,090 involve visual map making, registering with GPS. If you are into Panorama stitching or 496 00:46:00,510 --> 00:46:06,610 automatic target ID not a lot of teams are doing this. So there's lots much opportunity 497 00:46:06,610 --> 00:46:12,530 to put together a sophisticated entry. Roboboat is one of the most difficult competitions. 498 00:46:12,530 --> 00:46:18,510 We've got channel navigation, directing con canons on to a target, identifying thermally 499 00:46:18,510 --> 00:46:25,000 hot items on ground stations, disabling water sprays, deploying a rover and retrieving a 500 00:46:25,000 --> 00:46:27,660 package. A team this year had a boat that launched 501 00:46:27,660 --> 00:46:34,660 a quad‑copter to retrieve a package. Capture the flag from another boat. So this is all 502 00:46:34,680 --> 00:46:39,970 DEF CON stuff, right? Camera and light sensor integration. 503 00:46:39,970 --> 00:46:44,650 This year a team had a Light R they couldn't afford one off the shelf so they hacked one 504 00:46:44,650 --> 00:46:51,650 out of a robotic vacuum cleaner and reversed engineered it. That's what we need more. Discrimination 505 00:46:51,800 --> 00:46:56,300 between vegetation and water and detecting when the robot is stuck on things. People 506 00:46:56,300 --> 00:46:59,020 haven't been good at that. That's where we need people with a security mindset to think 507 00:46:59,020 --> 00:47:04,460 about it. Robosub. The big one. Underwater is the poster 508 00:47:04,460 --> 00:47:09,850 child for autonomy. Because you don't have the communications bandwidth to remote control. 509 00:47:09,850 --> 00:47:16,850 So 3D navigation, target recognition, shooting torpedoes, manipulating objects and package 510 00:47:17,040 --> 00:47:22,690 recovery with a sonar pinger. All without GPS. I'm flying through this because I know 511 00:47:22,690 --> 00:47:28,060 I'm running over time. One big thing again I think like we need the 512 00:47:28,060 --> 00:47:31,530 hacker mindset for is all these thing that's people don't think of before they go into 513 00:47:31,530 --> 00:47:36,220 water. Like thermal management the thing I most want people to be involved in is because 514 00:47:36,220 --> 00:47:41,330 I think the rules need to be hacked. They're there to have loopholes found. That's what 515 00:47:41,330 --> 00:47:45,360 people in this room do. This UAV is the scan eagle. It doesn't need 516 00:47:45,360 --> 00:47:50,160 a runway. Planes need runways? Hell no. It's fly night ‑‑ a cable and catch it. This 517 00:47:50,160 --> 00:47:57,160 is what I want to see. Nontraditional vehicles. There's dimension limits but they apply at 518 00:47:59,660 --> 00:48:03,780 the start. Who is to say you can't change the day mentions while doing it and hack things 519 00:48:03,780 --> 00:48:09,890 that way? Swarms of vehicles. Let's get Voltron on this stuff. I think this is the ultimate 520 00:48:09,890 --> 00:48:15,450 hacker sport. It's technologically awesome. It's bloody hard and there are loopholes to 521 00:48:15,450 --> 00:48:18,650 be exploited. I hope people here in the audience who are 522 00:48:18,650 --> 00:48:23,050 students and have eligibility for this will check them out. There's a Big Daddy Roboboat 523 00:48:23,050 --> 00:48:29,850 next year in Singapore. They'll give you a $50,000 boat and $25,000 for sensors if your 524 00:48:29,850 --> 00:48:36,850 team is selected to complete in it. That's it. At the start of this talk. 525 00:48:37,390 --> 00:48:41,330 (Applause) Just before the goons drag me off I want one 526 00:48:41,330 --> 00:48:48,330 more propaganda poster. When I was doing my disclaimer about I didn't want to spread fear, 527 00:48:49,660 --> 00:48:55,490 uncertainty and doubt. Here is a propaganda poster about how we might let the row live 528 00:48:55,490 --> 00:49:00,490 the Austin powers dream. Letting the robot take care of driving while we get down to 529 00:49:00,490 --> 00:49:05,920 business in the back seat. I hope you will get involved in making that come true. 530 00:49:05,920 --> 00:49:07,250 Thank you!