Having fun teaching cars to drive themselves
Thoughts on the Udacity Self-Driving Car Engineer Nanodegree
After two years, thirteen projects, and countless hours, I’ve finished the Udacity Self-Driving Car Engineer Nanodegree! It’s been a lot of fun, and I’ve learned an enormous amount about the underlying software systems that enable autonomous vehicles to navigate the world. Here are some of my biggest take-aways from the program as a whole (aside from the fact that it’s hard to complete this degree quickly while having a child, buying a house, and working full time).
First, autonomous vehicles are a major systems-engineering challenge. They combine advanced hardware, complex software, and real-world motion (often at high speed) and need to react to events on time scales of less than a second. Getting all the pieces of the puzzle to work together successfully is a huge task and is right at the heart of AV engineering.
To work successfully, AVs need to do many different tasks simultaneously. They need to figure out where they are and what direction they’re heading, with extremely high precision. They do this with high-resolution base maps of their environment and (noisy) data from various sensors, in a process known as localization. They have to identify and track other moving objects (vehicles, bikes, pedestrians) in their vicinity, using the same data and algorithms like Kalman filters, and/or using computer vision techniques for object detection.
AVs also have to plan and continuously update a path that gets to their destination while avoiding collisions, a process known as path planning. This can also include identifying lane markers on the road or using advanced computer vision to identify open areas of road surface for maneuvering. And they have to actuate the physical steering, accelerator and brakes to follow that path in the smoothest way possible constrained by the laws of physics, using a variety of techniques.
To complicate matters, there are multiple approaches to most of these tasks. The different possible algorithms perform better or worse in different contexts (such as parking lot navigation vs. highway driving) and have different computational costs. They also have different requirements for back-end data (such as updated, high-resolution base maps and vast amounts of on-road sensor test data). Choosing among them, or integrating multiple approaches simultaneously, is a daunting task.
There are two fundamental areas of rapid innovation in AV perception, lidar hardware and computer vision/deep learning software.
Second, the single hardest and most important part of the entire system is perception. As relayed by David Silver in this excellent overview post, Sebastian Thrun likes to say that perception is “80% of the challenge”. There are two fundamental areas of rapid innovation in AV perception, lidar hardware and computer vision/deep learning software. On the hardware side, a growing number of startups and established companies are developing brilliant new lidar devices that will reduce costs, enhance spatial resolution and add other important features that improve perception. Unfortunately, we weren’t able to touch on lidar hardware in this class (we were simply given simulated lidar data for our algorithms to ingest) but I’ve written about it elsewhere.
On the computer vision/deep learning side, the cutting edge of automotive perception innovation seems to be the use of semantic segmentation. (I say “seems” because new techniques are being introduced so quickly it’s a little hard to keep track.) While this is computationally expensive, it simultaneously produces valuable information about nearby objects and open road space, at the level of individual pixels in the video camera image.
I’ve written previously about the competition in the AV industry between lidar-dependent perception (used by most companies) and lidar-eschewing perception (used mostly by Tesla). I think it’s still an open question about which approach will win out, so stay tuned.
Making AVs work at scale will require just as much thinking about how people work as about how robotic cars do.
Third, for all the technical success of autonomous vehicle engineering, it’s clear that the on-road safety performance is going to depend on a lot of human and operational factors that are beyond the control of systems engineers. The tragic 2018 crash of a test Uber autonomous vehicle that killed a pedestrian in Tempe, AZ illustrated how the process of on-road testing can fail for reasons of operator error and poorly designed test protocols, rather than bad system integration or algorithms.
Uber in particular has made a lot of progress since then, but the core lesson remains that the technical wizardry of making an AV work in most situations must be accompanied by a systems-level perspective of overall operational safety and human behavior. Making AVs work at scale will require just as much thinking about how people work as about how robotic cars do. I’m glad that Udacity included some elements of this in the nanodegree program, although I’d like to see more.
I also can’t help noting that the issue of human behavior being at the core of overall AV safety is completely different from some of the most headline-grabbing discussion about AVs in the media. The “trolley problem” — which has gotten so much attention — is little more than a joke to people actually building AVs. It also reflects a profoundly flawed approach to thinking about AV ethics. Fixating on that kind of simple but completely unrealistic scenario while ignoring much more important policy issues (with some exceptions) is unfortunate, and reflects a lot of ignorance about the real issues with AV technology.
Finally, I have to give enormous credit to the team at Udacity, particularly David Silver, Ryan Keenan, and founder Sebastian Thrun, for the mere existence of this course. It’s an extraordinary achievement and it has had a huge impact on developing talent in the field of robotics and autonomous vehicles. Their enthusiasm and passion are infectious, and I for one really enjoyed the videos of them during the course, corny jokes and all!
Udacity is in the middle of turbulent times right now, but I am immensely grateful to it for making this kind of education possible, and teaching me so much about such a fascinating area of technology. If you’re at all interested in autonomous vehicles — and you should be — you should definitely check it out.
