Audi’s Next-Gen Leap In Self-Driving Cars: A Car That Watches You

Cars that drive themselves have been inching onto the market for years now. There are already dozens of cars with “adaptive cruise control”–that is, sit behind the wheel on the highway, and the car will stay in its lane and brake and accelerate to maintain a safe following distance. But that’s a far cry from true autonomy. You still have to keep your hands on the wheel; you still have to be ready to take over at all times. This is about to change. Audi recently announced that in 2018, it’ll be releasing an A8 that completely takes over during traffic jams. For the first time, you’ll be able to take your hands off the wheel, your foot off the pedals, and let your mind wander.

This is a giant shift, portending the next wave in self-driving cars. Gear heads will tell you that after years of Level 2 in the five stages of car autonomy, we’re now finally reaching Level 3. But behind that advancement lies a conceptual leap: Not only must the car monitor the road, it has to monitor you, the driver.

That’s because for a car to take over for a driver, it has to know its own capabilities. To hand that control back over, it has to know the driver’s. Consider what happens when the A8’s traffic-jam feature kicks in. You hit a knot of slow moving traffic–it has to be below 37 mph. Then a button on the center console tells you that self-driving can be engaged. After you push that button, the car takes over. But once the traffic jam comes to an end and the car prepares to hand control back over to you, it has to know whether you’re even ready.

The Conceptual Leap

“We know from studies that it takes about 10 seconds for a driver to understand what’s happening,” explains Alejandro Vukotich, Audi’s vice president of automated driving development. In other words, it takes about 10 seconds to switch from listening to a podcast–or whatever else you’re doing–back over to the task of driving. And to give you that time, the car itself has to constantly be looking 10 seconds into the future–both inside and outside the car.

Monitoring what goes on outside the car isn’t just about situational awareness. It’s about making sure the car doesn’t have blindspots. So instead of just one type of sensor, the A8 is packed with them: LiDAR (which scans for distances using lasers), short and long range radar using ultrasonics, and cameras. Each one has their pluses and minuses–cameras, for example, don’t work well in the rain–but combining them all means the car always has at least a couple optimal sensing options no matter what: if it’s raining, or if there’s a glare, or if the light suddenly changes. All the data from the sensing mechanisms is “fused” with a piece of hardware tucked under the driver’s seat, and algorithms constantly monitor whether there’s enough sensing data to detect any edge-cases in road conditions.

Other cars such as the Tesla Model S already use ultrasonics and cameras; the A8 will be the first to add LiDAR into the mix. But the conceptual leap that the A8 will introduce is a system for monitoring the driver’s affect. An inward-facing camera detects the driver’s head tilt and whether her eyes are open. “We make our own estimation of what you’re looking at,” explains Vukotich. If you’re asleep or distracted so much that it’ll take more than 10 seconds to rouse you, then the system will start alerting you first–then bring the car to a complete stop if you don’t respond. (This being a traffic jam, the idea is that the conditions will let you stop in your lane with the hazards on.) This isn’t facial recognition–the teeny tiny single camera is instead analyzing your posture to understand just how alert you are.

We’ve known for decades that truly smart autonomous systems have to sense the state of their users. Sometime in the 1980s, researchers at NASA Ames and other places started cottoning to the idea that auto-pilot systems in airplanes needed to behave a bit like a horse would: If you ride a horse, the horse senses whether you’re on the reins or not. If it senses that you’re not paying attention, it’ll take over and used its own instincts to guide itself. The key is that the horse has senses–whether it’s the feeling of the reins or the spurs, the horse can sense how much control the rider has. The so-called H-metaphor has become foundational in human-machine interaction. (As I’ve reported before, the translation of the H-metaphor into rules for autonomous driving systems actually belongs to an Audi researcher and former NASA staffer in California.)

Drawing The Battle Lines

But the complexity of creating a car that’s smart enough to understand what its driver is doing has led to a pronounced split within the car industry. Volvo and Ford have announced that they’re skipping out on trying to make cars that can pass control back and forth between car and driver. Instead, they’re aiming to launch cars sometime after 2020 that won’t require human input at all–and won’t even have steering wheels. They’ll instead drive themselves in all road conditions, and execute all the moves that entails. The idea is that the period in which we’ll be passing control back and forth with our cars will be so brief that they’d rather make a bet on what comes after that. Meanwhile, BMW, Mercedes, and GM are all making a different bet. They’re trying to own the pathway to autonomy, on the hopes that they’ll be able to make more luxurious cars–with more expensive features worth spending on, such as a traffic-jam assistant. Those automakers are creating Level 3 systems along the lines of the A8.

At its core, Audi and the rest pursuing Level 3 automation are making a bet on their own skills at human-machine interaction–and how confident they are in solving a problem as old as HMI itself. Since the dawn of aviation, it’s been a truth that about 90% of all accidents in airplanes are caused by mode confusion–when the pilot doesn’t know whether the auto-pilot is on or off, or doesn’t know if the plane’s landing gear is up or down. Solving the problem for cars means having a ton of redundancy not just in the car’s abilities, but in alerting the driver as to what’s going on. The traffic-jam feature in the 2018 A8 will thus be a pioneer in teaching us, the drivers, how to detect what a car can do and how much it knows.