Kalman filters and fake news
What autonomous vehicles can teach us about our social media feed
I just finished an autonomous vehicle programming project called “Catch the Run-away Car” (part of the Udacity Self-Driving Car Nano-degree). The goal of the project is to use (simulated) data from radar and lidar sensors to (a) figure out where a run-away car is, and (b) use that to steer an autonomous “pursuit” car to catch it.
The core of the project is an algorithm known as a “Kalman filter”. It’s complicated and involves a lot of math that I’ll skip here. But the basis of it is a really important idea about data, measurements and uncertainty. The sensors on autonomous cars — radar and lidar — are very impressive; in the real world, they cost a lot and they give you data about objects around the vehicle many times a second. That seems ideal for trying to catch something: just head towards where the sensors say it is. What could be simpler?
The point is that the sensors don’t ever tell you exactly the truth — they tell you something that’s close, but probably a little bit wrong.
Unfortunately, there’s a catch. The data the sensors give you isn’t perfect. Just like everything in the real world, it’s a little inaccurate, or to use more technical language, it’s “noisy”. Does the lidar say the target car is 12.3 meters away? Maybe it’s actually 12.5 or 12.1. Does the radar say it’s moving away at a speed of 2.1 meters/second? Maybe it’s actually 2.2, or 2.0. The point is that the sensors don’t ever tell you exactly the truth — they tell you something that’s close, but probably a little bit wrong.
The job of the Kalman filter is to take this mostly-right, slightly-wrong “noisy” information and use it to make the best guess possible about where the target car really is. To do this, the filter algorithm blends the sensor data with what it already knows from the past, based on previous sensor data.
“Confuse and disorient” sounds a lot like something many people experience these days, particularly when it comes to news.
Here’s the key part: it doesn’t just throw away the past information and blindly trust the latest measurement. It balances how certain (or uncertain) it already is about the target car’s location with how certain (or uncertain) the new data is. If it didn’t do this, then the target car would appear to be randomly jumping around from spot to spot, jittering chaotically — which would just confuse and disorient the pursuit car.
“Confuse and disorient” sounds a lot like something many people experience these days, particularly when it comes to news. Our Facebook and Twitter feeds are full of news stories that show up at a furious pace. Keeping track of them can make even the most relaxed reader feel anxious, confused, or worse. And the problem gets even bigger when you factor in fake news — some of these news stories are so inaccurate they’re completely false.
Modern information overload shares a lot of similarities with the problem that Kalman filters were designed to solve. The information we commonly get on social media — stories about news events, opinions on various political issues — is a lot like a series of “noisy” measurements about the world. The stories may be mostly on-target, but they’re never a perfectly accurate view of what’s going on. Unfortunately, it’s human nature to use the simple strategy I described above to handle all this information: treat the latest “measurement” (news story) you read as the whole truth, and “steer” towards that in your effort to understand what’s happening in the world. And just like the situation with autonomous vehicles, this produces confusion and disorientation, as your sense of what’s happening in the world constantly and unexpectedly shifts.
What we need as good consumers of modern information flows is a mental Kalman filter for how we read the news.
The problems with this are clear enough: people can get frustrated, overwhelmed, anxious or infuriated if they try to closely follow the news. Many people just give up and choose to disengage completely. Others tend to harden their views, rejecting anything that disagrees even slightly with what they believe they know and un-critically accepting news that agrees with it (a.k.a. confirmation bias). It’s harder and harder to find a productive middle ground, where news can shape our view of the world but not overwhelm it, and we can build an accurate-enough understanding of reality.
What we need as good consumers of modern information flows is a mental Kalman filter for how we read the news. We need to develop good habits of being able to integrate new information — which is inevitably “noisy” — into our existing knowledge, which we must also accept is imperfect and has its own uncertainties. If we could figure out how to do this, and we used these habits regularly, we could make a lot of progress in overcoming many of profound problems we see today of cognitive overload and political polarization.
What’s important is to learn from the Kalman filter metaphor that “noise” and uncertainty are fundamental to all information, and the only way to develop a robust and reliable view of the truth is to consistently evaluate and balance those uncertainties.
Of course, the actual Kalman filter algorithm is just a metaphor in this situation. I’m not talking about implementing an actual one in social media news feeds, although I’m sure many people have considered something like that. What’s important is to learn from the Kalman filter metaphor that “noise” and uncertainty are fundamental to all information, and the only way to develop a robust and reliable view of the truth is to consistently evaluate and balance those uncertainties.
Doing this would challenge a lot of easy habits we’ve all developed, and it would require more work from us when we consume news. Obviously, it’s much easier to simply accept or reject a news story as entirely true or false, because making nuanced judgements about how to nudge our understanding of a situation every time we read a news story is hard. But even if it takes a little more work, it’s definitely worth it.
And what about the run-away car in my project? With a working Kalman filter, the pursuit car has no problem interpreting the noisy, inaccurate sensor measurements to figure out exactly where its target is, and head in the right direction to catch it. But without one, it hasn’t got a chance.
