By Alex Winter, CEO & Co-Founder
“Chloe, zoom in to see the license plate…” / “Let’s enhance…”
It’s a classic scene in 24. Something bad happens under the scope of a lo-fi analog video cam that Chloe’s hacked into. Or a major terrorist attack takes place right under a satellite positioned above that specific area—it’s always amazing how fast Chloe can reposition satellites, but that will be the topic of a future blog post. The bad guys are getting away, and the good guys (Jack Bauer of course) can’t let them go unpunished. He needs the license plate number of that fleeing car. So Chloe zooms, zooms, and zooms more, and the license plate appears. Crystal clear. Well, this is just in the movies. At least for now.
[Update]: Thanks to Vinnie Quinn for pointing out to us this amazing collection of “enhance…” TV & movie quotes that perfectly sums up what we’re talking about!
To get a better sense of what we’re talking about here, let’s see how image sensors work, starting with the human eye. Our retina is a little flat disk in the back of our eyes that has a limited, yet very large number of light and color sensitive cells (6 to 7 million!) called cones and rods. A lens in our pupil concentrates light rays into this disk, causing cells to react and create an array of values—or a “natural” image—that is then encoded and transferred to our brain for processing or storing.
A camera is built in the same way: a lens focuses light rays on an array of sensors—CMOS or CCD. Each CMOS or CCD cell creates a pixel value, which together are assembled into a digital image and transferred to your Instagram account for storage, or to a computer vision algorithm for processing like we do at Placemeter.
Cameras have a fixed field of view and a matrix of elements that spans that same field of view, meaning they have a built-in ceiling for accuracy. You can zoom in easily if what you see on your screen is not the full resolution image; when you get to full physical resolution, you can still go a little further and correct small noise effects or blurs, using block-toeplitz matrix deconvolution for example (the first thing I studied on the way to my computer vision PhD). This is what you can get:
An example of image deconvolution
But that will NOT get you a clear view of a license plate that is just a dot in the original full resolution image.
All in all, an image sensor, e.g. a camera, is only as good as its individual components: it takes a really good lens and a really huge array of CMOS or CCD to build high definition images. The largest existing one has 3.2 Gigapixels, but more common ones are around 60 Megapixels.
Of course, closer is better, but lower satellites have to fly faster to stay in orbit, so they are not really useable for static surveillance.
The finest satellites today have a resolution of 30 cm (1 pixel=30 cm). Now, a license plate in the US is 12”x6”, or 30.5×15.2 cm, which is 1×0.5 pixels at 30cm resolution. This is what it looks like:
NY license plate
What it looks like at 30 cm resolution
Not quite at the level we need to see license plates, but close—even if it does not look like it.
As camera technology progresses, we’ll eventually see low-orbit satellites that have the proper resolution to do such things, but it’s some years down the road.
An interesting development that could soon make its way into satellite technology—and improve their resolution—is the concept of “light field cameras.” At its most basic level, a light field camera contains a bunch of microlenses that act like one huge lens. Images taken from light field cameras capture a much larger field of view, with much greater detail in an image, significantly improving resolution along the way.
One of the effects of light field camera technology is the ability to focus in on every different part of an image with much greater clarity than standard cameras. Another is the ability to 3D model objects captured in the 2D dimension of an image. True 24-level stuff.
You can already buy a consumer version of this in the Lytro camera. Who knows when light field cameras will be introduced to satellite technology. One things for certain: it’ll definitely be too late for Jack Bauer.