Light field videography

Multi Camera Array, Stanford University

Imagine a camera that can take photographs in such a way that when they are displayed, they appear as if you are looking through a window at the actual scene rather than at a flat image. You not only perceive depth, but the perspective actually shifts as you move your head to and fro, and side to side. If an object inside the image blocks your view, you can even look around it, as far as the border of the image allows. This is light field photography and it exists today. In fact, the Lytro, a small commercially available light field camera allows you to refocus a photograph after you've taken it, and even shift perspective within a narrow viewing angle.

As the name suggests, light field cameras do this by capturing not just the rays of light as seen from one point (the focal point of a traditional camera), but as seen by all points within a particular capture area, thereby capturing the entire light field of the scene. The captured information contains everything necessary to recreate the scene as seen by any point within that area. If we were to create a window-sized light field camera (a task only limited by cost and not technology), we could theoretically recreate a display that is indistinguishable from a real window. I say 'theoretically' because while we have the technology to record a light field, we do not yet have the technology to replay one. Our display screens are still flat. Our 3D technology only allows a single perspective, achieved only by tricking our eyes, not by creating an actual light field.

In order to create a light field, pixels that flood every direction with light rays of the same color and intensity are insufficient. We need a display made of elements that can emit different types of photons (or light particles) in different directions. We can theoretically do this, but we still do not have the technology to build such light emitting elements small enough to fit a screen, and in sufficiently large numbers. But it is not too much of a stretch to expect the technology to emerge within a decade or so.

While light field photography will be an interesting novelty for the most part, the real applications lie in light field videography, which, just as with normal videography, is not much more than a series of rapidly taken light field images displayed equally rapidly to form a moving picture. A teleconference call that uses a light field video display of sufficient resolution and frame rate would be utterly indistinguishable from the real scene right up until a participant reaches out to shake the hand of someone on the other side (and bangs his fingers against the screen). Commuting then, is unlikely to be needed for anything except those jobs that require you to physically and directly interact with things. Replace a window in your home with a light field screen and you could enjoy any location in the world as your garden.

The processing and data capacity requirements will be staggering. Roughly speaking, a single pixel will require the same amount of processing power and bandwidth an entire frame currently requires. But again, it is not unreasonable to expect such resources to be available within a decade, or at most, two.