TechTheDay

Photographers who couldn’t avoid shooting a scene or an object through a glass window usually encounter the problem of reflections of themselves on the photo they take. This usually happens when shooting at aquariums, from skyscrapers, while riding a bus or shuttle, or when taking photos of clouds or landscapes through the window of a plane. The good news now is that it is already possible to remove these reflections as part of post-production routines. Photographers as well as those who casually snap photos through glass windows can easily remove their reflections on the glass in the same way the mythical vampire reflection works.

The Vampire Effect Technology

To be presented by MIT researchers at the Computer Vision and Pattern Recognition conference in June, this reflection-removing technology is a new algorithm designed to exploit the presence of multiple reflections in images. Basically, what it does is to exploit the presence of multiple reflections in individual images to be able to distinguish a reflection from a transmission.

The researchers found that most windows are double-paned to ensure heat isolation during winters. These double-paned windows produce at least two reflections that make it easier for machines to make out the difference between a normal image and a reflected image. A reflected image would appear as an imperfectly aligned set of two similar images, one of which is coming from the inner pane while the other is produced by the outer pane.

Of course not every window is double-paned but according to YiChang Shih, a PhD in computer science at MIT and one of the researchers involved in the study, the same effect can be exhibited by single-paned but thick-glassed windows. Shih says that the inner and outer sides of a thick glass can give out their separate reflections.

Identifying a reflection and separating it from the rest of the objects that appear in an image is a very difficult process. However, thanks to the second reflection produced by double-paned or thick-glassed windows, the mathematical problem of extracting reflections becomes easier.

In standard computer vision technology, the approach used to capture the difference between a reflection and a regular image is through the characterization of blocks of pixels based on the chief direction of color change and the rate of observable change. However, this approach, according to Shih, does not appear adequately effective.

Shih and the rest of the team that created the “Vampire Effect” algorithm also used, in addition to the identification of double reflections, a technique co-developed by Daniel Zoran and Yair Weiss of the Hebrew University of Jerusalem. This technique involves an algorithm that separates images into 8×8 blocks of pixels and calculates the correlation between each pixel and each of the others. With this segregation, it becomes easier for a computer to pinpoint which ones stand out and are deemed to be different from the non-reflected objects being captured in an image. Around 50,000 images were used to test the effectivity of this approach and the researchers found that it performs well in distinguishing reflections from regular images.

Applications

Obviously, this technology can be useful for photography. It could someday be incorporated in the software installed in digital SLRs or mirrorless cameras used in pro and semi-pro photography. It would also be a great new feature for the cameras used by photojournalists.

Yoav Schechner, an electrical engineering professor at Israel’s Technion, noted how previous attempts at eliminating reflections from photos had limited success, citing the inadequacy of using a single shot and the lack of the right technology to automatically detect reflections. Schechner praised this new method, pointing out how it “does a good job on several fronts.” He adds that the ideas behind this “Vampire Effect” technology can be adopted in routine photography once the algorithm is made more robust. It may also be integrated with the processing functions of photo editing software like Adobe Photoshop.

Additionally, Schechner thinks that this algorithm can be beneficial for robot vision like Project Tango. It can help robots or drones in better handling confusing glass reflections. Well, this will be mostly a military application like in the case of drones peering through window glasses or surveillance cameras situated inside buildings but are also observing events outside.

It’s not clear, though, how the algorithm is able to restore or recreate the parts of the image that will be removed upon the deletion of the reflection. Obviously, images are flat and the parts covered by the reflection would have not been properly captured by the camera. We can’t wait to learn more about this interesting new photography tech.