You Can Create 3D Models Out of Simple 2D Images with This Software

Image from the research paper [fair use], “3D Object Manipulation in a Single Photograph using Stock 3D Models” (http://www.cs.cmu.edu/~om3d/)

Could a 2D still image reveal more information than just the static image captured by a camera as its shutter closes? Theoretically, the answer is a “no” but a trick could be done to transform a simple 2D photo into a 3D image, to see the parts that are not facing the camera. There’s a free software available for converting parts of 2D images into 3D ones.

This software is the creation of a team of researchers at the Carnegie Mellon University and the University of California – Berkeley led by robotics researcher Yasser Sheikh. It is a software that has the capability of extrapolating 3D models of objects in images, to be formally presented at SIGGRAPH 2014 in Vancouver. SIGGRAPH is an annual international conference and exhibition on computer graphics and interactive techniques. Other members of the team are Alexei Efros of University of California-Berkeley and Natasha Kholgade and Tomas Simon of Carnegie Mellon University.

From 2D to 3D

There’s an extensive variety of objects that can be extrapolated. These could be various household items including furniture, cookware, pieces of clothing, and appliances. Considering the array of stock 3D models available online, the software can be expected to work with different kinds everyday items, even including popular smartphones, buildings, and toys. In an interview with CNet, Robotics Institute PhD student and study lead author Natasha Kholgade said that they have “created an environment that gives you that same freedom when editing a photo.” With their software, it will be possible to “reveal” views of an object captured in a 2D photograph. Their experimental software enables 3D object manipulation in 2D images.

Screenshot from the demo video of Natasha Kholgade, one of the researchers

Kholgade has a couple of demo videos on her YouTube channel. Check them out to see how the software works in action. Of note, the videos show stop-motion animations of the different extrapolations that can be possibly done with just one still image. With this software,

The software has no specific name yet but it is generally referred to as an object manipulation 3D software.

How Does It Work?

Three important factors will have to be taken into account for the software to do a proper 3D manipulation. These are the 2D object (in an image, should be carefully or precisely selected), the suitable 3D model, and lighting configuration. The software comes with a selection tool to pick the object (to be manipulated) from a 2D image. It has a facility to obtain 3D models that can match the shape, size, and form of the object selected. It also has the function to configure the light source to create a more believable 3D rendering.

Image from the research paper [fair use], “3D Object Manipulation in a Single Photograph using Stock 3D Models” (http://www.cs.cmu.edu/~om3d/)

The software exploits the relatively great availability of 3D models online. Using these models, it becomes possible to extrapolate the different forms or appearances of an object as viewed from different angles. The researchers have developed a technique to partially automate the alignment of the selected object and the 3D models available. For example, when manipulating a chair on an image, the chair has to be selected from the image and matched to an available 3D model. Obviously, you can’t expect the selected object to perfectly match the shape and form of the available 3D model. This is where the semi-automatic alignment function of the software comes into play.

Possible Applications

Perhaps one of the most important applications of this software is animation. With it, 3D animators will no longer have to create renders of the objects they want to animate. They can simply animate something through this software by creating a 2D base image and obtaining an appropriate 3D model to go with it. This will cut animation time significantly, which will also reduce costs and help complete production sooner. Film outfits may also adopt the software to come up with a system wherein they already have their in-house compendium of 3D models to match new characters or animation elements so they wouldn’t have to repeatedly do animations, especially for supporting characters or elements.

Another application of this software is the generation of 3D print models for 3D printing. Since it can create 3D models out of selected 2D objects and available free 3D models online, tools for scanning items to produce 3D models may no longer be needed, especially for projects that don’t need a high level of precision.

Screenshot from the demo video of Natasha Kholgade, one of the researchers

The software is far from perfect so you can’t expect flawless 3D rendering. There will be blurs, distortions, or clear traces of image manipulation as what the demo videos show. Nevertheless, there are plans on further developing the software. For now, what’s important is that this new system proves to be a very promising tool for those in the fields of graphic design and animation. For more details on this exciting graphics manipulation software, visit its official website to find a copy of the paper prepared by the research team, which includes illustrations and video presentations. The software is free to download and use. Likewise, the source code can be downloaded for free, from this link.