|
Topic Name: Systems for editing or altering photographs using images on the Web
Category: Computer Graphics
Research persons: Jean-François Lalonde, Derek Hoeim, Alexei A. Efros, Carsten Rother, John Winn and Antonio Criminisi.
Location: Carnegie Mellon University,Pittsburgh, PA 15213, USA,Office: 4207 Newell-Simon Hall, United States
Details
To make the resulting image appear as realistic as possible, the system
analyzes the original photo to estimate the camera angle and lighting
conditions, and then looks in the clip art library for an object — a car, for
instance — that matches those criteria. The user need only identify the horizon
in the original photo to orient the system. Using previously developed Carnegie
Mellon technology for analyzing the geometric context of a photo, the system can
then place the object within the scene, adjusting its size as necessary to put
it in proportion to other objects of equal distance from the camera.
"Matching an object with the original photo and placing that object within
the 3-D landscape of the photo is a complex problem," said Lalonde, who led
development of the system. "But with our approach, and a lot of clip art data,
we can hide the complexity from the user and make the process simple and
intuitive."
The other system, called
Scene Completion
, was developed by graduate student James Hays, another member of Efros'
research team. It draws upon millions of photos from the Flickr Web site to fill
in holes in photos. Some of the holes might be from damage to a physical
photograph, but more often they are created when an editor cuts out part of an
image to eliminate an unsightly truck from a picturesque street scene, or
removing a passerby from a group shot of friends. Photo editors often try to
fill in those holes with sections derived elsewhere in the same image, but Efros
said that a better match can often be found in a different photo.
The system looks for image segments that match the colors and textures that
surround the hole on the original photo. It also looks for image segments that
make sense contextually — in other words, it wouldn't put an elephant in a
suburban backyard or a boat in a desert.
In the case of well-photographed cities or popular tourist attractions, Efros
said, the system might get lucky and find a photo of the same scene on the Web.
In other cases, it might offer a number of possible images that could fill in
the hole. A retaining wall edited out of one photo, for instance, might be
replaced by the image of a building, a grassy slope or a rock outcropping. The
system typically gives the user 20 different choices for filling in the hole.
The success of this approach depends on the number of photos available to the
system, Hays said. "We saw a dramatic improvement when we moved from a database
of 10,000 images to two million images," he noted. "And that is just a tiny
fraction of the hundreds of millions of images already available on sites like
Picasa and Flickr. We have tons of photos from which to choose."
Background-
We present a system for inserting new objects into existing photographs by
querying a vast image-based object library, precomputed using a publicly
available Internet object database. The central goal is to shield the user from
all of the arduous tasks typically involved in image compositing. The user is
only asked to do two simple things: 1) pick a 3D location in the scene to place
a new object; 2) select an object to insert using a hierarchical menu. We pose
the problem of object insertion as a data-driven, 3D-based, context-sensitive
object retrieval task. Instead of trying to manipulate the object to change its
orientation, color distribution, etc. to fit the new image, we simply retrieve
an object of a specified class that has all the required properties (camera
pose, lighting, resolution, etc) from our large object library. We present new
automatic algorithms for improving object segmentation and blending, estimating
true 3D object size and orientation, and estimating scene lighting conditions.
We also present an intuitive user interface that makes object insertion fast and
simple even for the artistically challenged.
About Researchers-
Alexei (Alyosha) Efros
Assistant Professor
The Robotics Institute
and Computer Science Department
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213, USA
Office: 4207 Newell-Simon Hall
Phone: +1 (412) 268-1234
Email: efros@cs.cmu.edu
& Jean-François Lalonde,
Derek Hoeim, ,
Carsten Rother,
John Winn and
Antonio Criminisi.
Funding
This research is supported by:
|
