Trade.Information

Topic Name: Stanford Electronics Researchers have Developed Multi-Aperture Image Sensor 3-D Camera

Category: Optoelectronics

Research persons: Professor Abbas El Gamal

Location: Stanford University, United States

Details

The camera you own has one main lens and produces a flat, two-dimensional photograph, whether you hold it in your hand or view it on your computer screen. On the other hand, a camera with two lenses (or two cameras placed apart from each other) can take more interesting 3-D photos.

But what if your digital camera saw the world through thousands of tiny lenses, each a miniature camera unto itself" You’d get a 2-D photo, but you’d also get something potentially more valuable: an electronic “depth map” containing the distance from the camera to every object in the picture, a kind of super 3-D.

Stanford University electronics researchers, lead by electrical engineering Professor Abbas El Gamal, are developing such a camera, built around their “multi-aperture image sensor.” They’ve shrunk the pixels on the sensor to 0.7 microns, several times smaller than pixels in standard digital cameras. They’ve grouped the pixels in arrays of 256 pixels each, and they’re preparing to place a tiny lens atop each array.

“It’s like having a lot of cameras on a single chip,” said Keith Fife, a graduate student working with El Gamal and another electrical engineering professor, H.-S. Philip Wong. In fact, if their prototype 3-megapixel chip had all its micro lenses in place, they would add up to 12,616 “cameras.”

Point such a camera at someone’s face, and it would, in addition to taking a photo, precisely record the distances to the subject’s eyes, nose, ears, chin, etc. One obvious potential use of the technology: facial recognition for security purposes.

But there are a number of other possibilities for a depth-information camera: biological imaging, 3-D printing, creation of 3-D objects or people to inhabit virtual worlds, or 3-D modeling of buildings.

The technology is expected to produce a photo in which almost everything, near or far, is in focus. But it would be possible to selectively defocus parts of the photo after the fact, using editing software on a computer

Knowing the exact distance to an object might give robots better spatial vision than humans and allow them to perform delicate tasks now beyond their abilities. “People are coming up with many things they might do with this,” Fife said. The three researchers published a paper on their work in the February edition of the IEEE ISSCC Digest of Technical Papers.

Their multi-aperture camera would look and feel like an ordinary camera, or even a smaller cell phone camera. The cell phone aspect is important, Fife said, given that “the majority of the cameras in the world are now on phones.”

Here’s how it works:

The main lens (also known as the objective lens) of an ordinary digital camera focuses its image directly on the camera’s image sensor, which records the photo. The objective lens of the multi-aperture camera, on the other hand, focuses its image about 40 microns (a micron is a millionth of a meter) above the image sensor arrays. As a result, any point in the photo is captured by at least four of the chip’s mini-cameras, producing overlapping views, each from a slightly different perspective, just as the left eye of a human sees things differently than the right eye.

The outcome is a detailed depth map, invisible in the photograph itself but electronically stored along with it. It’s a virtual model of the scene, ready for manipulation by computation. “You can choose to do things with that image that you weren’t able to do with the regular 2-D image,” Fife said. “You can say, ‘I want to see only the objects at this distance,’ and suddenly they’ll appear for you. And you can wipe away everything else.”

Or the sensor could be deployed naked, with no objective lens at all. By placing the sensor very close to an object, each micro lens would take its own photo without the need for an objective lens. It has been suggested that a very small probe could be placed against the brain of a laboratory mouse, for example, to detect the location of neural activity.

Other researchers are headed toward similar depth-map goals from different approaches. Some use intelligent software to inspect ordinary 2-D photos for the edges, shadows or focus differences that might infer the distances of objects. Others have tried cameras with multiple lenses, or prisms mounted in front of a single camera lens. One approach employs lasers; another attempts to stitch together photos taken from different angles, while yet another involves video shot from a moving camera.

But El Gamal, Fife and Wong believe their multi-aperture sensor has some key advantages. It’s small and doesn’t require lasers, bulky camera gear, multiple photos or complex calibration. And it has excellent color quality. Each of the 256 pixels in a specific array detects the same color. In an ordinary digital camera, red pixels may be arranged next to green pixels, leading to undesirable “crosstalk” between the pixels that degrade color.

The sensor also can take advantage of smaller pixels in a way that an ordinary digital camera cannot, El Gamal said, because camera lenses are nearing the optical limit of the smallest spot they can resolve. Using a pixel smaller than that spot will not produce a better photo. But with the multi-aperture sensor, smaller pixels produce even more depth information, he said.

The technology also may aid the quest for the huge photos possible with a gigapixel camera—that’s 140 times as many pixels as today’s typical 7-megapixel cameras. The first benefit of the Stanford technology is straightforward: Smaller pixels mean more pixels can be crowded onto the chip.

The second benefit involves chip architecture. With a billion pixels on one chip, some of them are sure to go bad, leaving dead spots, El Gamal said. But the overlapping views provided by the multi-aperture sensor provide backups when pixels fail.

The researchers are now working out the manufacturing details of fabricating the micro-optics onto a camera chip.

The finished product may cost less than existing digital cameras, the researchers say, because the quality of a camera’s main lens will no longer be of paramount importance. “We believe that you can reduce the complexity of the main lens by shifting the complexity to the semiconductor,” Fife said.

Note for Three-Dimensional Printing
Three-dimensional printing is a method of converting a virtual 3D model into a physical object. 3D printing is a category of rapid prototyping technology. 3D printers typically work by 'printing' successive layers on top of the previous to build up a three dimensional object. 3D printers are generally faster, more affordable and easier to use than other additive fabrication technologies.

One variation of 3D printing consists of an inkjet printing system. Layers of a fine powder (plaster, corn starch, or resins) are selectively bonded by "printing" an adhesive from the inkjet printhead in the shape of each cross-section as determined by a CAD file. This technology is the only one that allows for the printing of full color prototypes. It is also recognized as the fastest method.

Alternately, these machines feed liquids, such as photopolymer, through an inkjet-type printhead to form each layer of the model. These Photopolymer Phase machines use an ultraviolet (UV) flood lamp mounted in the print head to cure each layer as it is deposited.

Fused deposition modeling (FDM), a technology also used in traditional rapid prototyping, uses a nozzle to deposit molten polymer onto a support structure, layer by layer.

Another approach is selective fusing of print media in a granular bed. In this variation, the unfused media serves to support overhangs and thin walls in the part being produced, reducing the need for auxiliary temporary supports for the workpiece.

Finally, ultrasmall features may be made by the 3D microfabrication technique of 2-photon photopolymerization. In this approach, the desired 3D object is traced out in a block of gel by a focused laser. The gel is cured to a solid only in the places where the laser was focused, due to the nonlinear nature of photoexcitation, and then the remaining gel is washed away. Feature sizes of under 100 nm are easily produced, as well as complex structures such as moving and interlocked parts.

Each technology has its advantages and drawbacks, and consequently some companies offer a choice between powder and polymer as the material from which the object emerges. Generally, the main considerations are speed, cost of the printed prototype, cost of the 3D printer, choice of materials, color capabilities, etc.

Unlike "traditional" additive systems such as stereolithography, 3D printing is optimized for speed, low cost, and ease-of-use, making it suitable for visualizing during the conceptual stages of engineering design when dimensional accuracy and mechanical strength of prototypes are less important. No toxic chemicals like those used in stereolithography are required, and minimal post printing finish work is needed. One need only brush off surrounding powder after the printing process. Bonded powder prints can be further strengthened by wax or thermoset polymer impregnation. FDM parts can be strengthened by wicking another metal into the part.

Standard applications include design visualization, prototyping/CAD, metal casting, architecture, education, geospatial, healthcare, entertainment/retail, etc.

More recently, the use of 3D printing technology for artistic expression has been suggested.

3D printing technology is currently being studied by biotechnology firms and academia for possible use in tissue engineering applications where organs and body parts are built using inkjet techniques. Layers of living cells are deposited onto a gel medium and slowly built up to form three dimensional structures. Several terms have been used to refer to this field of research: Organ printing, bio-printing, and computer-aided tissue engineering among others.

Note for Virtual World
A virtual world is a computer-based simulated environment intended for its users to inhabit and interact via avatars. These avatars are usually depicted as textual, two-dimensional, or three-dimensional graphical) representations, although other forms are possible (auditory and touch sensations for example). Some, but not all, virtual worlds allow for multiple users.

The computer accesses a computer-simulated world and presents perceptual stimuli to the user, who in turn can manipulate elements of the modeled world and thus experiences telepresence to a certain degree. Such modeled worlds may appear similar to the real world or instead depict fantasy worlds. The model world may simulate rules based on the real world or some hybrid fantasy world. Example rules are gravity, topography, locomotion, real-time actions, and communication. Communication between users has ranged from text, graphical icons, visual gesture, sound, and rarely, forms using touch and balance senses.

Massively multiplayer online games commonly depict a world similar to the real world, with real world rules and real-time actions, and communication. Communication is usually textual, with real-time voice communication using VOIP also possible.

Virtual worlds are not limited to games but, depending on the degree of immediacy presented, can encompass computer conferencing and text based chatrooms.

The concept of virtual worlds predates computers and could be traced in some sense to Pliny. The mechanical-based 1962 Sensorama machine used the senses of vision, sound, balance, smells and touch (via wind) to simulate its world. Among the earliest virtual worlds to be implemented by computers were not games but generic virtual reality simulators, such as Ivan Sutherland's 1968 virtual reality device. This form of virtual reality is characterized by bulky headsets and other types of sensory input simulation. Contemporary virtual worlds, multi-user online virtual environments, emerged mostly independently of this virtual reality technology research, fueled instead by the gaming industry but drawing on similar inspiration. While classic sensory-imitating virtual reality relies on tricking the perceptual system into experiencing an imersive environment, virtual worlds typically rely on mentally and emotionally engaging content which gives rise to an immersive experience.

The first virtual worlds presented on the Internet were communities and chat rooms, some of which evolved into MUDs and MUSHes. MUDs, short for “Multi User Dungeons,” are examples of virtual worlds that consist of virtual space inhabited by representations of data and other users . Early virtual worlds were text-based, offering limited graphical representation, and often using a Command Line Interface.

Maze War (also known as The Maze Game, Maze Wars or simply Maze) was the first networked, 3D multi-user first person shooter game. Maze first brought us the concept of online players as eyeball "avatars" chasing each other around in a maze.” (http://www.digibarn.com/history/04-VCF7-MazeWar/index.html, 29th Feb). According to the website this was in 1974, it was played on Arpanet (the initial internet), however it could only be played on an Imlac, as it was specifically built for this type of computer.

Then in 1978 MUD was released, it however was not 3D, it was text-based and used a TELNET program, by following the link you will be able to play the game, and understand just how far virtual worlds have come since http://www.british-legends.com/. You can understandably argue whether or not this is a “virtual world” and that Maze War was more sophisticated (being 3D), but you must understand that MUD could be played by anyone, Maze War was computer specific. Perhaps in today’s senses it is not a true virtual world, but the idea of a virtual world in those days were different.

Some early prototyptes were WorldsAway, a prototype interactive communities featuring a virtual world by CompuServe called Dreamscape, Cityspace, an educational networking and 3D computer graphics project for children, and The Palace, a 2-dimensional community driven virtual world. However, credit for the first online virtual world usually goes to Habitat, developed in 1987 by LucasFilm Games for the Commodore 64 computer, and running on the Quantum Link service (the precursor to America Online).

In 1996, the city of Helsinki, Finland with Helsinki Telephone Company (since Elisa Group) launched what was called the first online virtual 3D depiction intended to map an entire city. The Virtual Helsinki project was eventually renamed Helsinki Arena 2000 project and parts of the city in modern and historical context were rendered in 3D.

Note for 3D Modeling
In 3D computer graphics, 3D modeling is the process of developing a mathematical, wireframe representation of any three-dimensional object (either inanimate or living) via specialized software. The product is called a 3D model. It can be displayed as a two-dimensional image through a process called 3D rendering or used in a computer simulation of physical phenomena. The model can also be physically created using 3D Printing devices.

Models may be created automatically or manually. The manual modeling process of preparing geometric data for 3D computer graphics is similar to plastic arts such as sculpting.

3D models, the product of modeling procedures, are often created with special software applications called 3D modelers. Being a collection of data (points and other information), 3D models can be created by hand, algorithmically (procedural modeling), or scanned. Though they most often exist virtually (on a computer or a file on disk), even a description of such a model on paper can be considered a 3D model.

3D models are widely used anywhere 3D graphics are used. Actually, their use predates the widespread use of 3D graphics on personal computers. Many computer games used pre-rendered images of 3D models as sprites before computers could render them in real-time.

Today, 3D models are used in a wide variety of fields. The medical industry uses detailed models of organs. The movie industry uses them as characters and objects for animated and real-life motion pictures. The video game industry uses them as assets for computer and video games. The science sector uses them as highly detailed models of chemical compounds. The architecture industry uses them to demonstrate proposed buildings and landscapes. The engineering community uses them as designs of new devices, vehicles and structures as well as a host of other uses. In recent decades the earth science community has started to construct 3D geological models as a standard practice. A model is not technically a graphic until it is visually displayed. Due to 3D printing, 3D models are not confined to virtual space.

Note for Medical Imaging
Medical imaging refers to the techniques and processes used to create images of the human body (or parts thereof) for clinical purposes (medical procedures seeking to reveal, diagnose or examine disease) or medical science (including the study of normal anatomy and function). As a discipline and in its widest sense, it is part of biological imaging and incorporates radiology (in the wider sense), radiological sciences, endoscopy, (medical) thermography, medical photography and microscopy (e.g. for human pathological investigations). Measurement and recording techniques which are not primarily designed to produce images, such as electroencephalography (EEG) and magnetoencephalography (MEG) and others, but which produce data susceptible to be represented as maps (i.e. containing positional information), can be seen as forms of medical imaging.

In the clinical context, medical imaging is generally equated to Radiology or "clinical imaging" and the medical practitioner responsible for interpreting (and sometimes acquiring) the images is a radiologist. Diagnostic radiography (see Radiography) designates the technical aspects of medical imaging and in particular the acquisition of medical images. The radiographer or radiologic technologist is usually responsible for acquiring medical images of diagnostic quality, although some radiological interventions are performed by radiologists.

As a field of scientific investigation, medical imaging constitutes a sub-discipline of biomedical engineering, medical physics or medicine depending on the context: Research and development in the area of instrumentation, image acquisition (e.g. radiography), modelling and quantification are usually the preserve of biomedical engineering, medical physics and computer science; Research into the application and interpretation of medical images is usually the preserve of radiology and the medical sub-discipline relevant to medical condition or area of medical science (neuroscience, cardiology, psychiatry, psychology, etc) under investigation. Many of the techniques developed for medical imaging also have scientific and industrial applications.

Medical imaging is often perceived to designate the set of techniques that noninvasively produce images of the internal aspect of the body. In this restricted sense, medical imaging can be seen as the solution of mathematical inverse problems. This means that cause (the properties of living tissue) is inferred from effect (the observed signal). In the case of ultrasonography the probe consists of ultrasonic pressure waves and echoes inside the tissue show the internal structure. In the case of projection radiography, the probe is X-ray radiation which is absorbed at different rates in different tissue types such as bone, muscle and fat.

A Magnetic Resonance Imaging instrument (MRI scanner) uses powerful magnets to polarise and excite hydrogen nuclei (single proton) in water molecules in human tissue, producing a detectable signal which is spatially encoded resulting in images of the body. In brief, MRI involves the use of three kinds of electromagnetic field: a very strong (of the order of units of teslas) static magnetic field to polarize the hydrogen nuclei, called the static field; a weaker time-varying (of the order of 1 kHz) for spatial encoding, called the gradient field(s); and a weak radio-frequency (RF) field for manipulation of the hydrogen nuclei to produce measurable signals, collected through an RF antenna. Like CT, MRI traditionally creates a 2D image of a thin "slice" of the body and is therefore considered a tomographic imaging technique. Modern MRI instruments are capable of producing images in the form of 3D blocks, which may be considered a generalisation of the single-slice, tomographic, concept. Unlike CT, MRI does not involve the use of ionizing radiation and is therefore not associated with the same health hazards; for example there are no known long term effects of exposure to strong static fields (this is the subject of some debate; see 'Safety' in MRI) and therefore there is no limit on the number of scans to which an individual can be subjected, in contrast with X-ray and CT. However, there are well identified health risks associated with tissue heating from exposure to the RF field and the presence of implanted devices in the body, such as pace makers. These risks are strictly controlled as part of the design of the instrument and the scanning protocols used. CT and MRI being sensitive to different properties of the tissue, the appearance of the images obtained with the two techniques differ markedly. In CT, X-rays must be blocked by some form of dense tissue to create an image, therefore the image quality when looking at soft tissues will be poor. While any nucleus with a net nuclear spin can be used, the proton of the hydrogen atom remains the most widely used, especially in the clinical setting, since it is so ubiquitous and returns much signal. This nucleus, present in water molecules, allows excellent soft-tissue contrast.

In figure 1, Philip Wong, Abbas El Gamal and Keith Fife are developing a digital camera that sees the world through thousands of tiny lenses, providing an electronic “depth map” containing the distance from the camera to every object in the picture.

In figure 2, The testing platform for the multi-aperture image sensor chip