The 11.5M pixel HMD

A few weeks ago, we had the opportunity to ship to a customer one of our high-end HMDs, with a total of 11.7M pixels. Each eye in this HMD has 5.76 million pixels, about 2.7 times as much as an HD1080 signal.
The piSight 166-43 has a horizontal field of view of 164 degrees, which approaches the human visual field.

Building, testing and shipping such a high-end HMD is always a cause for celebration here. We sell lots of 120-degree and 60-degree products every year, but just a handful of the super high-end ones.

This particular piSight has 12 micro-dispalys per eye. The top photo on the right shows the complete HMD. The photo below it shows the eye modules: the left side of the image shows our patented tiled optics in front of the 12 displays, while the right side shows the screen configuration with the array removed.

It takes one or two computers to drive the HMD to its full potential, and we typically recommend driving it with four HD1080P or 1920x1200 signals which feed into our video processing electronics.

The field of view is just spectacular, as well as the level of detail across it. It is simply fun to see how immersive such product is.

Side by Side 3D and HMDs

A key advantage of HMDs over 3D televisions is the ability to get true depth perception in a surround video setting. However, depth perception inside an HMD requires different images for the left and right eyes. There are multiple ways to provide these separate images:

• Use dual video inputs, thereby providing a completely separate video signal to each eye
• Time-based multiplexing. Techniques such as frame sequential combine two separate video signals into one signal by alternating the left and right images in successive frames.
• Side by side or top/bottom multiplexing. This method allocated half of the image to the left eye and the other half of the image to the right eye.

The advantage of dual video inputs is that it provides the maximum resolution for each image and the maximum frame rate for each eye. The disadvantage of dual video inputs is that it requires separate video outputs and cables from the device generating the content.

Time-based multiplexing preserves the full resolution per each image, but reduces the frame rate by half. For example, if the signal is presented at 60 Hz, each eye is receiving just 30 Hz updates. This becomes an issue with accurately presenting fast-moving images, or images that need to rapidly change as a result of tracking information or user movement.

Side-by-side and top/bottom multiplexing provide full-rate updates to each eye, but reduces the resolution presented to each eye because only half the available pixels in each frame are used for each eye. Many 3D broadcasts, such as ESPN, chose to provide side-by-side 3D which saves the need to allocate extra transmission bandwidth and is more suitable to fast-paced sports action relative to time-based multiplexing techniques. Of the multiplexing techniques, I think side by side 3D is best for HMDs.

Side by side 3D has some advantages over the full dual-port mode when using a wireless video link. Sending two completely separate images requires a dual-stream video link or two separate transmitters. With side-by-side, true interactive 3D can be economically achieved with a single wireless video link.

Not all HMDs provide depth perception. Some lower-end modules are essentially bi-ocular devices where both eyes are presented with the same image. Interestingly, most professional HMDs do not provide support for side-by-side format. Fortunately for users, both the xSight and zSight professional HMDs do provide such support. Such support also allows you to tap into the wealth of entertaining 3D videos on YouTube 3D, view video directly from a 3D camera and more.

How will HMDs be impacted by the growth in 3D TVs?

Several million 3D TV sets shipped in 2010, with some reports claiming shipments of over 6 million sets. The industry is clearly aiming for higher volume in 2011. Is this 3D TV growth good or bad for the HMD market?

Very good, in my opinion.

3D TV are useless without 3D content, and this content is immediately useful inside head-mounted displays. Just like surround sound at home did not inhibit portable music players - the Walkman or the iPod - 3D TVs don't inhibit HMD sales.

With 3D TVs, an increasing number of users are exposed to the power of 3D and are thus stimulated to come up with new uses.

3D TVs will drive 3D games, but HMDs offer 3D gaming in a dynamic, 360 degree surround video environment that a TV cannot offer. HMDs can be portable and battery operated, whereas TVs are stationary and power-hungry. You can run around with an HMD on your head (especially with a wireless video link) but no one would consider running inside a room carrying a TV.

Do advances in fuel economy of cars impact air travel? Not really. Similarly, the use cases for TV and HMDs are different in other aspects as well. TVs are better suited to group viewing, whereas HMDs offer privacy. TVs provide all users with the same viewpoint, whereas embedded trackers inside HMDs (such as the zSight) can offer dynamic and individual viewpoints. Some exciting 3D content comes from animated features. How long before such content can be streamed and changed depending on head position? This would be excellent news for HMDs that can provide great immersion.

You can get some great 3D TVs for $2500. When will you be able to get great HMDs for $2500? Perhaps as soon as there is demand for 6 million HMDs a year.

New video demo of xSight head-mounted display

The xSight HMD is difficult to experience on YouTube, but here's a movie anyway. The reason it's difficult to experience is that two of its most important attributes: field of view and weight are best experienced in person. The horizontal field of view of an xSight is 120 degrees. If you sat 8 feet from a TV and wanted that TV to provide you with 120 degrees field of view, the width of the TV needs to be about 28 feet. That's much more immersive than a front row seat in most movie theaters.

Weight is also hard to experience in a movie. The weight of the xSight is about 350 grams, so similar to a Coca Cola can. The weight of other HMDs is sometimes 1.5 kg, like a 6-pack of beer on your head. Quite a difference!

New video of zSight HMD with iMove

A new video is available showing the integration of the Sensics zSight HMD with the iMove GeoView software package. iMove allows truly spherical recording of video, which is useful for many applications such as surveillance, tourism, mission rehearsal and more. A vehicle is outfitted with a pole carrying several cameras. Video is simultaneously recorded from all cameras. iMove's software then stitches the individual video streams and prepares a file which can then be played back using a standard desktop or notebook PC. The player receives head tracking information via a simple USB interface from the zSight HMD and changes the viewing direction based on the head movements. The result is like being inside the car.

Various cool videos are available: riding in DC, sailing in a canal in Amsterdam and many others. Several companies also offer services where they can create such immersive videos for desired locations.

Since there is so much video data being recorded at once, iMove decided (at least for this recording) to record at about 8 frames per second. This is more choppy than normal video, but creates a nice balance between the amount of storage required and the smoothness of the image.

Integration is also available for the xSight which provides an even wider field of view and even higher resolution.

New optics make an excellent product even better

The hard work was worth it. After many months of engineering and testing, my company is now shipping new and improved optics for the zSight SXGA HMD, in both its commercial and helmet-mounted versions.
HMD design is a study in trade-offs, needing to balance weight, resolution, cost, field of view, features and more. Optical design for HMDs is no different. Key attributes to be balanced include:

• Field of view. Wider is better for an improved experience. Low-end HMDs  Low-end HMDs offer 30 or 35 degrees of diagonal field of view. Professional HMDs offer wider solutions, some even greater than 100 degrees.
• Clarity. it's not difficult to design an optical system that provides image clarity at the center of the image, but reaching clarity at the edges or corners is not always an easy task.
• Geometric distortion. Minimizing pincushion or barrel distortion is very desirable. Distortion is often expressed in percent as a function of radial distance from the center of the lens, showing how far a point is away from its perfect theoretical position. Some professional HMDs have distortions greater than 10% (ouch!), but 5% or better is desirable.
• Weight. After all, these optical systems are designed to be worn on the head. Every gram or ounce counts. 
• Material used. There are hundreds of optical-quality glass types, but only a few optical-quality acrylic/plastic materials that are suitable for optics design in the visible spectrum. Glass offers greater flexibility in the design (much greater selection of refractive index) but is heavier, more expensive to manufacture, and introduces safety concerns in some applications if there is concern about glass elements close to the eye.
• Chromatic aberration. Similar to light passing through a prism, an optical system might impact different colors/wavelengths differently. This might cause blur and color separation.
• Eye box. Unlike a telescope lens, HMD optics are used when moving the head. The eye box specifies how much the user's eye can stray away left/right/up/down from the location of the center of the lens without losing the image or suffering significant degradation in image quality. Bigger eye box is better.
• Eye relief. This specifies how far can the user's eyes be away from the lens. Being too close is a source of discomfort - your eyelashes could brush up against the optics. A larger eye relief allows those with glasses to wear them.
• Diopter adjustment. Some designs allow for compensating for glasses - just like the eyepiece of binoculars allow for some adjustment so that those with glasses can, but don't have to, wear them. This is a plus.
• Physical attributes. Since this optics design needs to fit inside an HMD, it cannot be too long. Eyepieces cannot be too wide either because two eyepieces need to be next to each other with a minimum distance of about 52 mm to accommodate the desired ranged of interpupillary distance (IPD).
• Light transmission. Some designs transfer only 25% of the light from the source. Keeping most of light allows higher brightness and more energy-efficient HMDs.

The process of optical design involves specifying the design targets but then fighting through the trade-offs. Should we use glass? How much eye relief is good enough? Are you willing to sacrifice clarity for geometrical distortion? on, and on.

Many months and thousands of dollars into it, I think we nailed it. The optics are very lightweight (about 10 grams each), offer 60 degree field of view that is crisp to the edges. Low distortion. Big eye box and nice eye relief. There is no perfect solution because of the trade-offs but I think we hit a very good one.

I like looking at great design: an Italian wine glass, a Frank Lloyd-Wright house, an iPad. With these new optics, I liked looking through a great design.

The optics are now available as part of the zSight, a 60-degree OLED SXGA HMD with integrated tracker, audio and stereo sound. Try it if you get a chance.

Eyelids as shutter glasses

Is it real or just French humor?

If not fake, an amazing  demo illustrating how you could conceivably get alternating-frame/frame sequential 3D without shutter glasses by modulating the eyelids.

Probably belongs to the 'don't try this at home' category?

How long can you use this without damaging your eyes?

Would you be brave enough to try it on yourself?

Can nostrils be modulated for stereo smell?

I still prefer head mounted displays for true stereo, motion tracking and the full immersive 3D experience.