Training using the zSight HMD and the Kinect

An excellent video showing how the Kinect can integrate with the zSight HMD to product a compelling and effective training scenario. Quoting the author of the video, David from 3DVia: "We made the choice to only use kinect to "watch" if the user is doing the correct motion he/she is supposed to be trained at: rotating the valve or walking. Engaging the body in the experience enables the trained person to take appropriate decision, based on a situation that is made as close as possible to the one he/she would face in an real emergency case The HMD also contributes to the presence of the virtual environment, stereo rendering have been disabled in the video."

3D and the iPhone

Last year, when Hasbro announced the My3D, a 3D device, I was intrigued: how could a US list price of $35 support the cost of a micro-display, optics, control electronics and a decent enclosure? The answer, of course, is that it doesn't. The device does not have its own display - it houses an iPhone with specialized applications that present 3D images in side-by-side format; it does not have a motion tracker - it uses the iPhone's. It does not have drive electronics - for the same reason.

When the My3D came on the market in April, I immediately went out and got one. After all, buying gadgets is one of the few perks of my job. I downloaded the apps from iTunes, installed them on an iPhone 3GS and put the device to a harsh, un-scientific but brutally accurate evaluation: the teenager test. This is a test where I give a device to a bunch of teenagers and tell them "it's all ready to use; it's free of charge for you; just use it and tell me what you think". As expected, the first five minutes of the test were filled with 'wow - this is cool!' but from the sixth minute till this date, the My3D has never been used again. In contrast, the Motorola Xoom tablet, in spite of surprisingly poor video playback performance, has passed the teenager test with flying colors and is constantly being used.

Before discussing the upside, here are some of the things I don't like about the My3D:

• The iPhone 3GS screen, when used in side-by-side mode and magnified through the My3D eyepieces, is nothing to write home about. The resolution is low and the fill ratio of the pixels 
• When using the My3D, your hands need to hold the device.
• User interface is clumsy - holes are provided to insert the thumbs through the My3D enclosure and touch the iPhone screen

But there is an upside. The concept of using the phone (or tablet) to generate the graphics, provide the communication link, app store and perhaps even the motion sensor, is certainly valid. With phone and tablet CPUs becoming ever stronger (the Xoom has a dual-core NVIDIA Tegra) and with goggles supporting side-by-side 3D, you can get pretty good 3D experience without a PC.

The HMD and the fMRI machine

A couple of weeks ago, I returned from the annual meeting of the Vision Sciences Society in Naples, Florida. VSS is an organization of scientists who are interested in the functional aspects of vision. The VSS conference brings together in scientists from a broad range of disciplines including visual psychophysics, neuroscience, computational vision and cognitive psychology.

There is something nice about coming back to a conference year after year in the same location. You already know where to stay, where to eat or not to eat and are free to focus on enjoying the academics, the exhibition and the Florida sun.

Because VSS is primarily focused on academics, the show floor where my company and others exhibited was not often crowded, leaving exhibitors plenty of time to gleam knowledge from the poster sessions as well as chat with other vendors. The VSS show is dominated by eye tracking companies - most of which we've had the pleasure to work with - but HMDs such as the zSight and other (heavier and lower contrast) HMDs were also on display.

The academic crowd is this show is very diverse in their virtual reality experience. It was great meeting existing customers that we've worked with over the years, but just as fun demonstrating the HMD to researchers admitting that they have heard of head mounted displays but never got around to trying one. 

A key part of many vision research projects is to generate visual stimuli. HMDs offer a superb vehicle for doing so, for several reasons:

Arrington Research Eye tracker inside a Sensics zSight

• The ability to offer stereoscopic stimuli, thus also adding depth perception to studies about motion, color, recognition and more.
• A good HMD is often immersive, meaning that it can block external distractions and make sure that the subject is focused on the stimuli inside the HMD.
• The ability to integrate eye tracking inside the HMD.
• The ability to engage peripheral vision, especially when using an HMD with a panoramic field of view

I had the opportunity to browse the poster session focusing on research involving fMRI. Functional MRI studies put a subject inside an MRI machine and generate certain visual stimuli. The brain function is then measured and analyzed. Some of these projects give scientific confirmation to seemingly obvious results and others open doors to deeper understanding of how people see. For instance, I saw a poster that confirmed that the brain produces different neural responses when presented with pictures of familiar people as opposed to non-familiar people. To me this falls more into the obvious category: if a picture is instantly recognized as familiar, doesn't recognition itself mean that the brain has a different neural response to it?

Some companies on the show such as Cambridge Vision Systems devise clever ways to inject visual stimuli into such a challenging magnetic environment such as an MRI, and even have the ability to do fMRI eye tracking. Really cool stuff. Over the years, we got asked several times of an HMD could be used inside an MRI machine either for research or as distraction to prevent claustrophobia. Unfortunately, our HMDs are not MRI compatible today, though I'm open to ideas on how to make one.

During a typical year, I meet different people that have different uses for HMDs: from training soldiers inside tanks to architecture and industrial design to academic research. The possibilities for using HMDs seem endless, and many people I meet have new and sometimes crazy ideas on what to do (such as the person suggesting HMDs as a way to revive the art of playing an ancient banjo). Regardless, I look forward to returning to VSS next year and learn even more.

Will Microsoft acquire Lumus Optical?

There is a rumor going around that Microsoft is negotiating the acquisition of Lumus Optical. Lumus is a startup company that developed a clever see-through eyewear. The company originally targeted both the professional (e.g. defense) market as well as the consumer market, though it reportedly sold off its defense business to focus on the consumer side. The public specifications of its product show a low-resolution (640x480), narrow field of view (27 degrees) display, but one that could be made to resemble a pair of normal glasses.

Publicly, Lumus has been keeping a very low profile. The most recent press release or news article on its Web site, for instance, is dated some two years ago. If it not for their promotion of the upcoming SID show, an outsider could conclude that Lumus is out of business.

Of course, it may very well be that this rumor is untrue or that the acquisition talks will not come through, but it's a good opportunity to analyze where such an acquisition could fit.

One place where a see-through display could fit is with the Microsoft mobile division as part of the Windows phone initiatives. A classic use case of a Bluetooth-equipped see-through display with a phone is showing the caller ID inside the glasses so that the user does not have to pull out or look down at the phone. With phones become more and more sophisticated, dynamic driving directions have also been discussed.

How much would people be willing to pay for such glasses and what would they cost to make? This has been a classic chicken-and-egg problem for goggle manufacturers, where price depends on quantity and quantity depends on price. If Microsoft were to bring this product to market, they could all but guarantee a large enough quantity to make these economical.

Another possible home for a see-through display ix the XBOX group. If motion sensing (e.g. Wii, Kinect) was the last big thing in gaming, 3D, immersion and freedom of motion could be the next. I find this a bit less likely that mobile because gaming often requires immersion and 27 degrees of FOV, at least for the current product, is far from being immersive.

One way or the other, such an acquisition would be a nice shot in the arm for all the goggle vendors trying to find their way to a mainstream market.

If Microsoft ends up acquiring Lumus, you've heard it first here! If they don't, I just might change the title of this post to "Should Microsoft acquire Lumus".

The 11.5M pixel HMD

A few weeks ago, we had the opportunity to ship to a customer one of our high-end HMDs, with a total of 11.7M pixels. Each eye in this HMD has 5.76 million pixels, about 2.7 times as much as an HD1080 signal.
The piSight 166-43 has a horizontal field of view of 164 degrees, which approaches the human visual field.

Building, testing and shipping such a high-end HMD is always a cause for celebration here. We sell lots of 120-degree and 60-degree products every year, but just a handful of the super high-end ones.

This particular piSight has 12 micro-dispalys per eye. The top photo on the right shows the complete HMD. The photo below it shows the eye modules: the left side of the image shows our patented tiled optics in front of the 12 displays, while the right side shows the screen configuration with the array removed.

It takes one or two computers to drive the HMD to its full potential, and we typically recommend driving it with four HD1080P or 1920x1200 signals which feed into our video processing electronics.

The field of view is just spectacular, as well as the level of detail across it. It is simply fun to see how immersive such product is.

Side by Side 3D and HMDs

A key advantage of HMDs over 3D televisions is the ability to get true depth perception in a surround video setting. However, depth perception inside an HMD requires different images for the left and right eyes. There are multiple ways to provide these separate images:

• Use dual video inputs, thereby providing a completely separate video signal to each eye
• Time-based multiplexing. Techniques such as frame sequential combine two separate video signals into one signal by alternating the left and right images in successive frames.
• Side by side or top/bottom multiplexing. This method allocated half of the image to the left eye and the other half of the image to the right eye.

The advantage of dual video inputs is that it provides the maximum resolution for each image and the maximum frame rate for each eye. The disadvantage of dual video inputs is that it requires separate video outputs and cables from the device generating the content.

Time-based multiplexing preserves the full resolution per each image, but reduces the frame rate by half. For example, if the signal is presented at 60 Hz, each eye is receiving just 30 Hz updates. This becomes an issue with accurately presenting fast-moving images, or images that need to rapidly change as a result of tracking information or user movement.

Side-by-side and top/bottom multiplexing provide full-rate updates to each eye, but reduces the resolution presented to each eye because only half the available pixels in each frame are used for each eye. Many 3D broadcasts, such as ESPN, chose to provide side-by-side 3D which saves the need to allocate extra transmission bandwidth and is more suitable to fast-paced sports action relative to time-based multiplexing techniques. Of the multiplexing techniques, I think side by side 3D is best for HMDs.

Side by side 3D has some advantages over the full dual-port mode when using a wireless video link. Sending two completely separate images requires a dual-stream video link or two separate transmitters. With side-by-side, true interactive 3D can be economically achieved with a single wireless video link.

Not all HMDs provide depth perception. Some lower-end modules are essentially bi-ocular devices where both eyes are presented with the same image. Interestingly, most professional HMDs do not provide support for side-by-side format. Fortunately for users, both the xSight and zSight professional HMDs do provide such support. Such support also allows you to tap into the wealth of entertaining 3D videos on YouTube 3D, view video directly from a 3D camera and more.

How will HMDs be impacted by the growth in 3D TVs?

Several million 3D TV sets shipped in 2010, with some reports claiming shipments of over 6 million sets. The industry is clearly aiming for higher volume in 2011. Is this 3D TV growth good or bad for the HMD market?

Very good, in my opinion.

3D TV are useless without 3D content, and this content is immediately useful inside head-mounted displays. Just like surround sound at home did not inhibit portable music players - the Walkman or the iPod - 3D TVs don't inhibit HMD sales.

With 3D TVs, an increasing number of users are exposed to the power of 3D and are thus stimulated to come up with new uses.

3D TVs will drive 3D games, but HMDs offer 3D gaming in a dynamic, 360 degree surround video environment that a TV cannot offer. HMDs can be portable and battery operated, whereas TVs are stationary and power-hungry. You can run around with an HMD on your head (especially with a wireless video link) but no one would consider running inside a room carrying a TV.

Do advances in fuel economy of cars impact air travel? Not really. Similarly, the use cases for TV and HMDs are different in other aspects as well. TVs are better suited to group viewing, whereas HMDs offer privacy. TVs provide all users with the same viewpoint, whereas embedded trackers inside HMDs (such as the zSight) can offer dynamic and individual viewpoints. Some exciting 3D content comes from animated features. How long before such content can be streamed and changed depending on head position? This would be excellent news for HMDs that can provide great immersion.

You can get some great 3D TVs for $2500. When will you be able to get great HMDs for $2500? Perhaps as soon as there is demand for 6 million HMDs a year.