Expect 2012 to be a transformational year for HMDs

2011 was a 'business as usual' year as HMDs go. Sure, there were some new products and product improvements, but nothing earth shattering. At the same time, improvements in displays, motion sensing, augmented reality and processing power are coming together to enable 2012 to be truly transformational.

Sure, some nice progress was made in 2011 in HMDs:

• Higher resolution OLEDs became part of a new generation of lighter, brighter HMDs such as the zSight.
• Low-latency wireless video is moving into mainstream. I believe it will become a common accessory to HMDs, much like head trackers.
• In general, companies like Sensics continued to improve their optics designs, packaging, level of integration and features of professional HMDs.

Having said that, I believe that what's truly going to impact HMDs - the professional models - and VR goggles - the consumer models in 2012 were recent improvements made outside the direct HMD realm:

• Lower-cost micro-displays of 'good enough' resolution are becoming available. Once 720P displays are available for ~50$, the economics of goggles change, because displays have historically been a key cost driver for the HMD/goggle cost. What's the difference between a $10K HMD and a $15K HMD? $5K, of course, but otherwise there is no big difference in market penetration. It's hard to see why a $10K price would suddenly cause cause skyrocketing quantities whereas a $15K HMD did not. However, the difference between a $1K goggle and a $6K HMD, or better yet a $500 goggle to a $6K HMD would be huge as the $500 product would suddenly allow an entirely new group of users and entirely new class of goggle applications.
• Motion tracking is improving - both inside and outside the HMD. Driven by new generations of cell phones, tablets and game controllers, embedded motion trackers with 6 degrees of freedom (X/Y/Z, yaw/pitch/roll) are becoming good enough and inexpensive enough to make it into practically every product. Not only does this this make them an automatic add-on to products, but their increased quantity in the market opens up new ideas and new development techniques to use them. Outside the HMD, the success of the Wii and the Kinect made it obvious how powerful gesture-based and natural interaction are, and these developments will soon make it into goggles, such as the new SmartGoggles offering.
• Just like micro-displays are experiencing their version of Moore's law, embedded CPUs are continuing their march. New phones and tablets these days could have 1.5 GHz dual-core processors with embedded graphics and 3D accelerators. This is a very powerful computing platform. Because they are embedded in phones and tablets, these processors have to be lower-cost, power-efficient and small - all a perfect fit to embedding them into or alongside goggles. At the same time, having an open operating system like Android allows support of new hardware platforms as well as a nearly-endless supply of apps and content to consume.
• The improvements in processing power as well as cost and resolution of embedded cameras have given a huge boost to augmented reality applications. Driven by phones and tablets, both the applications and the knowledge gathered while building them will become very useful for goggles.

All taken together, a new generation of goggles will appear. If the typical goggle in 2010 was a lens, a micro-display and a video cable, it will evolve to be much more in 2012/2013. Combine on-board processing, embedded motion tracking for head, hands and others, augmented reality power and lower-cost micro-displays, and the mindless HMD will now morph into something between a fully-fledged independent computing and user interface platform to at least a highly-sophisticated display and virtual reality co-processor for phones, tablets, consoles or PCs.

Sure, there will still be a market for traditional HMDs just like there is still a market for flip phones. Sometimes, users don't need the sophistication of the Smart Phone, or don't want to pay for the data plan or have some ruggedization requirement that can only be satisfied by a flip-phone. Similarly, a special defense packaging, an ultra-low cost or some other requirement will continue to keep traditional HMDs in the market for years. However, I believe true growth will come from the smarter goggles that will take advantage of the above technologies to create a more complete and compelling user experience.

Happy Near Year!

The SmartGoggle

Today, my company introduced the SmartGoggle, the solution for the common virtual reality goggle.
The issue with today's goggles is obvious: they are mindless; just a monitor on your head. Just like a monitor has a video input and will display almost anything you pipe into that video input, a traditional goggle will display that signal in front of your eyes. If you provide two signals, or side-by-side video, it can do so in stereo as well.

But a traditional goggle, like a monitor, requires an external video source. It can be a computer, in which case you can do something interactive. It can be your iPod, so you can use the goggle as a media viewer. Useful, but boring.

While all of this is going on, tablets and smart phones have become increasingly powerful in their computing and graphics capabilities. So, as we were thinking about how to make goggles better, it became obvious that putting some good processing power inside the goggles could do a lot of good. For one thing, you could run applications on the goggles and not have to carry around an external computer with you. The goggles, and these applications, could go with you anywhere, which is cool. These could be connected applications - streaming something from the Web, or these could be local applications such as those using an on-board camera to drive an augmented reality app. Yes, you should still be able to use an external video source to drive your goggle, but now you can also drive it 'from the inside'. This is a bit like the new crop of smart TVs that are becoming very popular. You could connect them to your cable provider or DVD player, but they can also stream a Netflix movie or surf the web using an on-board processor. So far, so good.

The problem of the user interaction still remains. Sure, you can put a head tracker so that the view can change as you rotate your head, but this is typically not enough for true interaction with the application. You can use an external device - a phone, joystick or even a Kinect - as input, but these approaches have limitations: you have to carry them with you; you have to stand in front of a sensor; you are limited in your tracking area; good, but not good enough for goggles.

We then started thinking: what if we put a camera that can track your hands and make that information - both in raw form as well as after gestural analysis - available to the application running on the goggle. That might be cool, because the hands go with you everywhere and because cameras on your head can often see your hands regardless of where you are and which direction you are looking at. Better yet, if we put an array of cameras on the head, we can get depth perception of the hand location as well as get a really wide tracking area. We call this 'first person hand tracking'.

Last, we realized that many goggles are essentially the same on the inside. Sensics, for instance, makes a commercial goggle and then repackages it into a different enclosure for training and simulation applications. Anyone wanting to build a goggle will need to cover several areas: driving displays; head tracking; video processing and more. Given this, it made sense to design a module that essentially encapsulates all these functions and allows goggle developers to focus on the design/styling aspects of the goggles rather than on building everything from scratch time and again.

We've been working on this for a while and are excited with the progress and the initial feedback we are receiving. If you combine these three innovations: on-board Android machine, real-time hand tracking from a first-person perspective, and a 'system on a module' approach for encapsulating most of what's needed to build a goggle, we think you get something. A SmartGoggle.

Remember when the iPhone came out and suddenly people started realizing that it's not just a phone, but much more than that? We think SmartGoggles can be to mindless goggles what smart phones are to flip phones. A major step forward, which we are very excited to take today.

HMDs are mindless - a monitor on your head

My company makes VR goggles (professionally called: HMDs), so when I say that HMDs are mindless (or stupid or unintelligent or dumb), it is with a somewhat heavy heart.

After all, HMDs are sophisticated devices. To build one, you need to understand optics, electronics, ergonomics, mechanical design and system engineering. You need to balance features with field of view with weight and comfort. HMDs are indeed sophisticated, but nevertheless mindless.

I call HMDs mindless because they are not much more than a fancy monitor on your head. Like a monitor, you need to connect an external video source: a computer, MP4 player, phone or tablet, to provide a signal to be viewed. If the video signal is provided by a cable, it limits your mobility. If it provided via a wireless video link, it cuts the cable but still provides limits on your distance from the video source.

The passive nature of HMDs is not just because it's a 'monitor on your head'. Unless you are content with using the HMD as a media viewer, you will want to interact somehow with the content and with the HMD. In today's products, you might have some push buttons on the HMD and you usually have a head orientation tracker than can let the application know where you are looking. This is a decent start, but most of the user interface experience - selecting menus, interacting with content, moving 3D objects around, still relies on external devices: a joystick, a mouse, a data glove and others. By the way, many wireless video links don't have the ability to send head tracking information back to the video source.

Sure, this is fine for some applications. Military training and simulation applications are sometimes OK with having the soldier being trained carry a computer on his back. Stationary applications (a tank trainer) or those relying on lots of peripheral equipment (such as an academic research project measuring brain activity) can greatly benefit from today's HMDs, but is this enough for widespread use?

The promise of HMDs was to be able to take 3D content anywhere and interact with it in a useful way. Though more complex, it is just like how the iPod allowed us to take our music library with us anywhere and sufficiently interact with it to be useful. I don't think today's HMDs are fulfilling this promise for a broad-enough market.

Something needs to be done about it.

.

Gentex acquires Intersense

A couple of days ago, Gentex Corporate announced that it had acquired Intersense. Gentex makes various military products including helmets, eye protection devices, aluminized fabric and helmet-mounted displays. Intersense makes various inertial trackers.

I've had the pleasure of working with Intersense over several years, where we partnered on delivering virtual reality solutions for academic and defense markets. To many of our customers, Intersense has  always been considered the 'gold standard' of tracking, which is why I find this acquisition disappointing.

The financial details of the acquisition were not published, so perhaps this is a big win for the shareholders of Intersense, but I feel the company will now take an even stronger defense focus and will thus miss the bigger consumer product opportunity.

It is ironic that the Intersense acquisition happened just a week after the Invensense IPO (INVN), which values Invensense at some $800M. 

Founded in 1996, Intersense makes professional motion sensing products. Motion sensing is becoming more and more commonplace - just look at the Wii or the iPhone. Surely, Intersense had plenty of expertise in sensors and signal processing to offer a compelling low-cost product that builds off  Intersense's professional reputation but offers a compelling price point. 

As motion tracking became more common, lower-cost solutions started to appear. For instance, Sensics offers an integrated three degree of freedom (yaw/pitch/roll) head tracker inside the zSight professional HMD. Though the performance of the embedded tracker is not as good an Intersense IC-3, for instance, we found that for many customers it was good enough and saved them the need to pay a couple of thousand dollars for an external tracker. Intersense could have probably easily offered this lower-cost tracker, but they did not. Missed opportunity, I think, both for Intersense and for the market.

Good luck to the excellent team at Intersense, in whatever market you choose to serve!

Towards socially-acceptable goggles

Holiday season is fast approaching in the US and retailers are starting to offer attractive shopping discounts. A few years ago, my wife asked me to go to a home goods store during holiday season and buy some wine glasses. It was a cold day, and my cell phone was tucked inside my jacket while I was using an earphone in the store to speaking with my wife and decide together on which wine glass set to buy. At that time, mobile earphones were such an uncommon sight that several older ladies approached me - as I had appeared to be speaking to myself - to inquire if everything was OK.

How times have changed. Using a cell phone earpiece or Bluetooth device in a shopping mall - a complete oddity just a few years back - is now completely socially acceptable.

When would it and what would it take to have wearing virtual reality goggles become socially acceptable? Today, when I meet someone at a coffee shop and take out the goggles, people start coming over from adjacent tables to take a look. That's great if you want to sell goggles, but what would need to happen so that you can wear goggles in public unnoticed?

Clearly, the definition of what is 'socially acceptable' changes over time. Purple hair color? Diamond studs for man? Body piercing? Bluetooth headsets? All of these used to be outside the norm and now hardly worth a second look.

What were the motivations of the first people that stepped out of the 'socially acceptably' domain? Using a headset is probably about utility: it is more convenient, perhaps healthier. I don't think too many people used a headset because they felt it was overly aesthetic. On the flip side, there was little actual utility in dying your hair purple. A diamond stud doesn't make you hear better, but some think it makes you look better or draw more attention.

In the case of virtual reality goggles, we'll probably have to go the utility route. If you just want a pair of cool-looking sunglasses, just go to the store and buy cool-looking sunglasses!

What will goggles need to do so that they would be widely worn? If a Bluetooth headset is an audio extension of your phone, what will the visual extension of the phone be? Is it just about reading text messages or looking at driving directions without starting at the phone? What is the killer app? Write back to let me know what you think.

The Stark HUD 2020 Augmented Reality goggles

A few weeks ago, I wrote about the Stark HUD augmented reality goggles. These are not real AR goggles, but rather a clever promotion for Iron Man 2. Nevertheless, the video is compelling and worth reflecting on what it is so.

1. The goggles are visually attractive and appear to be 'socially acceptable' to wear. Though the definition of socially acceptable is fluid - just a few years ago it seemed odd to walk around with a Bluetooth headset - the closer goggles look like glasses, the better.

2. Information is context sensitive. The idea of context-aware computing has been around for quite some time, and there are patent filings on it dating at least 10 years ago. However, the availability of location-based services, and development in applications that know more and more about the user (Apple's Siri is a great new example), make the goggles useful much more than just being used as a media viewer.

3. The field of view appears wide, and resolution is good enough to read.

4. There is plenty of natural interaction with the device. The video demonstrates interaction using hands and voice commands.

5. No battery or control box. Science fiction, maybe, but still cool.

Will we really need to wait until 2020 to get them?

Steve Jobs and the HMD

The Web is buzzing about iTV, Apple's HDTV and what it could look like and do. Though Apple has filed some patents related to HMDs, it does not appear that Steve Jobs left us with an HMD design.

What would an HMD designed by Steve Jobs look like? What would it do?

As a business person, I want to build successful companies, and while success can be measured in many financial metrics, one aspect of a successful company would be a company that creates an emotional attachment with its customers.

My little personal story with Apple began many years ago, when I was about 12 years old and living in Boston. I was one of these kids that stood at the corner Radio Shack stores trying to do fun stuff with their TRS-80 computers. The personal computer revolution just started. IBM had not introduced the PC yet, and I wanted a computer of my own. My family lived in an apartment building which, for a 12-year old, was fertile ground do earn money babysitting. Lots of babysitting. Once I decided I wanted an Apple (other candidates were the TRS-80, the CompuColor and perhaps some others that I don't remember), I saved every babysitting penny towards the $1200 goal of getting an Apple II. My parents opened a bank account for me and I would stand in line to deposit $12 here, $30 there, until I reached the magic number and get my Apple II. Its serial number had 4 digits and it was that older model that had a reset key on the keyboard that, while spring loaded, was too easy to press and wipe out everything. I spent hours and hours programming, playing and hacking on that computer. Many years later, my mother donated it to a local school - it was the right thing to do but I miss my old Apple II. I can't say I miss any of the numerous PCs I've went through since.

What was so good about that Apple II? I loved its design. It was open so that you could plug cards inside. It was simple to use and "it just worked" year after year after year. That allowed so many people to write cool software and so on in a vitreous cycle.

What can we learn from Apple about HMDs?

• Design matters. Make people want to use or - in the case of HMDs - wear your product
• Keep it simple. Plug and play. No drivers to install. No configuration to go through. "It just works".
• Open it up so that value could be added outside your company.
• Price matters, but you don't have to be the lowest-cost product to succeed
• Make it simple to use content that you already own
• Humanize the product, if possible. 

Realistic oil&gas plant simulation with an HMD

Our partner VRContext released a video demonstrating walking inside a sophisticated plant using an XBOX 360 controller and a zSight HMD.

What I find particularly appealing about this video is the level of detail. Walking through the plant (while hearing your footsteps, no less!), you say layers and layers of pipes, pumps, floors, etc. This is possible through integration with real engineering databases. Since the plant is already designed, why not use the same databases to ensure safer, lower-cost operation?
I can certainly see why learning about this plant in a comfortable, virtual setting, will prepare the operator for true troubleshooting and maintenance when needed.

Cloud watching with an HMD

Below is a comic strip, not an instruction manual, but I guess it is possible to try this at home. Some notes after the comic strip.

Normally, when placing stereoscopic cameras or rendering a virtual image, it is important to try and match the actual distance between the eyes. That's why 3D camera rigs try to minimize the distance between the cameras, which are pretty large:

Having said that, there is no depth perception in such a large distance as looking at the clouds, so substantially increasing the distance between the eyes might create some 3D effect. 

Preview to thoughts about Augmented Reality goggles

One of the followers of this blog asked what I think about augmented reality goggles. Great question, worth discussing in more detail. In short, I don't think today's solutions are there yet, partially because of technical limitations and partially because of the currently-available applications. More on that soon. In the meantime, this video shows what good augmented reality would be like.

The VR goggle as a media viewer

Several VR goggles (such as the Carl Zeiss Cinemizer) present themselves as media viewers, which translates to a really cool way to watch movies from your iPod.

Is there really a significant market for VR media viewers?

Have you ever gone to a movie theater to watch a movie alone? Sure, when I travel, I sometimes go to restaurants alone, but I don't remember going to a movie alone. Isn't watching a movie alone what VR media players are designed for?

I tried researching how many people go to see movies alone and could not find the answer. There are tons of movie stats from the Motion Picture Association of America, the movies alone is not one of them. Googling this brings up pages titled "Do you ever go to the movies along and feel weird for doing it?"

People watch movies alone at home, but then they can use a TV and don't need a low-resolution set of goggles to do so (Cinemizer had 640x480 pixels, though perhaps has gone up since). Watching movies on a plane is a potential use, but wouldn't an iPad or other tablet be a better experience? Privacy is a plus when using goggles, but would you really spend $500 on a pair of goggles only to use them on a plane to watch movies that you are uncomfortable having your neighbor see?

Sure, I get using an iPod, not to mention that since as of Dec 2010 "only" 297 million units were sold, there is clearly a big market. You can do other things while using an iPod, but you can't do much when watching a movie on a set of VR goggles.

So, I don't get it. Any ideas?

A 3D look at the Vuzix financial reports

In the spirit of the previous financially-related post, I decided to take a look at the Vuzix (VZX.V) 2nd quarter SEC 10-Q filing and see if there is anything interesting to share from a market direction standpoint.


Some disclaimers: I am not a financial adviser and I don't make stock recommendations. You should buy or sell Vuzix stock based on what you read in this post. It is just my opinion as an individual fairly versed in virtual reality, not a Wall Street person. Also, I don't own Vuzix shares so I don't stand to gain or lose if their share price changes. This analysis is fully based just on the public information contained in the 10-Q.


Having said that, let's take a look at some of the interesting nuggets in the report, in no particular order:


VR Market-focused:

• Sales of what the company calls consumer eye wear products are in decline. It sold $1M of consumer products in the first six months of the year compared with $2.2M in the first six months of 2010.
• Sales of defense products, such as eyepieces, are on the rise.
• The company notes that demand for their consumer eye wear has declined for two reasons. The first is that they pre-announced a higher-performance product which they did not deliver yet, causing customers to slow down their purchasing. The second is that the iPad and other high-resolution tablets are used as media viewers instead of goggles. Personally, I don't fully understand the market need for goggles as video viewers: Resolution on the low-end products is fairly poor (I don't think that Vuzix has an HD product - not even HD720 - in that category); Privacy is not truly needed unless you are viewing certain kinds of media in public places.
• The company says it is going to place lower emphasis on consumer products and phase out some of their low-end products. Instead, they will focus on the defense market and upscale products such as their recently announced $5K augmented reality glasses which is priced for academic research (a fun, but not very large market). I can understand the attraction of the defense market. Sales are less sporadic (Vuzix reports a backlog of $3.8M on page 18); margins are higher. But, how large can this defense market really get?
• It seems that Vuzix which is/was known for consumer goggles as trying to move uptown in price, whereas professional HMD providers are being encouraged to produce affordable mass-market products. As the saying goes: "The grass may look greener on the other side, but believe me, it's just as hard to cut."

Financially-focused:

• Cash is getting tight (page 2). Cash on hand + accounts receivable was approximately $1.8M at the end of Q2, as compared with approx $4M at the end of 2010.
• Accounts receivable ($1.2M) is substantially lower than accounts payable ($3.1M)
• Customer deposits (these are typically pre-pays or down-payments) are nearly $900K and are higher than cash on hand (approx $700K). Without down payments, the company looks like it would be in a difficult financial situation.
• Inventories ($3.8M, page 2) are more than twice the second quarter product sales (about $1.7M, page 11). Since inventory is typically recorded at cost yet product sales reflect actual prices, this is even more startling. Assuming material cost is the primary driver of cost of sales, and looking at Q2 cost of sales ($1M cost on $1.7M product revenue), $3.8M inventory could be enough for more than $6M worth of product sales, which are at least 9 month's worth of sales. It would appear that Vuzix is sitting on really large inventories. Are these obsolete parts? Do they have very long-lead items that require substantial stocking? Were they planning for much higher sales? Page 7 shows the breakdown of the inventory and shows more than $1M of finished goods waiting to be delivered or sold. On page 17, the first item mentioned in discussion on how to improve cash position is "managing our working capital through better optimization of inventory levels."
• There are two loans totaling over $400K made to the company by one or two of its officers. Clearly, cash is a bit of a struggle. There is also accrued compensation of over $100K to be paid in the future to officers.
• There is an interesting margin discussion on page 14. Apparently, video eyewear is not a truly profitable business. Augmented reality is a bit more, but the best business from a gross profit perspective is engineering services related to the defense side of the business.
• The shift away from consumer products is reflected in lower sales and marketing costs, which the company notes are in part due to reduction in catalog advertising costs (Skymall?)

It's great to have a small public company to provide everyone with some market data on virtual reality!

I think Vuzix was tempted to try and create a substantial consumer market for goggles by offering reasonably-priced, nice-looking products. In my opinion, the experience is just not compelling enough both in terms of resolution/field of view and also in terms of content (games/movies) that can take advantage of the goggle capabilities. As such, not enough goggles are being sold and the company is perhaps unable to get the economies of scale it was hoping for as well as to cover its relatively fixed costs for product development and advertising.

Would the company's product fared much better if they were offered by a bigger brand (e.g. Apple), or is there some other missing ingredient needed to break open the consumer market?

Popcorn carts and the economics of HMDs

The Sensics YouTube channel received a comment noting that for the price of an HMD, "I might as well build a 30' theater room onto my house with stadium seating and a popcorn cart. Please explain how that price is justified, I'd love to hear it.". 


So, at the risk of being a bit wonkish, here are some insights on how the price is determined.


First, professional HMDs are currently made in small quantities. Let's try to figure how many:

• A company making professional HMDs was recently listed in Inc magazine. Inc. reported their 2010 revenue as $4.8M. If the average price of an HMD they sell is $20K, and even if we assume that they sell nothing but HMDs, this company made 240 HMDs in 2010. Maybe my price assumptions are too high and they made 300 HMDs. Hundreds, not hundreds of thousands.
• A small public company that sells consumer-type goggles released their second quarter financials. They sold approximately $720K of consumer goggles in the second quarter. If all of these sales were goggles, and these goggles were sold for an average price of $250, then they sold just under 3,000 units in the quarter. Thousands, not hundreds of thousands.

In contrast, there were approximately 10 million 3D televisions made last year. With quantities, come economies of scale: you can get parts cheaper, you can invest in manufacturing technologies to make cost lower, you can assemble HMDs at low-wage areas and so forth.

Aside from quantities, one needs to consider the cost of the components that go into making an HMD. Have you ever been with a friend at a restaurant only to hear why the components of the $15 salad probably cost $2.34? It's pretty annoying, but I'm going to do this a little bit for HMDs now.

Historically, the key difference between professional HMDs and consumer goggles was the resolution of the display and the width of the field of view. Professional HMDs had higher resolution and wider field of view. Many professional vendors, Sensics included, use 1280x1024 OLED microdisplays from another small public companies. We use these microdisplays because they are high brightness, low power, high contrast and have rich colors. However, they are expensive. On the open market, these displays (and other like them) could cost approximately $2500 each, so approximately $5000 per system. Yes, manufacturers like Sensics do get quantity discounts, but in my opinion the displays drive the cost of the product into a cost that is in line with a professional market. Once prices are in the range of a professional market, vendors perform their own price/quantity optimization. If you priced a hypothetical product at $20,000 apiece, you could sell a certain quantity. If you priced it at $18,000 apiece, you could probably sell slightly larger quantity, but not dramatically larger. On the other hand, you just lost $2000 of profit. Now, if you could price the product at $995 you would sell many, many more, but then you'd be selling it at a significant loss.

Of course, a direct relationship between cost to make and price to sell is not required. Does a basketball shoe suddenly become much so more expensive to make when an 'Air Jordan' sticker is placed on it? Not necessarily. It is just deemed to be worth more. Sensics is  profitable, and the Sensics team would like to keep it this way. In fact, though customers clearly appreciate a discount when they can get one, customers are probably also interested in diversity and choice among HMD companies, so it's good to keep more of them around.


What to do? Priority one is to find a lower-cost solution to placing these dynamic images in front of your eyes. This could come from various places:

• Find some lower-cost display. Imagine if TI started making displays just like they make DLP components. Imagine if Samsung, or Sharp, or Micron started to sell high-resolution OLEDs to HMD vendors. Then, prices could significantly come down.
• Find some alternative display configuration that is not micro-displays. 
• Place one display in front of both eyes, though this could reduce the refresh rate that each eye sees.
• The passage of time. Just like Moore's laws with CPUs, time brings higher resolution displays at a lower price point.

Last, but not least, what you can do with a 30' theater room and a popcorn cart does not fully overlap with what you can do with a professional HMD. Try carrying the room around from place to place. Try training a soldier with the popcorn cart. But, I am on board. Let's hear it for lower-cost displays that will enable making lower-cost HMDs.

Training using the zSight HMD and the Kinect

An excellent video showing how the Kinect can integrate with the zSight HMD to product a compelling and effective training scenario. Quoting the author of the video, David from 3DVia: "We made the choice to only use kinect to "watch" if the user is doing the correct motion he/she is supposed to be trained at: rotating the valve or walking. Engaging the body in the experience enables the trained person to take appropriate decision, based on a situation that is made as close as possible to the one he/she would face in an real emergency case The HMD also contributes to the presence of the virtual environment, stereo rendering have been disabled in the video."

3D and the iPhone

Last year, when Hasbro announced the My3D, a 3D device, I was intrigued: how could a US list price of $35 support the cost of a micro-display, optics, control electronics and a decent enclosure? The answer, of course, is that it doesn't. The device does not have its own display - it houses an iPhone with specialized applications that present 3D images in side-by-side format; it does not have a motion tracker - it uses the iPhone's. It does not have drive electronics - for the same reason.

When the My3D came on the market in April, I immediately went out and got one. After all, buying gadgets is one of the few perks of my job. I downloaded the apps from iTunes, installed them on an iPhone 3GS and put the device to a harsh, un-scientific but brutally accurate evaluation: the teenager test. This is a test where I give a device to a bunch of teenagers and tell them "it's all ready to use; it's free of charge for you; just use it and tell me what you think". As expected, the first five minutes of the test were filled with 'wow - this is cool!' but from the sixth minute till this date, the My3D has never been used again. In contrast, the Motorola Xoom tablet, in spite of surprisingly poor video playback performance, has passed the teenager test with flying colors and is constantly being used.

Before discussing the upside, here are some of the things I don't like about the My3D:

• The iPhone 3GS screen, when used in side-by-side mode and magnified through the My3D eyepieces, is nothing to write home about. The resolution is low and the fill ratio of the pixels 
• When using the My3D, your hands need to hold the device.
• User interface is clumsy - holes are provided to insert the thumbs through the My3D enclosure and touch the iPhone screen

But there is an upside. The concept of using the phone (or tablet) to generate the graphics, provide the communication link, app store and perhaps even the motion sensor, is certainly valid. With phone and tablet CPUs becoming ever stronger (the Xoom has a dual-core NVIDIA Tegra) and with goggles supporting side-by-side 3D, you can get pretty good 3D experience without a PC.

The HMD and the fMRI machine

A couple of weeks ago, I returned from the annual meeting of the Vision Sciences Society in Naples, Florida. VSS is an organization of scientists who are interested in the functional aspects of vision. The VSS conference brings together in scientists from a broad range of disciplines including visual psychophysics, neuroscience, computational vision and cognitive psychology.

There is something nice about coming back to a conference year after year in the same location. You already know where to stay, where to eat or not to eat and are free to focus on enjoying the academics, the exhibition and the Florida sun.

Because VSS is primarily focused on academics, the show floor where my company and others exhibited was not often crowded, leaving exhibitors plenty of time to gleam knowledge from the poster sessions as well as chat with other vendors. The VSS show is dominated by eye tracking companies - most of which we've had the pleasure to work with - but HMDs such as the zSight and other (heavier and lower contrast) HMDs were also on display.

The academic crowd is this show is very diverse in their virtual reality experience. It was great meeting existing customers that we've worked with over the years, but just as fun demonstrating the HMD to researchers admitting that they have heard of head mounted displays but never got around to trying one. 

A key part of many vision research projects is to generate visual stimuli. HMDs offer a superb vehicle for doing so, for several reasons:

Arrington Research Eye tracker inside a Sensics zSight

• The ability to offer stereoscopic stimuli, thus also adding depth perception to studies about motion, color, recognition and more.
• A good HMD is often immersive, meaning that it can block external distractions and make sure that the subject is focused on the stimuli inside the HMD.
• The ability to integrate eye tracking inside the HMD.
• The ability to engage peripheral vision, especially when using an HMD with a panoramic field of view

I had the opportunity to browse the poster session focusing on research involving fMRI. Functional MRI studies put a subject inside an MRI machine and generate certain visual stimuli. The brain function is then measured and analyzed. Some of these projects give scientific confirmation to seemingly obvious results and others open doors to deeper understanding of how people see. For instance, I saw a poster that confirmed that the brain produces different neural responses when presented with pictures of familiar people as opposed to non-familiar people. To me this falls more into the obvious category: if a picture is instantly recognized as familiar, doesn't recognition itself mean that the brain has a different neural response to it?

Some companies on the show such as Cambridge Vision Systems devise clever ways to inject visual stimuli into such a challenging magnetic environment such as an MRI, and even have the ability to do fMRI eye tracking. Really cool stuff. Over the years, we got asked several times of an HMD could be used inside an MRI machine either for research or as distraction to prevent claustrophobia. Unfortunately, our HMDs are not MRI compatible today, though I'm open to ideas on how to make one.

During a typical year, I meet different people that have different uses for HMDs: from training soldiers inside tanks to architecture and industrial design to academic research. The possibilities for using HMDs seem endless, and many people I meet have new and sometimes crazy ideas on what to do (such as the person suggesting HMDs as a way to revive the art of playing an ancient banjo). Regardless, I look forward to returning to VSS next year and learn even more.

Will Microsoft acquire Lumus Optical?

There is a rumor going around that Microsoft is negotiating the acquisition of Lumus Optical. Lumus is a startup company that developed a clever see-through eyewear. The company originally targeted both the professional (e.g. defense) market as well as the consumer market, though it reportedly sold off its defense business to focus on the consumer side. The public specifications of its product show a low-resolution (640x480), narrow field of view (27 degrees) display, but one that could be made to resemble a pair of normal glasses.

Publicly, Lumus has been keeping a very low profile. The most recent press release or news article on its Web site, for instance, is dated some two years ago. If it not for their promotion of the upcoming SID show, an outsider could conclude that Lumus is out of business.

Of course, it may very well be that this rumor is untrue or that the acquisition talks will not come through, but it's a good opportunity to analyze where such an acquisition could fit.

One place where a see-through display could fit is with the Microsoft mobile division as part of the Windows phone initiatives. A classic use case of a Bluetooth-equipped see-through display with a phone is showing the caller ID inside the glasses so that the user does not have to pull out or look down at the phone. With phones become more and more sophisticated, dynamic driving directions have also been discussed.

How much would people be willing to pay for such glasses and what would they cost to make? This has been a classic chicken-and-egg problem for goggle manufacturers, where price depends on quantity and quantity depends on price. If Microsoft were to bring this product to market, they could all but guarantee a large enough quantity to make these economical.

Another possible home for a see-through display ix the XBOX group. If motion sensing (e.g. Wii, Kinect) was the last big thing in gaming, 3D, immersion and freedom of motion could be the next. I find this a bit less likely that mobile because gaming often requires immersion and 27 degrees of FOV, at least for the current product, is far from being immersive.

One way or the other, such an acquisition would be a nice shot in the arm for all the goggle vendors trying to find their way to a mainstream market.

If Microsoft ends up acquiring Lumus, you've heard it first here! If they don't, I just might change the title of this post to "Should Microsoft acquire Lumus".

The 11.5M pixel HMD

A few weeks ago, we had the opportunity to ship to a customer one of our high-end HMDs, with a total of 11.7M pixels. Each eye in this HMD has 5.76 million pixels, about 2.7 times as much as an HD1080 signal.
The piSight 166-43 has a horizontal field of view of 164 degrees, which approaches the human visual field.

Building, testing and shipping such a high-end HMD is always a cause for celebration here. We sell lots of 120-degree and 60-degree products every year, but just a handful of the super high-end ones.

This particular piSight has 12 micro-dispalys per eye. The top photo on the right shows the complete HMD. The photo below it shows the eye modules: the left side of the image shows our patented tiled optics in front of the 12 displays, while the right side shows the screen configuration with the array removed.

It takes one or two computers to drive the HMD to its full potential, and we typically recommend driving it with four HD1080P or 1920x1200 signals which feed into our video processing electronics.

The field of view is just spectacular, as well as the level of detail across it. It is simply fun to see how immersive such product is.

Side by Side 3D and HMDs

A key advantage of HMDs over 3D televisions is the ability to get true depth perception in a surround video setting. However, depth perception inside an HMD requires different images for the left and right eyes. There are multiple ways to provide these separate images:

• Use dual video inputs, thereby providing a completely separate video signal to each eye
• Time-based multiplexing. Techniques such as frame sequential combine two separate video signals into one signal by alternating the left and right images in successive frames.
• Side by side or top/bottom multiplexing. This method allocated half of the image to the left eye and the other half of the image to the right eye.

The advantage of dual video inputs is that it provides the maximum resolution for each image and the maximum frame rate for each eye. The disadvantage of dual video inputs is that it requires separate video outputs and cables from the device generating the content.

Time-based multiplexing preserves the full resolution per each image, but reduces the frame rate by half. For example, if the signal is presented at 60 Hz, each eye is receiving just 30 Hz updates. This becomes an issue with accurately presenting fast-moving images, or images that need to rapidly change as a result of tracking information or user movement.

Side-by-side and top/bottom multiplexing provide full-rate updates to each eye, but reduces the resolution presented to each eye because only half the available pixels in each frame are used for each eye. Many 3D broadcasts, such as ESPN, chose to provide side-by-side 3D which saves the need to allocate extra transmission bandwidth and is more suitable to fast-paced sports action relative to time-based multiplexing techniques. Of the multiplexing techniques, I think side by side 3D is best for HMDs.

Side by side 3D has some advantages over the full dual-port mode when using a wireless video link. Sending two completely separate images requires a dual-stream video link or two separate transmitters. With side-by-side, true interactive 3D can be economically achieved with a single wireless video link.

Not all HMDs provide depth perception. Some lower-end modules are essentially bi-ocular devices where both eyes are presented with the same image. Interestingly, most professional HMDs do not provide support for side-by-side format. Fortunately for users, both the xSight and zSight professional HMDs do provide such support. Such support also allows you to tap into the wealth of entertaining 3D videos on YouTube 3D, view video directly from a 3D camera and more.

How will HMDs be impacted by the growth in 3D TVs?

Several million 3D TV sets shipped in 2010, with some reports claiming shipments of over 6 million sets. The industry is clearly aiming for higher volume in 2011. Is this 3D TV growth good or bad for the HMD market?

Very good, in my opinion.

3D TV are useless without 3D content, and this content is immediately useful inside head-mounted displays. Just like surround sound at home did not inhibit portable music players - the Walkman or the iPod - 3D TVs don't inhibit HMD sales.

With 3D TVs, an increasing number of users are exposed to the power of 3D and are thus stimulated to come up with new uses.

3D TVs will drive 3D games, but HMDs offer 3D gaming in a dynamic, 360 degree surround video environment that a TV cannot offer. HMDs can be portable and battery operated, whereas TVs are stationary and power-hungry. You can run around with an HMD on your head (especially with a wireless video link) but no one would consider running inside a room carrying a TV.

Do advances in fuel economy of cars impact air travel? Not really. Similarly, the use cases for TV and HMDs are different in other aspects as well. TVs are better suited to group viewing, whereas HMDs offer privacy. TVs provide all users with the same viewpoint, whereas embedded trackers inside HMDs (such as the zSight) can offer dynamic and individual viewpoints. Some exciting 3D content comes from animated features. How long before such content can be streamed and changed depending on head position? This would be excellent news for HMDs that can provide great immersion.

You can get some great 3D TVs for $2500. When will you be able to get great HMDs for $2500? Perhaps as soon as there is demand for 6 million HMDs a year.

New video demo of xSight head-mounted display

The xSight HMD is difficult to experience on YouTube, but here's a movie anyway. The reason it's difficult to experience is that two of its most important attributes: field of view and weight are best experienced in person. The horizontal field of view of an xSight is 120 degrees. If you sat 8 feet from a TV and wanted that TV to provide you with 120 degrees field of view, the width of the TV needs to be about 28 feet. That's much more immersive than a front row seat in most movie theaters.

Weight is also hard to experience in a movie. The weight of the xSight is about 350 grams, so similar to a Coca Cola can. The weight of other HMDs is sometimes 1.5 kg, like a 6-pack of beer on your head. Quite a difference!

New video of zSight HMD with iMove

A new video is available showing the integration of the Sensics zSight HMD with the iMove GeoView software package. iMove allows truly spherical recording of video, which is useful for many applications such as surveillance, tourism, mission rehearsal and more. A vehicle is outfitted with a pole carrying several cameras. Video is simultaneously recorded from all cameras. iMove's software then stitches the individual video streams and prepares a file which can then be played back using a standard desktop or notebook PC. The player receives head tracking information via a simple USB interface from the zSight HMD and changes the viewing direction based on the head movements. The result is like being inside the car.

Various cool videos are available: riding in DC, sailing in a canal in Amsterdam and many others. Several companies also offer services where they can create such immersive videos for desired locations.

Since there is so much video data being recorded at once, iMove decided (at least for this recording) to record at about 8 frames per second. This is more choppy than normal video, but creates a nice balance between the amount of storage required and the smoothness of the image.

Integration is also available for the xSight which provides an even wider field of view and even higher resolution.

New optics make an excellent product even better

The hard work was worth it. After many months of engineering and testing, my company is now shipping new and improved optics for the zSight SXGA HMD, in both its commercial and helmet-mounted versions.
HMD design is a study in trade-offs, needing to balance weight, resolution, cost, field of view, features and more. Optical design for HMDs is no different. Key attributes to be balanced include:

• Field of view. Wider is better for an improved experience. Low-end HMDs  Low-end HMDs offer 30 or 35 degrees of diagonal field of view. Professional HMDs offer wider solutions, some even greater than 100 degrees.
• Clarity. it's not difficult to design an optical system that provides image clarity at the center of the image, but reaching clarity at the edges or corners is not always an easy task.
• Geometric distortion. Minimizing pincushion or barrel distortion is very desirable. Distortion is often expressed in percent as a function of radial distance from the center of the lens, showing how far a point is away from its perfect theoretical position. Some professional HMDs have distortions greater than 10% (ouch!), but 5% or better is desirable.
• Weight. After all, these optical systems are designed to be worn on the head. Every gram or ounce counts. 
• Material used. There are hundreds of optical-quality glass types, but only a few optical-quality acrylic/plastic materials that are suitable for optics design in the visible spectrum. Glass offers greater flexibility in the design (much greater selection of refractive index) but is heavier, more expensive to manufacture, and introduces safety concerns in some applications if there is concern about glass elements close to the eye.
• Chromatic aberration. Similar to light passing through a prism, an optical system might impact different colors/wavelengths differently. This might cause blur and color separation.
• Eye box. Unlike a telescope lens, HMD optics are used when moving the head. The eye box specifies how much the user's eye can stray away left/right/up/down from the location of the center of the lens without losing the image or suffering significant degradation in image quality. Bigger eye box is better.
• Eye relief. This specifies how far can the user's eyes be away from the lens. Being too close is a source of discomfort - your eyelashes could brush up against the optics. A larger eye relief allows those with glasses to wear them.
• Diopter adjustment. Some designs allow for compensating for glasses - just like the eyepiece of binoculars allow for some adjustment so that those with glasses can, but don't have to, wear them. This is a plus.
• Physical attributes. Since this optics design needs to fit inside an HMD, it cannot be too long. Eyepieces cannot be too wide either because two eyepieces need to be next to each other with a minimum distance of about 52 mm to accommodate the desired ranged of interpupillary distance (IPD).
• Light transmission. Some designs transfer only 25% of the light from the source. Keeping most of light allows higher brightness and more energy-efficient HMDs.

The process of optical design involves specifying the design targets but then fighting through the trade-offs. Should we use glass? How much eye relief is good enough? Are you willing to sacrifice clarity for geometrical distortion? on, and on.

Many months and thousands of dollars into it, I think we nailed it. The optics are very lightweight (about 10 grams each), offer 60 degree field of view that is crisp to the edges. Low distortion. Big eye box and nice eye relief. There is no perfect solution because of the trade-offs but I think we hit a very good one.

I like looking at great design: an Italian wine glass, a Frank Lloyd-Wright house, an iPad. With these new optics, I liked looking through a great design.

The optics are now available as part of the zSight, a 60-degree OLED SXGA HMD with integrated tracker, audio and stereo sound. Try it if you get a chance.

Eyelids as shutter glasses

Is it real or just French humor?

If not fake, an amazing  demo illustrating how you could conceivably get alternating-frame/frame sequential 3D without shutter glasses by modulating the eyelids.

Probably belongs to the 'don't try this at home' category?

How long can you use this without damaging your eyes?

Would you be brave enough to try it on yourself?

Can nostrils be modulated for stereo smell?

I still prefer head mounted displays for true stereo, motion tracking and the full immersive 3D experience.

Kopin acquires FDD. Would the market benefit if eMagin was also acquired?

Kopin, a publicly-traded maker of transmissive micro displays acquired Forth Dimension Displays (FDD), which makes reflective micro-displays.

Reflective micro-displays have been used for HMD applications, but the optical system and constant light that they require provide lower contrast relative to OLED alternatives, as well as makes for a more complicated, larger and heavier optical system. Reflective displays could still be an excellent choice for high-end projection applications that use very bright light sources, but not so much for HMDs. For HMDs, transmissive or self-emitting displays seem to be the better, easier solution. Kopin makes transmissive displays. eMagin makes self-emitting displays.

How would the market benefit if a large company acquired eMagin or if there was another way to invest larger amounts of money into eMagin?

For HMD sales to really take off, display components of reasonably high resolution need to be inexpensive. You can describe eMagin products in many good ways, but inexpensive is not one of them.

One would think that eMagin's prices are driven by a limited production capacity. If their production capacity was hypothetically unlimited, eMagin could select a different - and lower cost - point on the price/quantity curve to drive market adoption and maximize their profits. If their production is limited, it would not make sense to generate more demand than they can produce. The answer in this case is increasing production capacity, which has now been announced as something eMagin is doing according to their press release. However, eMagin's cash reserves allow for only a certain level of investment and they would need to generate more cash organically to increase it. What if someone could finance two, five or ten new lines instead of just one? Would that drive quantities of quality HMD products? Would that provide a boost for the industry?