Virtual reality and head-mounted displays

CES and the promise of falling micro-display prices

2012-01-23T09:34:00.003-05:00

One of our takeaway from the recent Consumer Electronics Show was that micro-display prices will be dropping nicely in 2012, which is good news for anyone clamoring for high-quality HMDs at lower costs. As I wrote previously, the cost of micro-displays has a big impact on the final cost of HMDs, so a substantial reduction if costs can allow companies to reduce prices on HMDs in the hope of reaching a broader market.

At CES, both on and off the show floor we saw evidence of micro-display prices coming down, both with new companies offering micro-display solutions, existing companies presenting 720p or higher solutions at affordable costs, or very large companies professing their ability to use large-screen technology to create economical micro-displays.

I believe all of these trends would also create downward price pressure on the established micro-display providers like Kopin/4DD and eMagin, unless they want to focus on higher-end military-type solutions.

At the same time, companies like Sensics are going beyond the traditional "an HMD is a microdisplay plus a lens" paradigm to offer smarter goggles and extra features. This combination of reduced component costs and valuable capabilities should make for a fascinating 2012.

Mindless goggles not going away anytime soon

2012-01-20T23:51:00.002-05:00

A reader writes:

"Hi,
You mentioned that goggles of the past were mindless monitors.

I would love mindless goggles provided that they had head tracking and a decent resolution and viewing angle, a bit like those uber expensive sensics ones. :-)

Will you one day release such a product? I am talking flight sim, racing sim, first person shooters. Sony seems to be the only one with something close at the moment but the field of view does not cover the full 180 degrees. If it did, the resultion would need to improve a lot."

Thank you for the input.

I think mindless goggles are not going away anytime soon, just like flip phones are still around (though their primary market these days are young kids and senior adults). Mindless goggles will most likely continue to be cheaper and offer somewhat of a limited experience. A starter kit, before you upgrade to SmartGoggles?

For one thing, SmartGoggles are not fully commercialized quite yet. Based on the wave of enthusiasm and support, I am sure they will be quite soon.

My company is happy to be mentioned in the same sentence as SONY, and we are glad to be impacting the discussion. Imagine how cool it would be if SONY decided to license SmartGoggles technology and make their media viewer a smart goggle - one with full head and hand tracking, wider field of view, higher resolution, untethered, and an Android processor on board. You could immerse yourself in PlayStation games, or even play them in a limitless and omni-directional tracking area. Or, you could connect to a SONY tablet or XPERIA or VITA and add intelligence, immersion and 3D to all the nice features that already exist on these devices.

One of my favorite CES exhibits

2012-01-17T09:21:00.002-05:00

Back from CES, where my company launched the SmartGoggles to substantial press, partner and customer excitement (see the verge, for instance)

One of my favorite exhibits at CES was the Smart Window by Samsung. It is a large transparent window that allows graphical overlays on top of the see-through pane.

Historically, see-through technology has been pretty much limited to heads-up displays and see-through goggles, all of which are fairly small in size. The Smart Window makes see-through panels that are much larger. If you couple this with some ability to modulate the intensity of light coming through the window (e.g. electronic shades), you can truly see the window of the future.

The micro/macro comparison in displays is interesting. For instance, there is a lot of effort on behalf of large companies to create ever-larger high-definition OLED televisions (Samsung and LG both introduced stunning 55" OLED TVs at the show), but all that innovation has not yet made it into micro-displays. Other than SONY, perhaps the really big companies are not yet convinced there is a very large market for OLED micro-displays. Such commitments would help drive down the cost for micro-displays, and make high-performance goggles substantially more affordable.

Expect 2012 to be a transformational year for HMDs

2011-12-31T23:51:00.000-05:00

2011 was a 'business as usual' year as HMDs go. Sure, there were some new products and product improvements, but nothing earth shattering. At the same time, improvements in displays, motion sensing, augmented reality and processing power are coming together to enable 2012 to be truly transformational.

Sure, some nice progress was made in 2011 in HMDs:

Higher resolution OLEDs became part of a new generation of lighter, brighter HMDs such as the zSight.
Low-latency wireless video is moving into mainstream. I believe it will become a common accessory to HMDs, much like head trackers.
In general, companies like Sensics continued to improve their optics designs, packaging, level of integration and features of professional HMDs.

Having said that, I believe that what's truly going to impact HMDs - the professional models - and VR goggles - the consumer models in 2012 were recent improvements made outside the direct HMD realm:

Lower-cost micro-displays of 'good enough' resolution are becoming available. Once 720P displays are available for ~50$, the economics of goggles change, because displays have historically been a key cost driver for the HMD/goggle cost. What's the difference between a $10K HMD and a $15K HMD? $5K, of course, but otherwise there is no big difference in market penetration. It's hard to see why a $10K price would suddenly cause cause skyrocketing quantities whereas a $15K HMD did not. However, the difference between a $1K goggle and a $6K HMD, or better yet a $500 goggle to a $6K HMD would be huge as the $500 product would suddenly allow an entirely new group of users and entirely new class of goggle applications.
Motion tracking is improving - both inside and outside the HMD. Driven by new generations of cell phones, tablets and game controllers, embedded motion trackers with 6 degrees of freedom (X/Y/Z, yaw/pitch/roll) are becoming good enough and inexpensive enough to make it into practically every product. Not only does this this make them an automatic add-on to products, but their increased quantity in the market opens up new ideas and new development techniques to use them. Outside the HMD, the success of the Wii and the Kinect made it obvious how powerful gesture-based and natural interaction are, and these developments will soon make it into goggles, such as the new SmartGoggles offering.
Just like micro-displays are experiencing their version of Moore's law, embedded CPUs are continuing their march. New phones and tablets these days could have 1.5 GHz dual-core processors with embedded graphics and 3D accelerators. This is a very powerful computing platform. Because they are embedded in phones and tablets, these processors have to be lower-cost, power-efficient and small - all a perfect fit to embedding them into or alongside goggles. At the same time, having an open operating system like Android allows support of new hardware platforms as well as a nearly-endless supply of apps and content to consume.
The improvements in processing power as well as cost and resolution of embedded cameras have given a huge boost to augmented reality applications. Driven by phones and tablets, both the applications and the knowledge gathered while building them will become very useful for goggles.

All taken together, a new generation of goggles will appear. If the typical goggle in 2010 was a lens, a micro-display and a video cable, it will evolve to be much more in 2012/2013. Combine on-board processing, embedded motion tracking for head, hands and others, augmented reality power and lower-cost micro-displays, and the mindless HMD will now morph into something between a fully-fledged independent computing and user interface platform to at least a highly-sophisticated display and virtual reality co-processor for phones, tablets, consoles or PCs.

Sure, there will still be a market for traditional HMDs just like there is still a market for flip phones. Sometimes, users don't need the sophistication of the Smart Phone, or don't want to pay for the data plan or have some ruggedization requirement that can only be satisfied by a flip-phone. Similarly, a special defense packaging, an ultra-low cost or some other requirement will continue to keep traditional HMDs in the market for years. However, I believe true growth will come from the smarter goggles that will take advantage of the above technologies to create a more complete and compelling user experience.

Happy Near Year!

The SmartGoggle

2011-12-22T21:55:00.003-05:00

Today, my company introduced the SmartGoggle, the solution for the common virtual reality goggle.
The issue with today's goggles is obvious: they are mindless; just a monitor on your head. Just like a monitor has a video input and will display almost anything you pipe into that video input, a traditional goggle will display that signal in front of your eyes. If you provide two signals, or side-by-side video, it can do so in stereo as well.

But a traditional goggle, like a monitor, requires an external video source. It can be a computer, in which case you can do something interactive. It can be your iPod, so you can use the goggle as a media viewer. Useful, but boring.

While all of this is going on, tablets and smart phones have become increasingly powerful in their computing and graphics capabilities. So, as we were thinking about how to make goggles better, it became obvious that putting some good processing power inside the goggles could do a lot of good. For one thing, you could run applications on the goggles and not have to carry around an external computer with you. The goggles, and these applications, could go with you anywhere, which is cool. These could be connected applications - streaming something from the Web, or these could be local applications such as those using an on-board camera to drive an augmented reality app. Yes, you should still be able to use an external video source to drive your goggle, but now you can also drive it 'from the inside'. This is a bit like the new crop of smart TVs that are becoming very popular. You could connect them to your cable provider or DVD player, but they can also stream a Netflix movie or surf the web using an on-board processor. So far, so good.

The problem of the user interaction still remains. Sure, you can put a head tracker so that the view can change as you rotate your head, but this is typically not enough for true interaction with the application. You can use an external device - a phone, joystick or even a Kinect - as input, but these approaches have limitations: you have to carry them with you; you have to stand in front of a sensor; you are limited in your tracking area; good, but not good enough for goggles.

We then started thinking: what if we put a camera that can track your hands and make that information - both in raw form as well as after gestural analysis - available to the application running on the goggle. That might be cool, because the hands go with you everywhere and because cameras on your head can often see your hands regardless of where you are and which direction you are looking at. Better yet, if we put an array of cameras on the head, we can get depth perception of the hand location as well as get a really wide tracking area. We call this 'first person hand tracking'.

Last, we realized that many goggles are essentially the same on the inside. Sensics, for instance, makes a commercial goggle and then repackages it into a different enclosure for training and simulation applications. Anyone wanting to build a goggle will need to cover several areas: driving displays; head tracking; video processing and more. Given this, it made sense to design a module that essentially encapsulates all these functions and allows goggle developers to focus on the design/styling aspects of the goggles rather than on building everything from scratch time and again.

We've been working on this for a while and are excited with the progress and the initial feedback we are receiving. If you combine these three innovations: on-board Android machine, real-time hand tracking from a first-person perspective, and a 'system on a module' approach for encapsulating most of what's needed to build a goggle, we think you get something. A SmartGoggle.

Remember when the iPhone came out and suddenly people started realizing that it's not just a phone, but much more than that? We think SmartGoggles can be to mindless goggles what smart phones are to flip phones. A major step forward, which we are very excited to take today.

HMDs are mindless - a monitor on your head

2011-12-06T22:12:00.000-05:00

My company makes VR goggles (professionally called: HMDs), so when I say that HMDs are mindless (or stupid or unintelligent or dumb), it is with a somewhat heavy heart.

After all, HMDs are sophisticated devices. To build one, you need to understand optics, electronics, ergonomics, mechanical design and system engineering. You need to balance features with field of view with weight and comfort. HMDs are indeed sophisticated, but nevertheless mindless.

I call HMDs mindless because they are not much more than a fancy monitor on your head. Like a monitor, you need to connect an external video source: a computer, MP4 player, phone or tablet, to provide a signal to be viewed. If the video signal is provided by a cable, it limits your mobility. If it provided via a wireless video link, it cuts the cable but still provides limits on your distance from the video source.

The passive nature of HMDs is not just because it's a 'monitor on your head'. Unless you are content with using the HMD as a media viewer, you will want to interact somehow with the content and with the HMD. In today's products, you might have some push buttons on the HMD and you usually have a head orientation tracker than can let the application know where you are looking. This is a decent start, but most of the user interface experience - selecting menus, interacting with content, moving 3D objects around, still relies on external devices: a joystick, a mouse, a data glove and others. By the way, many wireless video links don't have the ability to send head tracking information back to the video source.

Sure, this is fine for some applications. Military training and simulation applications are sometimes OK with having the soldier being trained carry a computer on his back. Stationary applications (a tank trainer) or those relying on lots of peripheral equipment (such as an academic research project measuring brain activity) can greatly benefit from today's HMDs, but is this enough for widespread use?

The promise of HMDs was to be able to take 3D content anywhere and interact with it in a useful way. Though more complex, it is just like how the iPod allowed us to take our music library with us anywhere and sufficiently interact with it to be useful. I don't think today's HMDs are fulfilling this promise for a broad-enough market.

Something needs to be done about it.

Gentex acquires Intersense

2011-11-25T20:11:00.001-05:00

A couple of days ago, Gentex Corporate announced that it had acquired Intersense. Gentex makes various military products including helmets, eye protection devices, aluminized fabric and helmet-mounted displays. Intersense makes various inertial trackers.

I've had the pleasure of working with Intersense over several years, where we partnered on delivering virtual reality solutions for academic and defense markets. To many of our customers, Intersense has always been considered the 'gold standard' of tracking, which is why I find this acquisition disappointing.

The financial details of the acquisition were not published, so perhaps this is a big win for the shareholders of Intersense, but I feel the company will now take an even stronger defense focus and will thus miss the bigger consumer product opportunity.

It is ironic that the Intersense acquisition happened just a week after the Invensense IPO (INVN), which values Invensense at some $800M.

Founded in 1996, Intersense makes professional motion sensing products. Motion sensing is becoming more and more commonplace - just look at the Wii or the iPhone. Surely, Intersense had plenty of expertise in sensors and signal processing to offer a compelling low-cost product that builds off Intersense's professional reputation but offers a compelling price point.

As motion tracking became more common, lower-cost solutions started to appear. For instance, Sensics offers an integrated three degree of freedom (yaw/pitch/roll) head tracker inside the zSight professional HMD. Though the performance of the embedded tracker is not as good an Intersense IC-3, for instance, we found that for many customers it was good enough and saved them the need to pay a couple of thousand dollars for an external tracker. Intersense could have probably easily offered this lower-cost tracker, but they did not. Missed opportunity, I think, both for Intersense and for the market.

Good luck to the excellent team at Intersense, in whatever market you choose to serve!

Towards socially-acceptable goggles

2011-11-18T21:13:00.001-05:00

Holiday season is fast approaching in the US and retailers are starting to offer attractive shopping discounts. A few years ago, my wife asked me to go to a home goods store during holiday season and buy some wine glasses. It was a cold day, and my cell phone was tucked inside my jacket while I was using an earphone in the store to speaking with my wife and decide together on which wine glass set to buy. At that time, mobile earphones were such an uncommon sight that several older ladies approached me - as I had appeared to be speaking to myself - to inquire if everything was OK.

How times have changed. Using a cell phone earpiece or Bluetooth device in a shopping mall - a complete oddity just a few years back - is now completely socially acceptable.

When would it and what would it take to have wearing virtual reality goggles become socially acceptable? Today, when I meet someone at a coffee shop and take out the goggles, people start coming over from adjacent tables to take a look. That's great if you want to sell goggles, but what would need to happen so that you can wear goggles in public unnoticed?

Clearly, the definition of what is 'socially acceptable' changes over time. Purple hair color? Diamond studs for man? Body piercing? Bluetooth headsets? All of these used to be outside the norm and now hardly worth a second look.

What were the motivations of the first people that stepped out of the 'socially acceptably' domain? Using a headset is probably about utility: it is more convenient, perhaps healthier. I don't think too many people used a headset because they felt it was overly aesthetic. On the flip side, there was little actual utility in dying your hair purple. A diamond stud doesn't make you hear better, but some think it makes you look better or draw more attention.

In the case of virtual reality goggles, we'll probably have to go the utility route. If you just want a pair of cool-looking sunglasses, just go to the store and buy cool-looking sunglasses!

What will goggles need to do so that they would be widely worn? If a Bluetooth headset is an audio extension of your phone, what will the visual extension of the phone be? Is it just about reading text messages or looking at driving directions without starting at the phone? What is the killer app? Write back to let me know what you think.

The Stark HUD 2020 Augmented Reality goggles

2011-11-07T20:44:00.002-05:00

A few weeks ago, I wrote about the Stark HUD augmented reality goggles. These are not real AR goggles, but rather a clever promotion for Iron Man 2. Nevertheless, the video is compelling and worth reflecting on what it is so.

1. The goggles are visually attractive and appear to be 'socially acceptable' to wear. Though the definition of socially acceptable is fluid - just a few years ago it seemed odd to walk around with a Bluetooth headset - the closer goggles look like glasses, the better.

2. Information is context sensitive. The idea of context-aware computing has been around for quite some time, and there are patent filings on it dating at least 10 years ago. However, the availability of location-based services, and development in applications that know more and more about the user (Apple's Siri is a great new example), make the goggles useful much more than just being used as a media viewer.

3. The field of view appears wide, and resolution is good enough to read.

4. There is plenty of natural interaction with the device. The video demonstrates interaction using hands and voice commands.

5. No battery or control box. Science fiction, maybe, but still cool.

Will we really need to wait until 2020 to get them?

Steve Jobs and the HMD

2011-11-03T12:19:00.001-04:00

The Web is buzzing about iTV, Apple's HDTV and what it could look like and do. Though Apple has filed some patents related to HMDs, it does not appear that Steve Jobs left us with an HMD design.

What would an HMD designed by Steve Jobs look like? What would it do?

As a business person, I want to build successful companies, and while success can be measured in many financial metrics, one aspect of a successful company would be a company that creates an emotional attachment with its customers.

My little personal story with Apple began many years ago, when I was about 12 years old and living in Boston. I was one of these kids that stood at the corner Radio Shack stores trying to do fun stuff with their TRS-80 computers. The personal computer revolution just started. IBM had not introduced the PC yet, and I wanted a computer of my own. My family lived in an apartment building which, for a 12-year old, was fertile ground do earn money babysitting. Lots of babysitting. Once I decided I wanted an Apple (other candidates were the TRS-80, the CompuColor and perhaps some others that I don't remember), I saved every babysitting penny towards the $1200 goal of getting an Apple II. My parents opened a bank account for me and I would stand in line to deposit $12 here, $30 there, until I reached the magic number and get my Apple II. Its serial number had 4 digits and it was that older model that had a reset key on the keyboard that, while spring loaded, was too easy to press and wipe out everything. I spent hours and hours programming, playing and hacking on that computer. Many years later, my mother donated it to a local school - it was the right thing to do but I miss my old Apple II. I can't say I miss any of the numerous PCs I've went through since.

What was so good about that Apple II? I loved its design. It was open so that you could plug cards inside. It was simple to use and "it just worked" year after year after year. That allowed so many people to write cool software and so on in a vitreous cycle.

What can we learn from Apple about HMDs?

Design matters. Make people want to use or - in the case of HMDs - wear your product
Keep it simple. Plug and play. No drivers to install. No configuration to go through. "It just works".
Open it up so that value could be added outside your company.
Price matters, but you don't have to be the lowest-cost product to succeed
Make it simple to use content that you already own
Humanize the product, if possible.

Realistic oil&gas plant simulation with an HMD

2011-10-29T09:57:00.000-04:00

Our partner VRContext released a video demonstrating walking inside a sophisticated plant using an XBOX 360 controller and a zSight HMD.

What I find particularly appealing about this video is the level of detail. Walking through the plant (while hearing your footsteps, no less!), you say layers and layers of pipes, pumps, floors, etc. This is possible through integration with real engineering databases. Since the plant is already designed, why not use the same databases to ensure safer, lower-cost operation?
I can certainly see why learning about this plant in a comfortable, virtual setting, will prepare the operator for true troubleshooting and maintenance when needed.

Cloud watching with an HMD

2011-09-20T23:16:00.002-04:00

Below is a comic strip, not an instruction manual, but I guess it is possible to try this at home. Some notes after the comic strip.

Normally, when placing stereoscopic cameras or rendering a virtual image, it is important to try and match the actual distance between the eyes. That's why 3D camera rigs try to minimize the distance between the cameras, which are pretty large:

Having said that, there is no depth perception in such a large distance as looking at the clouds, so substantially increasing the distance between the eyes might create some 3D effect.

Preview to thoughts about Augmented Reality goggles

2011-09-17T22:36:00.000-04:00

One of the followers of this blog asked what I think about augmented reality goggles. Great question, worth discussing in more detail. In short, I don't think today's solutions are there yet, partially because of technical limitations and partially because of the currently-available applications. More on that soon. In the meantime, this video shows what good augmented reality would be like.

The VR goggle as a media viewer

2011-09-15T22:32:00.001-04:00

Several VR goggles (such as the Carl Zeiss Cinemizer) present themselves as media viewers, which translates to a really cool way to watch movies from your iPod.

Is there really a significant market for VR media viewers?

Have you ever gone to a movie theater to watch a movie alone? Sure, when I travel, I sometimes go to restaurants alone, but I don't remember going to a movie alone. Isn't watching a movie alone what VR media players are designed for?

I tried researching how many people go to see movies alone and could not find the answer. There are tons of movie stats from the Motion Picture Association of America, the movies alone is not one of them. Googling this brings up pages titled "Do you ever go to the movies along and feel weird for doing it?"

People watch movies alone at home, but then they can use a TV and don't need a low-resolution set of goggles to do so (Cinemizer had 640x480 pixels, though perhaps has gone up since). Watching movies on a plane is a potential use, but wouldn't an iPad or other tablet be a better experience? Privacy is a plus when using goggles, but would you really spend $500 on a pair of goggles only to use them on a plane to watch movies that you are uncomfortable having your neighbor see?

Sure, I get using an iPod, not to mention that since as of Dec 2010 "only" 297 million units were sold, there is clearly a big market. You can do other things while using an iPod, but you can't do much when watching a movie on a set of VR goggles.

So, I don't get it. Any ideas?

A 3D look at the Vuzix financial reports

2011-09-10T23:56:00.000-04:00

In the spirit of the previous financially-related post, I decided to take a look at the Vuzix (VZX.V) 2nd quarter SEC 10-Q filing and see if there is anything interesting to share from a market direction standpoint.

Some disclaimers: I am not a financial adviser and I don't make stock recommendations. You should buy or sell Vuzix stock based on what you read in this post. It is just my opinion as an individual fairly versed in virtual reality, not a Wall Street person. Also, I don't own Vuzix shares so I don't stand to gain or lose if their share price changes. This analysis is fully based just on the public information contained in the 10-Q.

Having said that, let's take a look at some of the interesting nuggets in the report, in no particular order:

VR Market-focused:

Sales of what the company calls consumer eye wear products are in decline. It sold $1M of consumer products in the first six months of the year compared with $2.2M in the first six months of 2010.
Sales of defense products, such as eyepieces, are on the rise.
The company notes that demand for their consumer eye wear has declined for two reasons. The first is that they pre-announced a higher-performance product which they did not deliver yet, causing customers to slow down their purchasing. The second is that the iPad and other high-resolution tablets are used as media viewers instead of goggles. Personally, I don't fully understand the market need for goggles as video viewers: Resolution on the low-end products is fairly poor (I don't think that Vuzix has an HD product - not even HD720 - in that category); Privacy is not truly needed unless you are viewing certain kinds of media in public places.
The company says it is going to place lower emphasis on consumer products and phase out some of their low-end products. Instead, they will focus on the defense market and upscale products such as their recently announced $5K augmented reality glasses which is priced for academic research (a fun, but not very large market). I can understand the attraction of the defense market. Sales are less sporadic (Vuzix reports a backlog of $3.8M on page 18); margins are higher. But, how large can this defense market really get?
It seems that Vuzix which is/was known for consumer goggles as trying to move uptown in price, whereas professional HMD providers are being encouraged to produce affordable mass-market products. As the saying goes: "The grass may look greener on the other side, but believe me, it's just as hard to cut."

Financially-focused:

Cash is getting tight (page 2). Cash on hand + accounts receivable was approximately $1.8M at the end of Q2, as compared with approx $4M at the end of 2010.
Accounts receivable ($1.2M) is substantially lower than accounts payable ($3.1M)
Customer deposits (these are typically pre-pays or down-payments) are nearly $900K and are higher than cash on hand (approx $700K). Without down payments, the company looks like it would be in a difficult financial situation.
Inventories ($3.8M, page 2) are more than twice the second quarter product sales (about $1.7M, page 11). Since inventory is typically recorded at cost yet product sales reflect actual prices, this is even more startling. Assuming material cost is the primary driver of cost of sales, and looking at Q2 cost of sales ($1M cost on $1.7M product revenue), $3.8M inventory could be enough for more than $6M worth of product sales, which are at least 9 month's worth of sales. It would appear that Vuzix is sitting on really large inventories. Are these obsolete parts? Do they have very long-lead items that require substantial stocking? Were they planning for much higher sales? Page 7 shows the breakdown of the inventory and shows more than $1M of finished goods waiting to be delivered or sold. On page 17, the first item mentioned in discussion on how to improve cash position is "managing our working capital through better optimization of inventory levels."
There are two loans totaling over $400K made to the company by one or two of its officers. Clearly, cash is a bit of a struggle. There is also accrued compensation of over $100K to be paid in the future to officers.
There is an interesting margin discussion on page 14. Apparently, video eyewear is not a truly profitable business. Augmented reality is a bit more, but the best business from a gross profit perspective is engineering services related to the defense side of the business.
The shift away from consumer products is reflected in lower sales and marketing costs, which the company notes are in part due to reduction in catalog advertising costs (Skymall?)

It's great to have a small public company to provide everyone with some market data on virtual reality!

I think Vuzix was tempted to try and create a substantial consumer market for goggles by offering reasonably-priced, nice-looking products. In my opinion, the experience is just not compelling enough both in terms of resolution/field of view and also in terms of content (games/movies) that can take advantage of the goggle capabilities. As such, not enough goggles are being sold and the company is perhaps unable to get the economies of scale it was hoping for as well as to cover its relatively fixed costs for product development and advertising.

Would the company's product fared much better if they were offered by a bigger brand (e.g. Apple), or is there some other missing ingredient needed to break open the consumer market?

Popcorn carts and the economics of HMDs

2011-09-05T19:43:00.000-04:00

The Sensics YouTube channel received a comment noting that for the price of an HMD, "I might as well build a 30' theater room onto my house with stadium seating and a popcorn cart. Please explain how that price is justified, I'd love to hear it.".

So, at the risk of being a bit wonkish, here are some insights on how the price is determined.

First, professional HMDs are currently made in small quantities. Let's try to figure how many:

A company making professional HMDs was recently listed in Inc magazine. Inc. reported their 2010 revenue as $4.8M. If the average price of an HMD they sell is $20K, and even if we assume that they sell nothing but HMDs, this company made 240 HMDs in 2010. Maybe my price assumptions are too high and they made 300 HMDs. Hundreds, not hundreds of thousands.
A small public company that sells consumer-type goggles released their second quarter financials. They sold approximately $720K of consumer goggles in the second quarter. If all of these sales were goggles, and these goggles were sold for an average price of $250, then they sold just under 3,000 units in the quarter. Thousands, not hundreds of thousands.

In contrast, there were approximately 10 million 3D televisions made last year. With quantities, come economies of scale: you can get parts cheaper, you can invest in manufacturing technologies to make cost lower, you can assemble HMDs at low-wage areas and so forth.

Aside from quantities, one needs to consider the cost of the components that go into making an HMD. Have you ever been with a friend at a restaurant only to hear why the components of the $15 salad probably cost $2.34? It's pretty annoying, but I'm going to do this a little bit for HMDs now.

Historically, the key difference between professional HMDs and consumer goggles was the resolution of the display and the width of the field of view. Professional HMDs had higher resolution and wider field of view. Many professional vendors, Sensics included, use 1280x1024 OLED microdisplays from another small public companies. We use these microdisplays because they are high brightness, low power, high contrast and have rich colors. However, they are expensive. On the open market, these displays (and other like them) could cost approximately $2500 each, so approximately $5000 per system. Yes, manufacturers like Sensics do get quantity discounts, but in my opinion the displays drive the cost of the product into a cost that is in line with a professional market. Once prices are in the range of a professional market, vendors perform their own price/quantity optimization. If you priced a hypothetical product at $20,000 apiece, you could sell a certain quantity. If you priced it at $18,000 apiece, you could probably sell slightly larger quantity, but not dramatically larger. On the other hand, you just lost $2000 of profit. Now, if you could price the product at $995 you would sell many, many more, but then you'd be selling it at a significant loss.

Of course, a direct relationship between cost to make and price to sell is not required. Does a basketball shoe suddenly become much so more expensive to make when an 'Air Jordan' sticker is placed on it? Not necessarily. It is just deemed to be worth more. Sensics is profitable, and the Sensics team would like to keep it this way. In fact, though customers clearly appreciate a discount when they can get one, customers are probably also interested in diversity and choice among HMD companies, so it's good to keep more of them around.

What to do? Priority one is to find a lower-cost solution to placing these dynamic images in front of your eyes. This could come from various places:

Find some lower-cost display. Imagine if TI started making displays just like they make DLP components. Imagine if Samsung, or Sharp, or Micron started to sell high-resolution OLEDs to HMD vendors. Then, prices could significantly come down.
Find some alternative display configuration that is not micro-displays.
Place one display in front of both eyes, though this could reduce the refresh rate that each eye sees.
The passage of time. Just like Moore's laws with CPUs, time brings higher resolution displays at a lower price point.

Last, but not least, what you can do with a 30' theater room and a popcorn cart does not fully overlap with what you can do with a professional HMD. Try carrying the room around from place to place. Try training a soldier with the popcorn cart. But, I am on board. Let's hear it for lower-cost displays that will enable making lower-cost HMDs.

Training using the zSight HMD and the Kinect

2011-07-15T17:14:00.002-04:00

An excellent video showing how the Kinect can integrate with the zSight HMD to product a compelling and effective training scenario. Quoting the author of the video, David from 3DVia: "We made the choice to only use kinect to "watch" if the user is doing the correct motion he/she is supposed to be trained at: rotating the valve or walking. Engaging the body in the experience enables the trained person to take appropriate decision, based on a situation that is made as close as possible to the one he/she would face in an real emergency case The HMD also contributes to the presence of the virtual environment, stereo rendering have been disabled in the video."

3D and the iPhone

2011-07-04T11:09:00.000-04:00

Last year, when Hasbro announced the My3D, a 3D device, I was intrigued: how could a US list price of $35 support the cost of a micro-display, optics, control electronics and a decent enclosure? The answer, of course, is that it doesn't. The device does not have its own display - it houses an iPhone with specialized applications that present 3D images in side-by-side format; it does not have a motion tracker - it uses the iPhone's. It does not have drive electronics - for the same reason.

When the My3D came on the market in April, I immediately went out and got one. After all, buying gadgets is one of the few perks of my job. I downloaded the apps from iTunes, installed them on an iPhone 3GS and put the device to a harsh, un-scientific but brutally accurate evaluation: the teenager test. This is a test where I give a device to a bunch of teenagers and tell them "it's all ready to use; it's free of charge for you; just use it and tell me what you think". As expected, the first five minutes of the test were filled with 'wow - this is cool!' but from the sixth minute till this date, the My3D has never been used again. In contrast, the Motorola Xoom tablet, in spite of surprisingly poor video playback performance, has passed the teenager test with flying colors and is constantly being used.

Before discussing the upside, here are some of the things I don't like about the My3D:

The iPhone 3GS screen, when used in side-by-side mode and magnified through the My3D eyepieces, is nothing to write home about. The resolution is low and the fill ratio of the pixels
When using the My3D, your hands need to hold the device.
User interface is clumsy - holes are provided to insert the thumbs through the My3D enclosure and touch the iPhone screen

But there is an upside. The concept of using the phone (or tablet) to generate the graphics, provide the communication link, app store and perhaps even the motion sensor, is certainly valid. With phone and tablet CPUs becoming ever stronger (the Xoom has a dual-core NVIDIA Tegra) and with goggles supporting side-by-side 3D, you can get pretty good 3D experience without a PC.

The HMD and the fMRI machine

2011-05-24T21:50:00.000-04:00

A couple of weeks ago, I returned from the annual meeting of the Vision Sciences Society in Naples, Florida. VSS is an organization of scientists who are interested in the functional aspects of vision. The VSS conference brings together in scientists from a broad range of disciplines including visual psychophysics, neuroscience, computational vision and cognitive psychology.

There is something nice about coming back to a conference year after year in the same location. You already know where to stay, where to eat or not to eat and are free to focus on enjoying the academics, the exhibition and the Florida sun.

Because VSS is primarily focused on academics, the show floor where my company and others exhibited was not often crowded, leaving exhibitors plenty of time to gleam knowledge from the poster sessions as well as chat with other vendors. The VSS show is dominated by eye tracking companies - most of which we've had the pleasure to work with - but HMDs such as the zSight and other (heavier and lower contrast) HMDs were also on display.

The academic crowd is this show is very diverse in their virtual reality experience. It was great meeting existing customers that we've worked with over the years, but just as fun demonstrating the HMD to researchers admitting that they have heard of head mounted displays but never got around to trying one.

A key part of many vision research projects is to generate visual stimuli. HMDs offer a superb vehicle for doing so, for several reasons:

Arrington Research Eye tracker inside a Sensics zSight

The ability to offer stereoscopic stimuli, thus also adding depth perception to studies about motion, color, recognition and more.
A good HMD is often immersive, meaning that it can block external distractions and make sure that the subject is focused on the stimuli inside the HMD.
The ability to integrate eye tracking inside the HMD.
The ability to engage peripheral vision, especially when using an HMD with a panoramic field of view

I had the opportunity to browse the poster session focusing on research involving fMRI. Functional MRI studies put a subject inside an MRI machine and generate certain visual stimuli. The brain function is then measured and analyzed. Some of these projects give scientific confirmation to seemingly obvious results and others open doors to deeper understanding of how people see. For instance, I saw a poster that confirmed that the brain produces different neural responses when presented with pictures of familiar people as opposed to non-familiar people. To me this falls more into the obvious category: if a picture is instantly recognized as familiar, doesn't recognition itself mean that the brain has a different neural response to it?

Some companies on the show such as Cambridge Vision Systems devise clever ways to inject visual stimuli into such a challenging magnetic environment such as an MRI, and even have the ability to do fMRI eye tracking. Really cool stuff. Over the years, we got asked several times of an HMD could be used inside an MRI machine either for research or as distraction to prevent claustrophobia. Unfortunately, our HMDs are not MRI compatible today, though I'm open to ideas on how to make one.

During a typical year, I meet different people that have different uses for HMDs: from training soldiers inside tanks to architecture and industrial design to academic research. The possibilities for using HMDs seem endless, and many people I meet have new and sometimes crazy ideas on what to do (such as the person suggesting HMDs as a way to revive the art of playing an ancient banjo). Regardless, I look forward to returning to VSS next year and learn even more.

Will Microsoft acquire Lumus Optical?

2011-05-15T18:24:00.001-04:00

There is a rumor going around that Microsoft is negotiating the acquisition of Lumus Optical. Lumus is a startup company that developed a clever see-through eyewear. The company originally targeted both the professional (e.g. defense) market as well as the consumer market, though it reportedly sold off its defense business to focus on the consumer side. The public specifications of its product show a low-resolution (640x480), narrow field of view (27 degrees) display, but one that could be made to resemble a pair of normal glasses.

Publicly, Lumus has been keeping a very low profile. The most recent press release or news article on its Web site, for instance, is dated some two years ago. If it not for their promotion of the upcoming SID show, an outsider could conclude that Lumus is out of business.

Of course, it may very well be that this rumor is untrue or that the acquisition talks will not come through, but it's a good opportunity to analyze where such an acquisition could fit.

One place where a see-through display could fit is with the Microsoft mobile division as part of the Windows phone initiatives. A classic use case of a Bluetooth-equipped see-through display with a phone is showing the caller ID inside the glasses so that the user does not have to pull out or look down at the phone. With phones become more and more sophisticated, dynamic driving directions have also been discussed.

How much would people be willing to pay for such glasses and what would they cost to make? This has been a classic chicken-and-egg problem for goggle manufacturers, where price depends on quantity and quantity depends on price. If Microsoft were to bring this product to market, they could all but guarantee a large enough quantity to make these economical.

Another possible home for a see-through display ix the XBOX group. If motion sensing (e.g. Wii, Kinect) was the last big thing in gaming, 3D, immersion and freedom of motion could be the next. I find this a bit less likely that mobile because gaming often requires immersion and 27 degrees of FOV, at least for the current product, is far from being immersive.

One way or the other, such an acquisition would be a nice shot in the arm for all the goggle vendors trying to find their way to a mainstream market.

If Microsoft ends up acquiring Lumus, you've heard it first here! If they don't, I just might change the title of this post to "Should Microsoft acquire Lumus".

The 11.5M pixel HMD

2011-04-25T11:11:00.000-04:00

A few weeks ago, we had the opportunity to ship to a customer one of our high-end HMDs, with a total of 11.7M pixels. Each eye in this HMD has 5.76 million pixels, about 2.7 times as much as an HD1080 signal.
The piSight 166-43 has a horizontal field of view of 164 degrees, which approaches the human visual field.

Building, testing and shipping such a high-end HMD is always a cause for celebration here. We sell lots of 120-degree and 60-degree products every year, but just a handful of the super high-end ones.

This particular piSight has 12 micro-dispalys per eye. The top photo on the right shows the complete HMD. The photo below it shows the eye modules: the left side of the image shows our patented tiled optics in front of the 12 displays, while the right side shows the screen configuration with the array removed.

It takes one or two computers to drive the HMD to its full potential, and we typically recommend driving it with four HD1080P or 1920x1200 signals which feed into our video processing electronics.

The field of view is just spectacular, as well as the level of detail across it. It is simply fun to see how immersive such product is.

Side by Side 3D and HMDs

2011-03-07T22:29:00.000-05:00

A key advantage of HMDs over 3D televisions is the ability to get true depth perception in a surround video setting. However, depth perception inside an HMD requires different images for the left and right eyes. There are multiple ways to provide these separate images:

Use dual video inputs, thereby providing a completely separate video signal to each eye
Time-based multiplexing. Techniques such as frame sequential combine two separate video signals into one signal by alternating the left and right images in successive frames.
Side by side or top/bottom multiplexing. This method allocated half of the image to the left eye and the other half of the image to the right eye.

The advantage of dual video inputs is that it provides the maximum resolution for each image and the maximum frame rate for each eye. The disadvantage of dual video inputs is that it requires separate video outputs and cables from the device generating the content.

Time-based multiplexing preserves the full resolution per each image, but reduces the frame rate by half. For example, if the signal is presented at 60 Hz, each eye is receiving just 30 Hz updates. This becomes an issue with accurately presenting fast-moving images, or images that need to rapidly change as a result of tracking information or user movement.

Side-by-side and top/bottom multiplexing provide full-rate updates to each eye, but reduces the resolution presented to each eye because only half the available pixels in each frame are used for each eye. Many 3D broadcasts, such as ESPN, chose to provide side-by-side 3D which saves the need to allocate extra transmission bandwidth and is more suitable to fast-paced sports action relative to time-based multiplexing techniques. Of the multiplexing techniques, I think side by side 3D is best for HMDs.

Side by side 3D has some advantages over the full dual-port mode when using a wireless video link. Sending two completely separate images requires a dual-stream video link or two separate transmitters. With side-by-side, true interactive 3D can be economically achieved with a single wireless video link.

Not all HMDs provide depth perception. Some lower-end modules are essentially bi-ocular devices where both eyes are presented with the same image. Interestingly, most professional HMDs do not provide support for side-by-side format. Fortunately for users, both the xSight and zSight professional HMDs do provide such support. Such support also allows you to tap into the wealth of entertaining 3D videos on YouTube 3D, view video directly from a 3D camera and more.

How will HMDs be impacted by the growth in 3D TVs?

2011-02-19T17:21:00.001-05:00

Several million 3D TV sets shipped in 2010, with some reports claiming shipments of over 6 million sets. The industry is clearly aiming for higher volume in 2011. Is this 3D TV growth good or bad for the HMD market?

Very good, in my opinion.

3D TV are useless without 3D content, and this content is immediately useful inside head-mounted displays. Just like surround sound at home did not inhibit portable music players - the Walkman or the iPod - 3D TVs don't inhibit HMD sales.

With 3D TVs, an increasing number of users are exposed to the power of 3D and are thus stimulated to come up with new uses.

3D TVs will drive 3D games, but HMDs offer 3D gaming in a dynamic, 360 degree surround video environment that a TV cannot offer. HMDs can be portable and battery operated, whereas TVs are stationary and power-hungry. You can run around with an HMD on your head (especially with a wireless video link) but no one would consider running inside a room carrying a TV.

Do advances in fuel economy of cars impact air travel? Not really. Similarly, the use cases for TV and HMDs are different in other aspects as well. TVs are better suited to group viewing, whereas HMDs offer privacy. TVs provide all users with the same viewpoint, whereas embedded trackers inside HMDs (such as the zSight) can offer dynamic and individual viewpoints. Some exciting 3D content comes from animated features. How long before such content can be streamed and changed depending on head position? This would be excellent news for HMDs that can provide great immersion.

You can get some great 3D TVs for $2500. When will you be able to get great HMDs for $2500? Perhaps as soon as there is demand for 6 million HMDs a year.

New video demo of xSight head-mounted display

2011-02-12T20:54:00.000-05:00

The xSight HMD is difficult to experience on YouTube, but here's a movie anyway. The reason it's difficult to experience is that two of its most important attributes: field of view and weight are best experienced in person. The horizontal field of view of an xSight is 120 degrees. If you sat 8 feet from a TV and wanted that TV to provide you with 120 degrees field of view, the width of the TV needs to be about 28 feet. That's much more immersive than a front row seat in most movie theaters.

Weight is also hard to experience in a movie. The weight of the xSight is about 350 grams, so similar to a Coca Cola can. The weight of other HMDs is sometimes 1.5 kg, like a 6-pack of beer on your head. Quite a difference!

New video of zSight HMD with iMove

2011-02-01T23:34:00.001-05:00

A new video is available showing the integration of the Sensics zSight HMD with the iMove GeoView software package. iMove allows truly spherical recording of video, which is useful for many applications such as surveillance, tourism, mission rehearsal and more. A vehicle is outfitted with a pole carrying several cameras. Video is simultaneously recorded from all cameras. iMove's software then stitches the individual video streams and prepares a file which can then be played back using a standard desktop or notebook PC. The player receives head tracking information via a simple USB interface from the zSight HMD and changes the viewing direction based on the head movements. The result is like being inside the car.

Various cool videos are available: riding in DC, sailing in a canal in Amsterdam and many others. Several companies also offer services where they can create such immersive videos for desired locations.

Since there is so much video data being recorded at once, iMove decided (at least for this recording) to record at about 8 frames per second. This is more choppy than normal video, but creates a nice balance between the amount of storage required and the smoothness of the image.

Integration is also available for the xSight which provides an even wider field of view and even higher resolution.

New optics make an excellent product even better

2011-01-30T10:50:00.000-05:00

The hard work was worth it. After many months of engineering and testing, my company is now shipping new and improved optics for the zSight SXGA HMD, in both its commercial and helmet-mounted versions.
HMD design is a study in trade-offs, needing to balance weight, resolution, cost, field of view, features and more. Optical design for HMDs is no different. Key attributes to be balanced include:

Field of view. Wider is better for an improved experience. Low-end HMDs Low-end HMDs offer 30 or 35 degrees of diagonal field of view. Professional HMDs offer wider solutions, some even greater than 100 degrees.
Clarity. it's not difficult to design an optical system that provides image clarity at the center of the image, but reaching clarity at the edges or corners is not always an easy task.
Geometric distortion. Minimizing pincushion or barrel distortion is very desirable. Distortion is often expressed in percent as a function of radial distance from the center of the lens, showing how far a point is away from its perfect theoretical position. Some professional HMDs have distortions greater than 10% (ouch!), but 5% or better is desirable.
Weight. After all, these optical systems are designed to be worn on the head. Every gram or ounce counts.
Material used. There are hundreds of optical-quality glass types, but only a few optical-quality acrylic/plastic materials that are suitable for optics design in the visible spectrum. Glass offers greater flexibility in the design (much greater selection of refractive index) but is heavier, more expensive to manufacture, and introduces safety concerns in some applications if there is concern about glass elements close to the eye.
Chromatic aberration. Similar to light passing through a prism, an optical system might impact different colors/wavelengths differently. This might cause blur and color separation.
Eye box. Unlike a telescope lens, HMD optics are used when moving the head. The eye box specifies how much the user's eye can stray away left/right/up/down from the location of the center of the lens without losing the image or suffering significant degradation in image quality. Bigger eye box is better.
Eye relief. This specifies how far can the user's eyes be away from the lens. Being too close is a source of discomfort - your eyelashes could brush up against the optics. A larger eye relief allows those with glasses to wear them.
Diopter adjustment. Some designs allow for compensating for glasses - just like the eyepiece of binoculars allow for some adjustment so that those with glasses can, but don't have to, wear them. This is a plus.
Physical attributes. Since this optics design needs to fit inside an HMD, it cannot be too long. Eyepieces cannot be too wide either because two eyepieces need to be next to each other with a minimum distance of about 52 mm to accommodate the desired ranged of interpupillary distance (IPD).
Light transmission. Some designs transfer only 25% of the light from the source. Keeping most of light allows higher brightness and more energy-efficient HMDs.

The process of optical design involves specifying the design targets but then fighting through the trade-offs. Should we use glass? How much eye relief is good enough? Are you willing to sacrifice clarity for geometrical distortion? on, and on.

Many months and thousands of dollars into it, I think we nailed it. The optics are very lightweight (about 10 grams each), offer 60 degree field of view that is crisp to the edges. Low distortion. Big eye box and nice eye relief. There is no perfect solution because of the trade-offs but I think we hit a very good one.

I like looking at great design: an Italian wine glass, a Frank Lloyd-Wright house, an iPad. With these new optics, I liked looking through a great design.

The optics are now available as part of the zSight, a 60-degree OLED SXGA HMD with integrated tracker, audio and stereo sound. Try it if you get a chance.

Eyelids as shutter glasses

2011-01-19T07:35:00.001-05:00

Is it real or just French humor?

If not fake, an amazing demo illustrating how you could conceivably get alternating-frame/frame sequential 3D without shutter glasses by modulating the eyelids.

Probably belongs to the 'don't try this at home' category?

How long can you use this without damaging your eyes?

Would you be brave enough to try it on yourself?

Can nostrils be modulated for stereo smell?

I still prefer head mounted displays for true stereo, motion tracking and the full immersive 3D experience.

Kopin acquires FDD. Would the market benefit if eMagin was also acquired?

2011-01-16T21:08:00.000-05:00

Kopin, a publicly-traded maker of transmissive micro displays acquired Forth Dimension Displays (FDD), which makes reflective micro-displays.

Reflective micro-displays have been used for HMD applications, but the optical system and constant light that they require provide lower contrast relative to OLED alternatives, as well as makes for a more complicated, larger and heavier optical system. Reflective displays could still be an excellent choice for high-end projection applications that use very bright light sources, but not so much for HMDs. For HMDs, transmissive or self-emitting displays seem to be the better, easier solution. Kopin makes transmissive displays. eMagin makes self-emitting displays.

How would the market benefit if a large company acquired eMagin or if there was another way to invest larger amounts of money into eMagin?

For HMD sales to really take off, display components of reasonably high resolution need to be inexpensive. You can describe eMagin products in many good ways, but inexpensive is not one of them.

One would think that eMagin's prices are driven by a limited production capacity. If their production capacity was hypothetically unlimited, eMagin could select a different - and lower cost - point on the price/quantity curve to drive market adoption and maximize their profits. If their production is limited, it would not make sense to generate more demand than they can produce. The answer in this case is increasing production capacity, which has now been announced as something eMagin is doing according to their press release. However, eMagin's cash reserves allow for only a certain level of investment and they would need to generate more cash organically to increase it. What if someone could finance two, five or ten new lines instead of just one? Would that drive quantities of quality HMD products? Would that provide a boost for the industry?

HMDs and the Out-of-Body Experience

2010-12-17T10:05:00.001-05:00

It seems that one the favorite things for HMD users is to look at themselves - virtually. There are several ways to do this.

One way is to install video cameras that capture the user's image. This could be from the front to reproduce the feeling of using a mirror, but it is often more interesting to capture the user from the top or back, this providing an 'out of body' experience. Experiments along these lines have been reported several years ago by the Karolinska Institute.

Another way is to perform 3D laser scanning of self, such as using the services of our friends at ShapeShot. This allows creating a virtual 3D representation of a person and then incorporating that representation in a virtual scene. The effect of browsing a virtual museum and then seeing your head rotate on a pedestal like a Greek head sculpture is quite amazing.

A third way is to use motion capture technologies and then create a moving avatar that mimics your movements. The avatar does not have to be a true representation. It could have a over sized head, or a severed leg, or have limited range of motion. This can be used to illustrate the impact of several disabilities, or sometimes to help with a treatment or physical therapy plan.

In a recent trade show, we installed an iMove spherical video system on top of a post and allowed users to explore the trade show from high above by wearing an HMD with an integrated head tracker and having that head tracker control the direction of the camera. Interestingly, a favorite target was to look down, which caused the camera on the post to look down, thus providing a view of self.

Why is this fascination with viewing self? How can this be used, in 3D, to provide new experiences such as a virtual shopper that lets you see how you would look in a new piece of clothing, or help explore what it would feel like to be taller, or shorter, or otherwise different?

Reflections on a reflective LCOS press release

2010-12-11T15:43:00.000-05:00

A few days ago, a press release from one of the liquid crystal on silicon (LCOS) chip vendors landed in my inbox. Two things caught my interest:
1. That sometimes in 2012 (e.g. 13 to 24 months from today) they would release a display with 2048x1536 pixels.
2. That this chip would eventually power head-mounted displays that could serve as an alternative to achieving wide field of view and high resolution through multiple low-resolution displays.

I think Sensics is the only company that has tiled head-mounted displays as part of the product line, I took that as presenting a 2012 alternative to Sensics. If imitation is the most sincere form of flattery, than I and the rest of the Sensics team should take this press release with pride.

Other than pride, it gives us an opportunity to review the criteria we use for selecting the display chips that power our products. We are not married to a specific display technology, but strive the provide an excellent combination of panoramic field of view and high definition and a lightweight design. Panoramic field of view provide greater sense of immersion and heightened situational awareness. High definition provides more realistic images. Lightweight designs can be used for extended periods without neck strain or long-term health concerns. Customers have come to understand why all these attributes are important and want to go forward, not backward, with future HMDs.

Last time we made that choice, OLEDs were best for us and our customers as they offer unique advantages:
- Simpler optical design which translates into smaller, lighter products. OLEDs are self-emitting whereas LCOS is a reflective technology requiring an external light source and more complex optical path.
- Higher contrast. Because the external light source for LCOS shines even when the displays need to show black, the overall contrast in LCOS is typically much lower than OLED
- Faster switching and no motion blur. LCOS relies on an external light source an LCOS chip is typically lit by three alternating light sources. In contrast, OLED is faster and the colors are continuously lit.
- Lower power consumption which translates into less heat near and head and longer operation when used with a battery.
- Simpler drive electronics which make for smaller products.

LCOS chips do offer two advantages:
- Greater fill ratio, which is the percent of useful area in the chip.
- Ability to offer very high brightness if lit by a powerful lamp. This could be important for actual flight helmets that need to be used in direct sunlight but not as much for HMDs.

As part of the xSight and piSight product line, Sensics achieves panoramic field of view, high definition 1080P and light weight by optically tiling several 800x600 OLED displays. Tiling is getting better and better every year, and tiling HMDs could be introduced for 1280x1024 or even higher resolution OLEDs, effectively decreasing the number of tiles while maintaining all the other advantages of the tiled design.

BTW, the Sensics zSight is not tiled and offers a single-screen solution for those that prefer it.

Come late 2011, 2012 or whenever a higher-resolution LCOS chip is ready, Sensics will be more than happy to take a serious look at it and see if it could improve the customer experience relative to then-available alternatives. Until then, we're just glad to be on everyone's mind.

Vega Prime, VBS2, Adacel, iMove, WorldViz and Harry Potter integration

2010-12-03T11:59:00.002-05:00

We're back from I/ITSEC, the world's largest training and simulation show where we demonstrated several really cool integrations between popular software packages and the our head-mounted displays.

Here are some of the demos we showed:

xSight HMD with Vega Prime from Presagis. Vega Prime is a popular environment for creating 3D simulations, and powers many commercial flight simulators, often in dome or multi-screen configurations. Integrating Vega Prime with a panoramic HMD allows a very portable solution which is useful if the simulation needs to come to the trainee, or if there are very confined spaces. With a good graphics card and a professional motion tracker from Intersense, we allowed visitors to immerse themselves in a virtual construction site. Update: Presagis also thinks it was cool
zSight HMD with VBS2 from Bohemia Interactive coupled with voice recognition from Adacel. VBS2 is often used for ground training and provides a rich environment for creating, executing and debriefing training missions. In our demo, we coupled VBS2 with the zSight SXGA HMD including integrating the zSight's built-in three degree of freedom tracker, so that the user head movements are reflected on the screen. The voice recognition engine from Adacel, coupled with the zSight's built-in microphone and stereo sound, allowed simulated artillery training. Speaking commands while wearing the HMD results the computer recognizing, processing, executing and then responding to the commands by voice for a very compelling experience.
iMove provide multi-camera solution that allow complete spherical viewing and recording of space. We placed an iMove camera pod on top of our booth and it provided true 360 degree views of the conference floor. By integrating the iMove viewing application with the built-in tracker inside the zSight HMD, conference attendees were able to don the zSight and explore the conference from up above. Since the camera was right above our booth and provides full spherical imaging, users could sometimes see themselves wearing the HMD for somewhat of an 'out of body' experience.
The Vizard software by WorldViz is an interactive development environment for all things 3D. We have been working with our friends from WorldViz for several years now, and their demos do a great job in showcasing the power of VR. At the show, we have Vizard running with an xSight panoramic, high-definition HMD with full 6-degree (yaw/pitch/roll, x/y/z) tracking.
Last but not least, we demonstrated the low-latency wireless link for the zSight HMD as well as the zSight's ability to display 3D side-by-side content being popularized by YouTube and 3D broadcasts. The wireless link worked great at the show providing high-definition images, long range (we were able to walk more than 100 feet without losing signal in an otherwise busy show environment), stereo audio and long battery life. For a full show day, we probably used 3 to 4 rechargeable batteries, and with the quick charge feature it was easy to charge one while another was being used. The highlight the 3D aspect of the zSight, we found several side-by-side movie trailers. The new Harry Potter movie (3D trailer link here) beat Cloudy with a chance of meatballs as well as Toy Story 3 as the show favorite.

Overall, lots of fun in an otherwise serious exhibition.

What users want from a wireless video link for HMDs

2010-11-16T09:33:00.000-05:00

Sensics surveyed 179 users to determine their needs and wants on a wireless video link for head-mounted displays. The results of the survey, already shared with its participants, will be published in a few weeks, but I wanted to share some of the insights from it.

As part of the survey, we asked respondents to indicate the importance of several wireless link attributes such as distance, latency or battery life. Each feature was given an importance score from 1 (completely unimportant) to 6 (very important). We then sorted the aggregate results by the average importance assigned to them by the users. Results were:

Importance of various attributes of wireless video links for HMDs

It seems that the highest priorities were assigned to these features that make the wireless link nearly transparent to the application: high refresh rate, low latency, closed-loop operation to include down link tracking and stereo operation. The combination of these features differentiate wireless video products designed for HMDs with those targeted towards in-home use primarily to extend the distance between the set top box and the HDTV. A 1/2 second video link latency, for instance, might be quite insignificant when watching a basketball game on TV, but is devastating to closed-loop virtual reality applications.

Another key difference between in-home HDMI extenders and professional HMD-driven wireless links is the transmission technology. In-home HDMI extenders typically use ultra-wide band or the new 60 GHz extremely high-frequency radio band. Both of these need line of sight between the wireless transmitter and the wireless receiver. However, imagine an HMD user wearing the receiver in a backpack and then turning around. The user's body can easily cut off the line of sight between transmitter and receiver. Thus, technologies such as Wireless N which are not sensitive to line of sight limitations are clearly preferable for wireless HMDs.

I believe that wireless video links will help unleash the true potential of the personal, portable HMD. The freedom of movement, multi-user interaction opportunities offered by wireless links are substantial and I'd expect many exciting applications to materialize in 2011.

Fourth generation optics are here

2010-11-11T09:42:00.003-05:00

Someone once said 'Science is like a horse, not a cow; feed it, don't worship it'. I read this to say that even if you have very good technology, don't rest on your laurels but rather keep investing to make it great.

For several years now, My company has been building virtual reality goggles based on a unique optical tiling technology which allows creating goggles that are lighter, have wider field of view and higher definition.

At the core, the problem to be solved and the tiling solution are both simple to explain. Users want panoramic field of view because once the peripheral vision is engaged when wearing goggles, the sense of realism, immersion, "being there" is greatly enhanced. At the same time, users want high resolution and high pixel density so that they can get lifelike images. With today's micro displays, it is difficult to do both. If you magnify a micro-display just a little bit, you get good pixel density but narrow field of view, or "tunnel vision". If you magnify a display too much, you can get good field of view but low pixel density. High magnification also brings concerns about total weight, image distortion, clarity at the edge of the image, and all kinds of things that goggle manufacturers worry about.

If you could somehow make micro-displays that physically overlap, you would be in good shape because you can create a really high-definition micro display from many lower-resolution ones. However, making displays physically overlap is practically impossible. Our technology makes the displays optically overlap, essentially by carefully putting small magnifying glasses in front of each display. If they are properly positioned and aligned, the result is a nearly seamless image of both high resolution and panoramic field of view.

After investing quite a lot, we have now made 'nearly seamless' even better. Our fourth-generation optics are even better now. We've improved image clarity at the edges. We've improved what's called the 'eye box' (larger eye box means that you can still see a good image even if your eyes are not at the optimal position relative to the goggles). We've even changed the material from which we make the lenses.

So, we are excited about these fourth-generation optics and people that have tried it on are also excited. If you are in the market for a professional set of goggles, make sure you stop by to see our progress.

Overweight HMDs are still heavy, regardless of how they are supported

2010-11-02T17:02:00.003-04:00

When I was a kid, I was discussing with a friend how overweight people sometimes wear black because it makes them look thinner. "Yeah", he replied, "but such clothing still uses a lot of fabric, even if it's black fabric".

I feel the same way about some of the professional HMDs out there. 2 lbs (1 Kg) is still 2 lbs no matter how hard you work to distribute the weight on the head. It's still heavy.

Most of the weight in head-mounted displays is concentrated in the front. This is where the optics are, as well as the micro-displays, and some electronics. Using such front-heavy HMDs can cause neck strain and general discomfort. Some companies actually put a counter-weight on the back of the head to offset the front-loaded weight. Others experiment with sophisticated head mounts that are designed to more evenly-distribute the weight on the head. But even if not heavy immediately, wear a 2 lb HMD for 10 minutes and you won't forget that it's 2 lbs.

HMDs should simply go on a diet.

Users get this. We just returned from Europe where we'd have a chance to see potential customers try on different HMDs from different vendors. Often times, heavy products were discarded very quickly. "Just too heavy", we heard.

Would you put up with sunglasses that hurt your nose? How about a hat that became truly heavy after 30 minutes?

Big may be beautiful, and black fabric does not hurt either, but heavy on the head is just too much to take.

Cut the Cord

2010-07-11T09:23:00.008-04:00

My company has been busy demonstrating a high-performance wireless video solution that works with our virtual reality goggles. As part of product launch preparations, we've surveyed many professionals worldwide to understand what they are looking for in a wireless HMD and why would they find such a product attractive.

A simple question that was asking the survey participants what the key benefits they are interested in obtaining from a wireless HMD. About 170 people answered this survey and the benefits they identified were as follows:

Survey participants could choose more than one benefit, so it's ok that the total number of benefits selected exceeds the number of respondents.

These responses also came with an area for entering free text. Many of the free text responses associated with this question dealt with the complexity and inconvenience of having an HMD being tethered. For instance, some respondent wrote “Detangle the user, prevent them from pulling on the cord and knocking the HMD off”, and “The main reason I have for obtaining a wireless HMD is that users get entangled with the cables VERY often, causing discomfort and frustration”.

This is perfectly understandable. A key attraction of using HMDs is the potential for a portable, personal experience with 360° freedom of movement. However, as long as HMDs are connected via an 'umbilical cord' to a PC, movement is restricted and cables become a major distraction.

I do think early implementations of wireless goggles will have the wireless receiver and perhaps a battery pack as some sort of 'belt pack' from which a short cable will run to the HMD. This still provides very substantial mobility benefits. Just try rotating 360 degrees with wired HMD! As far as we have seen, the level of integration and power consumption is not there yet to combine power and wireless in package that is so small and so light that it can be worn on the head next to the goggles.

Wireless, Anyone?

2009-11-20T11:47:00.003-05:00

The most common question I get asked when demonstrating the Sensics head-mounted displays is "Can you make them wireless?" Just like the transition from a corded phone to a cordless phone and then to a mobile phone, there are numerous reasons to wish that HMDs could be untethered: no more having to carry an 'umbilical cord' cable around, or worry about tripping over it; fewer distance limitations; greater freedom of movement.

The day of 'cutting the cord' is now here. During the first week of December, my company will demonstrate a wireless HMD at the I/ITSEC show in Orlando, and I can't wait to see the responses.

Imagine a virtual reality application to train NFL quarterbacks, who already wear helmets for a living. Let's attach a high-performance HMD to the helmet and let the quarterback move around while seeing next week's opposing defense. Maybe even add an eye tracker to make sure he is scanning the field correctly and looking at the right passing options. With tethered HMDs, this application is a problem: quarterbacks are so expensive these days, that teams might be concerned about having them trip over a cable or get tangled up. With a wireless HMD, many additional options are open.

The same is true for numerous other applications: driving simulators, maintenance trainers, infantry trainers, just to name a few.

What makes wireless video for virtual reality applications more difficult is the concern about latency and frame rate. If you are watching a movie on a TV at home, and that TV receives a wireless video signal from your PC, there is no problem if there is a consistent delay between transmit and receive. Who cares of the movie is 2 seconds delayed? But if you are wearing an HMD with a motion tracker, and the image changes based on your movements, you are on a strict latency budget. Too much latency, and you are likely to experience motion sickness. The same goes for frame rate, where interactive applications need high refresh rate.

Wireless, anyone? Come to Orlando.

Weight watchers

2009-09-23T11:44:00.005-04:00

How much can an HMD weigh before using it becomes a long-term wellness risk? 1 lb? 2lbs? 3lbs? Is a 2 lb (1 Kg) HMD that has a front-loaded weight concentration better or worse than a 3 lb model that has a counterweight on the back to balance it? Should users be more concerned about neck strain from trying to balance an HMD or about neck compression from wearing a large weight on their head for more than a minute or two?

Clearly, the answer also depends on the expected usage scenario. If the HMD is to be used in a static position such as near a desk, a 'boom' arrangement might be just fine. Alternatively, if rapid movement or usage that is longer than 10 minutes is expected, the answer might be different.

When My company designed the panoramic, high-definition xSight (12 oz, about 300 grams), we decided to conduct our own experiments to determine what acceptable weight would be. The head-mounted portion of the xSight was modeled after a ski goggle and for the experiments we would wear ski goggles with fishing weight taped to their sides. We would sit through meetings and other parts of the work days wearing the ski goggles with fishing weights. It was clearly evident for each of us how much weight is acceptable and how much is not.

One concern when potential customers look at an HMD is that they use it for a minute or two at the trade show. Under these situations, even a 3 lb HMD might seem acceptable, though if the user would wear it for 10 minutes, weight would become a very big issue.

So - watch the weight of your HMD.

HMD Knowledge in a Bottle

2009-07-24T09:45:00.003-04:00

We took the time to capture some of HMD knowledge in a fairly extensive document which can be obtained here. It summarizes hundreds of customer discussions regarding important attributes in professional HMDs, (NOT old-fashioned HMDs like the one in the picture) and many of the little things that make a big difference to HMD users, some of which you might not find on product spec sheets. In a way, we captured 'HMD knowledge in a bottle'. I'd love to hear back comments on where this white paper is wrong or right, and what else is missing.

Movie Night

2009-06-13T17:05:00.003-04:00

In my virtual reality company, we demo our product to many people. It seems that everyone has their favorite movie including a head-mounted display. Some of the common ones we hear are:

Disclosure
The Lawnmower Man
Star Trek (as in 'can you create a Holodeck?')
Demolition Man (though no one remembers the name of the movie, only Stallone's HMD scene)
Vanilla Sky

and an honorable mention to Forbidden Science, a late-night series that used one of our products as a prop.

The Year of the Lightweight HMD

2009-02-20T14:26:00.002-05:00

I think it's time that the weight of professional HMDs (those with high resolution and wide field of view) gets measured in ounces (or grams) and not in pounds (or kilograms), and I think this will happen in 2009.

Weight is so important in HMDs that it is sometimes overlooked. Lower weight means greater comfort. It means the ability to wear the HMD for longer amounts of time. It means that you can almost forget you are wearing it. Balance (as opposed to front-heavy) is also important as it impacts neck strain. Sometimes, HMD vendors add a counter weight in the back to balance their front-heavy design, but that's clearly the wrong solution.

Why do professional HMDs have to be heavy? The traditional thinking is that this is like asking why a manned fighter jet has to be heavier than an unmanned plane (e.g. drone or UAV). The pilot and flight suit add about 100 Kg to the jet, but that's just the beginning. Now you need oxygen, and a seat that you can eject, and instrumentation you can read. Before you know it, you need larger engines to carry this weight and longer wings and more fuel, so the net difference is much larger than the weight of the pilot.

Traditionally, that has been the thought of professional HMD vendors. The idea was that if you want wider field of view, you need a large lens, and to minimize distortion it needs to be complex and heavy. If you needed high resolution, you had to use more expensive display elements that needed external lighting (and a way to diffuse the light so that it is even). Before you know it, professional started equating with heavy. Heavier designs meant larger (and yes, heavier) head mounts.

Not good.

And not true. At least not for long. By using new types of displays, optics and components, professional is no longer heavy. New products, such as the xSight weight about 10-12 ounces. Not quite as light as some of the consumer HMDs in the 'Skymall magazine', but getting there.

So will this be the year of the 'good enough' professional HMD that combines wide field of view, high definition and light weight? I hope so, and I think it will.

Harvard Business Review covers Virtual Worlds

2008-10-10T10:08:00.002-04:00

HBR published a short piece last month titled "Getting Real About Virtual Worlds" in which Paul Hemp talks about the business benefits of using 3D environments. Coverage by the Harvard Business Review, given its stature and high-level distribution, is a good sign for an industry.

To me, the article underscores once more the notion that compelling, useful content is already out there. Unlike many non-impressive virtual reality offerings in the 90's, CPU and GPU power is good enough to create a very compelling experience...if you have a good-enough display system.

Using these compelling 3D worlds on a desktop monitor - usually in 2D - is like watching a new High-Def DVD release on a small black&white television. Try it on a high-resolution HMD with wide FOV and good peripheral vision, and you'll never want to go back.

The HMD and the Washing Machine

2008-06-18T11:56:00.002-04:00

Should see-through head-mounted displays be also used for completely immersive (non see-through) applications? Technically, it's certainly possible. Just put a dark cover on the front glass, and you have something similar to being completely immersive. However, having the extra weight, the not-quite-as-bright image and a bulkier product is certainly a downside for dual-use HMDs.

This reminds me of the washing machine / dryer combination. What a great idea on paper - no need to move laundry from one to the other. In practice, there are so many drawbacks to this combined approach that most everyone still has separate washers and dryers.

Not to showcase my age too much, but as a teenager I owned an Apple II computer. Not an Apple IIc. Not an Apple IIe. A real Apple II (serial number had 4 digits) and at that time, it would connect to our standard TV through a small conversion box. But, even though it made sense at the time, computers migrated very quickly to separating the TV from the computer monitor. Same concept, different time.

So... I still think see-through HMDs and fully-immersive HMDs should be kept separate for now.

The Mind-Reading Virtual Binoculars

2008-06-12T10:33:00.005-04:00

Northrop Grumman Corporation (NYSE:NOC) announced that it was awarded a contract to develop electronic binoculars that use brain activity to detect threats. The press release is here. My company is proud to partner with Northrop on this project.

The project seeks to present a panoramic day/night optical system to a user and monitor brain waves - yes, brain waves - to detect targets of interest. A panoramic view and high-resolution image was deemed very important for this project, which is why Sensics panoramic displays were chosen.

However, the drive to present users with diverse visual stimuli and measure brain activity is certainly not unique to the defense markets. We are working with several academic institutions to achieve much of the same goals within a research context.

Presenting users with narrow field of view displays could be considered not to put the human brain to full use as a super-fast object/pattern recognition computer. This is especially true given that difference parts of the visual field are used for different purposes: central vision is used for reading and perception of fine detail. Peripheral vision is used in humans and also in animals for threat detection, sensing movement and presence of peripheral objects. Try walking around with toilet paper rolls on your eyes (see this demo on the Washington Post) and you'll understand how critical peripheral vision is.

See-through HMDs have lots of ground to cover

2008-06-02T19:23:00.004-04:00

See-through HMDs (sometimes referred to as 'personal displays') are sometimes portrayed as combining eyeglasses with a video ipod. In a perfect world, they look like cool fashion accessories, are so light that you can wear them anytime, yet overlay what you see through them with computer-generated images. Perhaps it's a movie that you can watch on a plane. Perhaps it's driving directions or text messages from your friends.

However, to get there, many problems remain to be solved:
- Brightness: can these devices be used in a sunny day?
- Weight: can you really walk around with them all day?
- Power consumption: how long does the battery last and where is the battery placed? If the battery is part of the HMD, it adds weight. If it's on a belt-clip, it adds inconvenience and perhaps a cable.
- User interface: how can you control what information is displayed, or when it is shown.
- Dual use: if the goal is to watch a movie, it's often better to block out external imagery. But how? Do you add a cover to the front? Do you make the glass part so dark that it's effectively like sunglasses?
- Placement of information: if the information is placed in the central vision, it might intefere with reading tasks. If it's placed in the peripheral vision, it is difficult to read. If it's both, you need a very wide display element.
- Stereo vision? At some level, a monocular display (one eye) might be sufficient for textual information. However, if you're looking to augment a scene, the feeling of stereo may be very important. Stereo, however, nearly doubles the weight, cost, power consumption and cabling requirements of the display electronics.
- Sound - can you or should you integrate the display with an earpiece
- Cost, for obvious reasons

I'm curious to see what Apple (who recently published some patents on this) and Sony (recently showing a cool display at SID 2008) have up their sleeves for this product category.

The Alienware curved monitor and a curved HMD

2008-05-16T10:40:00.003-04:00

Earlier this year, Alienware (now part of Dell), demonstrated a high-resolution, panoramic monitor. The 'Curved monitor' blends together four DLP projectors to create a wide display that is over 3 ft wide and has 2880x900 pixel resolution. You can read the Gizmodo review of this monitor, and see some videos at here

The process of tiling together multiple displays to achieve a curved, panoramic, high-resolution product is exactly what my company does to generate a curved, panoramic, high-resolution head-mounted display. The multiple displays are positioned so that they approximate a sphere around the eye, for a 'surround video' experience. Resolution is similar or higher than the curved monitor.

The decision whether to use a curved panoramic HMD or a curved panoramic monitor
is a bit like comparing a car to a plane: there are uses where one is better than the other, with some area of overlap. If you need many people to see the image at once, a monitor is better. If you are not inclined to wear anything on the head, the monitor is better. However, if you want stereo images, if you need portability, or if you want 360 degree immersion (image changes when you move your head), the HMD is your choice.

Some of the things I'm curious regarding the curved monitor:
- Does it require special video output modes, or can it accept stadnard display resolutions?
- How many video inputs does it require?
- How good is the tiling/stitching between the displays? In the Gizmodo video you can clearly see the seams between the projectors. Is there a way to achieve a seamless image?
- From a manufacturing standpoint, how does Alienware perfectly align the DLP projectors to achieve pixel-by-pixel matching? Does this alignment change over time?
- How good is the projector-to-projector color matching?

It's great to see tiled displays hit high-end consumer applications. I'm looking forward to trying a curved monitor one day soon and comparing it - with the same application - to a curved HMD

The Display Inside The Display

2008-03-21T14:10:00.003-04:00

When users talk about their recent experience with head-mounted displays, they often refer to image quality. "The colors were great", "the image was very bright" and so forth. Indeed, image quality (lack of smear, brightness, contrast, color gamut) scores very high on HMD requirement surveys, right next to high resolution and wide field of view.

It turns out that the micro-display technology used inside the HMD can critically impact the user experience, often more than other design decisions for a particular HMD.

Many HMDs today using LCOS (liquid crystal on silicon), which has similarities to LCD technology, with the exception the LCD is transmissive - that is, the light source is behind the display, where LCOS is reflective - which means the light source needs to project the front of the display. LCOS displays are a common choice because they come in higher resolution than comparable single-chip OLED (Organic LED) display, and are available from multiple vendors. Unfortunately, LCOS-based designs have some disadvantages:
- They require a light source which reduces the display contrast (the light is on even when a black picture is displayed) as well as substantially increases power consumption.
- They often have limited temperature range, or require somewhat exotic solutions like local LCOS heaters to warm up the display before initial use.
- They typically exhibit motion blur and smear, just like many of us experienced with LCD monitors. This is particularly relevant in HMDs since images inside an HMD constantly change to track user head movements.
- Depending on the specific design, users may experience color flashes or other artifacts during head movement.

OLEDs are also used in several HMDs. OLEDs are self-emitting, meaning that they do not require independent lighting. OLEDs, such as those available from eMagin, offer a larger color gamut, higher brightness, higher contrast and lower power consumption. OLEDs are also faster than LCOS and thus are free of LCOS artifacts. However, OLEDs typically come in lower resolutions that LCOS displays and are available from fewer vendors. To overcome the lower resolution of individual displays, vendors such as my company, use a tiled approach to combine multiple micro-displays into a larger, higher-resolution image.

So... look inside! Next time you look at an HMD, inquire what display technology is inside the display.

The 2008 HMD Market Study Survey

2008-03-05T21:20:00.002-05:00

Sensics released today the 2nd annual HMD market survey. In this survey, conducted in February of 2008, we re-examine the same questions asked last year in the 2007 HMD survey, and then go on to compare and contrast the market needs.

The 2007 survey was titled "Are existing HMDs good enough?". The answer was a resounding "no!", and it helped quantify what we all intuitively know - that HMDs with narrow field of view ("tunnel vision") and low resolution may be sufficient for very early adopters, but are certainly not good enough for widespread use.

The 2008 survey is titled "Are we there yet?". The short answer is also "no", though perhaps without the exclamation point. Requirements have not changed much, and vendors are certainly making progress in working towards what users are asking for - wider field of view, higher resolution, in a light-weight, easy-to-use product. Panoramic field of view products such as those that we make are maturing, and other companies have innovative products (though sometimes curious-looking, such as the "wedding cake HMD") that try to solve these problems. However, the "holy grail of HMDs": panoramic field of view, high def, low cost, easy to use - has not been conquered yet. Look for additional fascinating developments in 2008.

I'll be posting additional insight from these survey from time to time. Stay tuned.

Optical see-through vs. Video see-through

2008-02-21T23:20:00.002-05:00

If you're interested in augmented reality, you've probably thought about optical see-through and video see-through.

Is one better than the other? That's like asking whether a car is better than an plane. It depends on what you need to do: get groceries? the car is unmatched. Get from New York to San Francisco? a plane is very often your choice.

Video-see through systems present video feeds from cameras inside head-mounted devices. You can see an example here. This can be useful when you need to experience something remotely: a robot which you send to fix a leak inside a chemical plant; a vacation destination that you're thinking about. This is also useful when using an image enhancement system: a thermal imagery, night-vision devices, etc.

Optical see-through systems combine computer-generated imagery with "through the glasses" image of the real world, usually through a slanted semi-transparent mirror. If you are in a mission-critical application and you're concerned what happens should your power fail, an optical see-through solution will allow you to see something in that extreme situation. If you are concerned about the utmost image quality, portable cameras and fully-immersive head-mounted display can't match the "direct view" experience.

One aspect of video see-through systems is that it's much easier to match the video latency with the computer graphics latency. Latency (delay) is inherent to immersive imaging systems: motion trackers are not instantaneous; computer graphic generation is not immediate and even when refreshing images at 60, 70, even 120 Hz, there is a lag from sensing to imaging. When computer graphics need to be overlaid on the image from the actual world, there is a difference between video see-through and optical see-through. Optical see through offers no latency, which sounds great, except that there is an inherent mismatch or lack of synchronization between what you see through the glasses and the graphics. If you're showing a virtual sofa inside a real living room, this mismatch can be distracting. In contrast, using video see-through allows you to synchronize the delay so that your video and graphics are always in sync.

With optical and video see-through, no solution is perfect for all problems.

Are you blind when using a head-mounted display?

2007-12-02T09:43:00.000-05:00

I received a letter from the Maryland Motor Vehicle Administration that my driver's license needs to be renewed. This happens once every five years, and also is a nice reminder that my birthday is coming up.

Checking the MVA site to determine the closest location and list of required documents, I also saw the vision test requirements. Take a look at these requirements from the Web site:

To qualify for an unrestricted driver’s license, the State of Maryland requires drivers to have:

Binocular vision
Visual acuity (Snellen) of at least 20/40 in each eye
A continuous field of vision of at least 140 degrees

Restricted licenses may be issued to persons having:

Visual acuity of at least 20/70 in one or both eyes
A continuous field of vision of at least 110 degrees, with at least 35 degrees lateral to the midline of each side

A related definition of visual performance is the 1934 American Medical Association of blindness: "Central visual acuity of 20/200 or less in the better eye with corrective glasses or central visual acuity of more than 20/200 if there is a visual field defect in which the peripheral field is contracted to such an extent that the widest diameter of the visual field subtends an angular distance no greater than 20 degrees in the better eye."

Those definitions made me wonder: how close you are to being blind when you wear an HMD?

With most head-mounted displays, field of view is limited: typically 30 to 50 degrees horizontal (not far from the definition of legal blindness). Only very few products, such as those from my company offer wide field of view that would be enough to meet the driving requirements.

Similarly, many HMDs have limited resolution that would constitute legal blindness. While resolution is sometimes reported as total number of pixels (e.g. 800x600 pixels/eye), an important measure is the number of pixels per degree, or pixel density. I saw a product recently that claimed wide field of view but offered less than 10 pixels/degree. This is about 20/125 visual acuity according to Snellen notation (6/38 metric). For more technical information, see this excellent Visual Acuity web page at the University of Utah.

I'm not suggesting in any way that anyone should attempt to drive with an HMD. It does seem, however, that most HMDs still have a long way to go in enhancing field of view and resolution/pixel density before they become truly usable.

How important is VR in education?

2007-11-09T16:49:00.000-05:00

I had the pleasure of participating in a panel discussion in the "Simulation and the learners of tomorrow" conference organized by the Center for Technology (CTE) in Education at Johns Hopkins University. The conference was attended by teachers, educational administrators, technologists, government and state officials and more.

CTE is one of our first customer, focusing their research in science and math education on the middle-school level. It was fun spending the day with educators to discuss the future of education and how simulations, virtual reality and other advanced technologies can take a part in educating our kids.

I picked up on several interesting questions :

How effective will VR be in education? There is no question "VR" is super cool - just ask the dozens of kids that tried Sensics head-mounted displays, but does this coolness translate into more effective learning and retention?
What is the preferred display device for VR learning? Is it a large projection display which engages all the class at the same time, or is it head-mounted displays that allow different students to learn immersively at their own pace with little or no distraction from other classmates?
Who will pay for VR? Will the school system do it using technology budgets? will this become an after-school enrichment program?
How will VR get into the classroom? Will it be a top-down decision from the administrators, or a grass-roots action where rebel teachers that are excited about the VR experience they had elsewhere (home, arcade, science museum) experiment with it in class?
If VR is effective, is this effectiveness the result of the newness factor (students are glued to the device because it is new and they have not experienced 3D immersion before), or is it really more effective than standard interactive software on a desktop?
What material will be most effective for VR? Is it math and science (e.g. "fantastic voyage inside the body"), history (where VR can take you to other civilizations)? something else?
How quickly will VR in education become mainstream?

CTE plans to have a follow-up conference next year, and I hope to be able to report progress from there as well.

Yuval

Fulfilling your childhood dreams

2007-10-04T13:34:00.000-04:00

Make sure you view Randy Pausch's lecture "fulfilling your childhood dreams" about life, VR and everything in-between.

Motion Capture at Siggraph - Linking Real Motion to Virtual Worlds

2007-08-27T10:00:00.000-04:00

There were many motion capture demonstrations at the recent Siggraph show. Applications for motion capture include animated films, special-effects sequences, video games (e.g. capturing the throwing motion of a quarterback), academic studies in motion and more.

Most demonstrations involved models wearing body suits and moving in a large area with a uniform background. Real-time systems captured the movement and displayed it on computer screens.

A particularly exciting motion capture demonstration was staged at the WorldViz booth. Data from the motion capture system was fed into Vizard, the VR software platform from WorldViz and then used to drive the on-screen movement of an avatar. The avatar operated in a virtual world and interacted with it. In the particular Siggraph demo, the avatar was initially surrounded by many wooden boxes. By moving around, the model was able to rearrange the wooden boxes so as to show a compelling interaction between the virtual and real worlds. Moreover, the human model had two viewing options: view the complete scene on a projection screen (which drew a large crowd throughout the show), or wear an HMD (see image) and get a first-person experience as if being there. Incidentally, the software could also be configured to show the scene from another person's viewpoint, thus providing an "out of body" experience to the model who was able to view her own movements driving the avatar from a different viewpoint.

Are Existing Head-Mounted Displays ‘Good Enough’?

2007-08-01T08:53:00.000-04:00

During the spring of 2007, Sensics, my company, conducted a broad worldwide survey amongst academic, commercial, and government users of virtual reality systems. The survey was designed to understand user perceptions of current head-mounted display (HMD)
technology as well as desired performance characteristics of what was termed a 'good enough'
HMD.

Key survey findings include:

1. Most existing HMDs are not ‘good enough’ according to survey participants. Commonplace horizontal field of view (50 degrees or lower) and commonplace vertical field of view (30 degrees or lower) are considered ‘good enough’ by fewer than 10% of surveyed population.

2. The lack of ‘good enough’ performance is cited in practically all the cases where buyers with appropriate budgets considered purchasing head-mounted displays yet ultimately did not do so.

3. Users consider the most important HMD attributes to be: panoramic field of view (over 100 degrees horizontal), large vertical field of view (over 50 degrees), very fast dynamic response (no smear or fade effects), high contrast display, high resolution display and a lightweight design.

A full copy of the survey can be obtained by contacting Sensics. I think it's well worth a read.

Of Mice and Apes

2007-07-28T09:35:00.000-04:00

At Sensics, my company, we get our fair share of curious inquiries. It's one of the fun parts about this business: every day the phone rings with someone eager to use our panoramic head-mounted display for a unique application.

A few weeks ago, someone called to ask about the range of our IPD (Inter-Pupillary Distance). Just like binoculars the often allow you to adjust the distance between the eyepieces to accommodate different individuals, head-mounted displays allow the same.
- "55 to 72 mm", we said
- "can you do an IPD in the 30 mm range?", said the caller
- "???"
- "well, I want to use the HMD on apes"

Just a few weeks went by, and another person called. After discussing the merits of immersion - high resolution and panoramic field of view - the caller asked whether he can use the HMD on mice.
- "So you want to use the HMD on mice?"
- "Yes"
- "???"

The thought of a mice wearing an HMD triggered many interesting visuals. Maybe mice can also wear football helmets? Is there a market for mice-sized HMD?

As interesting as this proposed research might be, we're going to stick with providing human an unmatched visual experience.

Industrial VR in France and Japan

2007-07-06T06:19:00.000-04:00

We just returned from two exciting trade shows: the Paris Air Show (Le Bourget, France) and Industrial Virtual Reality in Tokyo, Japan. Hundreds of visitors came to experience how a high-performance head-mounted display enables engineers and designers to be immersed in their 3D data in the most realistic way possible (without actually building a physical version, of course).

In Tokyo, we were visited by Many of engineers and executives from some of Japan's most recognizable companies including Canon, Mitusubishi, Honda, Sony, NEC, Toyota, Nissan, Mazda, Fujitsu, Hitachi, Ricoh, Epson, Toshiba, Nikon, Fuji, Brother and more. Similarly, in Paris, big aerospace giants came to see how VR technology can make a difference on the effectiveness of their designs, training systems, and products. Amazingly, our visitors in Paris were able to concentrate on the demo, even when test pilots where performing air stunts with the newest fighter jets and passenger airplanes.

An HMD for every engineer is still a long way ahead of us, but the interest level and excitement that these industrial giants make us all believe that mainstream virtual reality will become reality.

Yuval

How Much Immersion is Enough?

2007-06-27T09:56:00.000-04:00

Doug Bowman and Ryan McMahan of Virgina Tech wrote an interesting article titled "How Much Immersion Is Enough" which appears as a cover feature of July's IEEE Computer magazine.

They discuss several aspects of Immersion (defined as "the objective level of sensory fidelity a VR system provides") and presence ("a user’s subjective psychological response to a VR system") and talk about the main factors that drive immersion, including field of view, field of regard, resolution, stereoscopy, head-tracking, frame rate and more.

The article describes several studies that investigate how the degree of immersion impacts the performance of a person trying to complete complex tasks (such as planning the path of an oil well or visualizing a tunnel through rock structures) and describes the significant and measurable benefits attributed to higher degree of immersion.

It is nice to see measurable benefits tied to the "wow" feeling that people experience when they try on immersive head-mounted displays that offer both panoramic field of view and high resolution. My company has built several high-immersion models by optically stitching together small micro-displays. One unique side-effect of this approach is that during demonstrations, we are able to turn off individual displays and thus take a user through a full range of immersion options (e.g. from 150 degree field of view to 120 to 100 to 80) within seconds. This has been an excellent way to experience the benefits of true immersion.

Once you've had a chance to try a "business class" seat in an airplane, it's not easy to go back to economy. Similarly, once you try on an HMD or a CAVE with wide FOV, it will be difficult to go back to narrow displays.

Yuval

HMDs and "the last mile"

2007-06-24T21:16:00.000-04:00

Remember the time that most home users could only use dial-up access to the Internet, even though the Internet backbone was very fast? These users could not enjoy the full Internet experience because of the bandwidth limitations in the "last mile" before their home.

Recent improvements in CPU and GPU power as well as rendering software, leads me to believe that the display quality of head-mounted displays could solve the "last mile" problem of bringing immersive virtual reality to the masses.

When I speak to audiences about head-mounted displays, I usually ask for a show of hands: "how many in the audience have Blockbuster or Netflx subscriptions or regularly rent movies?". A healthy majority usually raises their hands. "Now how many would still continue these subscriptions if all you had at home was a 3 inch black and white television?". Most hands come down. Without quality displays (with high resolution, wide field of view such as those offered by my company), the full potential of virtual reality will not be realized.

Yuval