Got new or custom HMD? Need direct render, time warping, distortion correction and game engine integration?

Google Concept from Sensics

Let's assume you built a new HMD. You've done all the hardware work - figured out the optics and display setup, created driver boards, integrated a motion tracker and perhaps an eye tracker, worked out the ergonomics issues. It's a lot of work and you should be commended on doing it.

How do you get software support for it? Specifically, you probably want to:
- Model the field of view and allow users to correct any optical or color distortion
- Obtain support for direct render and asynchronous time warp
- Get Unity and Unreal drivers for your trackers
- Get some demo content and some cool games running on it
- and, in general, make it easy for others to support your new creation.

 A solution to consider is OSVR. By integrating your new HMD into the open-source OSVR framework, you can get all that done (and more) very quickly. Get low-latency rendering for your HMD; correct distortion in one of several possible ways; support for many game engines; debug and demonstration software.

OSVR already supports many devices (full list here) and based on the work team Sensics did with several HMD vendors over the last few months, we put up documentation on how to add an HMD to OSVR. You can find it here and it is part of the official OSVR developer documentation repository.

Let me know what you think and how we can make it easier to get this done.

Snow World

As I write this, not far from Washington, DC, a major snow storm is raging outside. Forecasts call for over 20 inches (50 cm) of snow on the ground, and current accumulation is already not far from that.

With all this snow outside, I can't help but remember Snow World, a VR pain management experience that was one of first useful non-gaming VR applications. Developed many years ago at the University of Washington by Dr. Hunter Hoffman, Dr. David Patterson and a team from Firsthand, SnowWorld is a simple game that takes place in an icy canyon where users throw snow balls at snowmen and penguins, Clinical trials have shown that when burn victims use this game, patients report a dramatic reduction in pain.


See this report from NBC News about the experience:

Visit NBCNews.com for breaking news, world news, and news about the economy

The ability of VR to 'trick' the brain is being used not just for therapy - just like in snow-world - but also in other areas. For instance, Redirected Walking uses subtle visual cues to make people walk in circles inside a limited space, even though they think they are walking in a straight line. This technique is being used in The Void experience today. Redirected walking allows experiencing a large virtual world in spite of being constricted to a smaller physical space, helps avoid physical obstacles and It allows multiple people to be immersed in the same physical space without bumping into each other.

Similarly, the usefulness of VR in therapy is not limited to pain management but extends to other areas such as PTSD, fear of flying, fear of heights, fear of public speaking and much, much more.

VR gaming is great, but VR applications like Snow World can have a more important impact on those that need it.

Everyman's VR: Bridging the Gap between $15 and $1500 for VR Goggles

There is a gap in VR pricing that could slow down the adoption of VR.

Today, you can spend about $15 and buy a Cardboard-type phone attachment to get simple VR with your phone. You can also spend $1500 and in a few months you will get a nice computer and what would be considered today a high-performance consumer VR experience, touch controller not included. As I write this, the price for an HTC Vive has not been published - perhaps it will be even higher.

$1500 is a lot of money. Maybe I just don't have $1500 available to spend. Maybe I spent $900 a year ago on a nice computer and I don't want to buy a new one. Maybe I'm running a classroom and we just can't afford a $1500 machine on every desk? Maybe I worry that VR is moving so fast that the $1500 VR system I buy in 2016 will be obsolete in 2017?

$15. $1500. Excluding Gear VR ($100 if you already have a fairly new Samsung phone), is there any good solution in-between? How can we bridge the gap to allow "Everyman's VR"?

A few things could - and perhaps should - happen:

• HMD vendors would select different trade-offs geared towards different use cases (media viewing, casual gaming) and offer them at lower price points. 
• Software platforms, like OSVR, would continue to allow games to modulate the required computing power. For instance, the Unreal Showdown demo on OSVR runs nicely on an NVIDIA 6xx card.
• Benchmark utilities would allow users to determine how fast a game can run on their PC and have the game adjust to it. Can't run Elite: Dangerous at 90 FPS? Run it at 45 FPS with time-warping or reduce the resolution and image quality.
• Eye trackers embedded inside goggles allow using foveated rendering, which is the process of doing high-quality rendering at the portion of the image you are directly looking at and lower quality rendering elsewhere. This would save GPU resources and allow running on lower-end PCs. Similarly, VR driver optimizations from NVIDIA/AMD/Intel could make better use of GPU resources.

This is where OSVR can make a big impact. I'm not focused so much on the OSVR HDK ($300 at osvr.org), but rather on the OSVR software platform. A middleware platform like OSVR allow running the same application at different performance levels on a wide range of devices, with striving towards nearly universal compatibility. If an HMD vendor knew that lots of compelling content could be immediately available to their customers if they built an OSVR plugin, many HMD variations are likely to appear.

Does this remind you of something? Think about what Android did to the phone and tablet market. It created choice. It allowed tablets and phone to appear at all kinds of performance and price specs, all running off a common software platform. We saw high-end phones like the Samsung S6 or LG G4 all the way to low-end generic phones. Could a $50 tablet run all the apps that a high-end Android tablet runs? No, but for some users the $50 tablet is perfectly adequate.

There is a wide gap between $15 and $1500 and there are many people that would like to experience VR that's better than $15 but prefer not to spend $1500. Let's work together to fill this gap and create "Everyman's VR"

Diagramming the VR Stack

While preparing for last week's Consumer Electronics Show, I created a simple diagram to explain and VR application stack and how it related to the work that we are doing with OSVR. I thought it might be interesting to share this diagram and explain its contents:

The VR Stack diagram, Copyright (c) Sensics, Inc.

At the risk of contradicting Copernicus, we see the HMD at the center. Without the display, whether VR or AR, there is not much of an experience.

The display by itself does not make the experience, just like a game console without game controllers are not very interesting. Thus, the HMD is surrounded by a variety of sensors and output devices. These are often orientation and position trackers, input devices, but could also be eye trackers, motion platforms, gesture and skeleton sensors, depth sensors and more.

Moving for a moment to the outer circle, there are games and other applications. These needs to be distributed (and often monetized), but let's leave the distribution discussion for another time.

Very often, these applications are written using graphics or game engines. They provide a wide range of services: from assets to lighting to physics and more. These engines - such as Unity, Unreal, Cryengine or other are shown in the next to last circle.

To me, what is often overlooked in VR is the need for a middleware layer - a layer that connects the various engines to the various sensors and peripherals. I submit that no one likes to write applications that work only on one device and because of that, a layer that allows a game engine to connect to many different HMDs, to many different eye trackers, to many different sensors is critical. Think about your computer at home. Does everyone use the same monitor? Keyboard? Mouse? Printer? Not everyone would want to use the same HMD. That's why middleware is required and an excellent example of middleware is the OSVR software platform supporting hundreds of devices, peripherals and game engines. All layers are important, so don't forget the middleware!

The post "Diagramming the VR Stack" first appeared on "Virtual Reality and Head Mounted Displays"

Conversation with Kevin Williams on Digital Out of Home VR Entertainment

My conversation with Kevin Williams now available for listening or reading here

Kevin and I discuss using virtual reality technology for digital out of the home entertainment such as theme parks. Can consumer HMDs be used for commercial uses? Are there special requirements? Are there existing installations of public VR and much more.

English born Kevin Williams has spent most of his career in the electronic entertainment industry, first as a fascinated player of the fledgling video amusement, then as an amateur computer devotee. At present he is a leading member of the emerging Digital Out-of-Home (DOOH) Leisure entertainment industry, both as consultant, observer and founding chairman of the sector association.

You can also subscribe to the podcast on iTunes and on Android

You can find many additional interviews that I did with CEOS, analysts and other interesting people by selecting the "interviews" category in the Sensics insight page

New Year "Resolutions", part 2

A couple of years ago I wrote a blog post 'Resolutions for the New Year' which described upcoming 2560x1440 displays and how they impact virtual reality. I think it's time to revisit that discussion.

Back in 2014, HMD makers were happy to use displays made for smartphones because that was the best way to get inexpensive, high-resolution displays that were small enough to be used in HMDs. Some of these displays made it possible to implement low-persistence schemes (OLED displays, strobing) and some could be over-driven beyond the standard 60 FPS refresh rate.

Now that virtual reality is coming into the mainstream, we are seeing that it is starting to make sense for screen manufacturers to build screens where VR is the main goal, not a secondary afterthought. As a result, we start seeing a divergence between the specifications of smartphone screens and of VR-specific screens. Let's consider this along three dimensions: higher resolutions, physically smaller screens and flexible screens.

Higher resolutions

The 2560x1440 screens that I wrote about 2 years ago became commonplace in phones like the Samsung Galaxy S6 and LG G3. These have become popular in phone-based VR products such as the Samsung Gear VR.

The next step-up in resolution could be 3840x2160, but many have argued that aside from using the phone for VR, there is very little end-user benefit to having a 5-6" phone with such high resolution. Thus, first point of divergence where what's good for VR is not very useful for a regular phone.

For VR, if we use a standard phone-based VR configuration where the screen is divided into two and a lens provides approx 90 degree horizontal field of view from each half, a screen with 3840 pixels across would provide 1920 pixels per eye or approximately 21 pixels/degree. This is about the pixel density that professional-grade HMDs delivered 5-10 years ago. For comparison purposes, "eye limiting resolution" beyond which increased resolution isn't noticeable is considered to be about 60 pixels/degree, so there is still a long way to go before photo-realistic VR with wide field of view.

Smaller screens

The downside of basing an HMD on a single 5.5" phone screen is that it physically looks like you are wearing a phone screen, so the designs are a bit more bulky than desired. If, instead, displays with a similar pixel density appeared in a smaller size, say 3.5", these displays would open up several opportunities if two such displays - one per eye - is being used:

• Greater flexibility for industrial designers
• The ability to deliver an convenient IPD mechanism to accommodate the distance between the eyes. One very convenient feature of a 5.5" screen is that it works well for nominal IPD. For instance, the AUO H546DLB01 1920x1080 OLED screen that is used in the OSVR HDK has an active area of approx 68 x 121 mm. If we assume one half of the screen is used for each eye, and if we assume that the optics are designed to be centered on precisely the middle of each half, than the distance between the optical centers is about 60.5 mm, which is not far from an average IPD of about 63mm. However, a single screen make it difficult to increase or decrease the IPD because you "run out of screen" if you move the optics farther from each other. However, if a separate 3.5" screen was dedicated to each eye, you could move the screen and the optics together and easily accommodate larger IPD. 
• Ability of both screens to update together. if a single 5.5" screen is updated left to right at 80 FPS, each frame lasts 12.5 mSec and thus the middle of the screen lags behind the beginning of the screen by about 6.25 mSec, so the software has to consider this when deciding what to render for each eye. However, if the two screens are separate, they can start their update at the same time.
• Opportunity to tilt each screen outwards, thus generating wider binocular field of view at the expense of generating partial overlap.
• Opportunity to tile multiple screens to achieve even wider field of view and higher resolution, similar to what Sensics has done many years ago with micro-displays

For completeness, it is worth mentioning that direct-to-eye DLP technologies are also beginning to provide wider field of view for high resolutions, so phone screens or their derivatives are not the only options.

Flexible screens

If a screen was very flexible, it could generate some value for phones: it could allow a curved edge of a phone; it could allow slightly curved phones that fit the face better (like the LG G Flex phones) and it might be better protected from shock (like the Motorola Droid Turbo 2). However, a flexible screen could generate a lot of value for HMDs as it would allow curving the screen around the head. This could generate two benefits:

• Better aesthetics
• Wider field of view. If one desires to get closer and closer to truly panoramic field of view - say 180 degrees - a screen that wraps around the head will make that easier, though special optics will need to be designed. This approach is reported to be a key part of the HMD being developed by The Void

The new displays bring new challenges together with the new opportunities. One such challenge is optimizing the optical design to these newer configurations (though companies like Sensics can help in this effort). But in all, as HMDs move from the background to the forefront in terms of an interesting market, we will this continue to drive display innovation.