VR and AR in 12 variations

I've been thinking about how to classify VR and AR headsets and am starting to look at them along three dimensions (no pun intended):

1. VR vs AR
2. PC-powered vs. Phone-powered vs. Self-powered. This looks at where the processing and video generation is coming from. Is it connected to a PC? Is it using a standard phone? Or does it embed processing inside the headset 
3. Wide field of view vs. Narrow FOV

This generates a total of 2 x 3 x 2 = 12 options as follows

Configuration Example and typical use 1: VR, PC-powered, Wide-field Examples: Oculus, HTC Vive, Sensics dSight, OSVR HDK. This immersive VR configuration is used in many applications, though the most popular one is gaming. One attribute that separated consumer-grade goggles like the HTC Vive from professional-grade goggles such as the Sensics dSight is pixel density: the number of pixels per degree. You can think about this as the diffence between watching a movie on a 50 inch standard-definition TV as opposed to a 50 inch HDTV. 2: VR, PC-powered, Narrow-field Example: Sensics zSight 1920. With a given number of pixels per eye, narrow-field systems allow for much higher pixel density, which allows observing fine details or very small objects. For instance, imagine that you are training to land a UAV. The first step in landing a UAV is spotting it in the sky. The higher the pixel density is, the farther out you can spot an object of a given size. The zSight 1920 has about 32 pixels/degree whereas a modern consumer goggle like the HTC Vive has less than half that. 3: VR, Phone-powered, Wide-field Examples: Samsung Gear VR, Google Cardboard, Zeiss VROne.. This configuration where the phone is inserted into some kind of holster is used for general-purpose mobile VR. The advantages of this configuration is its portability as well as its low cost - assuming you already own a compatible phone. The downside of this configuration is that the processing power of a phone is inferior to a high-end PC and thus the experience is more limited in terms of frame rate and scene complexity. Today's phones were not fully designed with VR in mind, so there are sometimes concerns about overheating and battery life. 4: VR, Phone-powered, Narrow-field Example: LG 369 VR. In this configuration, the phone is not carried on the head but rather connected via a thin wire to a smaller unit on the head. The advantage of this configuration is that it can be very lightweight and compact. Also, the phone could potentially be used as an input pad. The downside is that the phone is connected via a cable. Another downside is often the cost. Because this configuration does not use the phone screens, it needs to include its own screens that might add to the cost. Another advantage is that the phone camera can not be used for video see-through or for sensing. 5: VR, Self-powered, Wide-field Examples: Gameface Labs, Pico Neo. These configurations aim for standalone, mobile VR without requiring the mobile phone. They potentially save weight by not using unnecessary phone components such as the casing and touch screen, but would typically be more expensive than phone based VR for those users that already own the phone. They might have additional flexibility with regards to which sensors to include, camera placement and battery location. They are more difficult to upgrade relative to a phone-based VR solution, but the fact that the phone cannot be taken out might be an advantage for applications such as public VR where a fully-integrated system that cannot be easily taken apart is a plus. 6: VR, Self-powered, Narrow-field Example: Sensics SmartGoggles.  These configurations are less popular today. Even the Sensics SmartGoggles which included on-board Android processor as well as wide-area hand sensors was built with relatively narrow field of view (60 degrees) because of the components available at the time. 7: AR, PC-powered, Wide-field Example: Meta 2. In many augmented reality applications, people ask for wide field so that, for instance, a virtual object that appears overlaid on the real world does not disappear when the user looks to the side.  This configuration may end up being transient because in many cases the value of augmented reality is in being able to interact with the real world, and the user's motion when tethered to a PC is more limited. However, one might see uses in applications such as engineering workstation. 8: AR, PC-powered, Narrow-field I am not aware of good examples of this configuration. It combines the limit of narrow-field AR with the tether to the PC. 9: AR, Phone-powered, Wide-field This could become one of the standard AR configuration just like phone-powered, wide-field VR is becoming a mainstream configuration. To get there, the processing power and optics/display technology catch up with the requirements. 10: AR, Phone-powered, Narrow-field Example: Seebright. In this configuration, a phone is worn on the head and its screen becomes the display for the goggles. Semi-transparent optics combine phone-generated imagery with the real world. I believe this is primarily a transient configuration until wide-field models appear. 11: AR, Self-powered, Wide-field I am unaware of current examples of this configuration though one would assume it could be very attractive because of the mobility on one hand and the ability to interact in a wide field of view. 12: AR, Self-powered, Narrow-field Examples: Microsoft Hololens, Google glass, Vuzix M300. There are two types of devices here: one is an 'information appliance' like Google Glass, designed to provide contextually-relevant information without taking over the field of view. These configurations are very attractive in industrial settings for applications like field technicians, workers in a warehouse or even customer service representatives needing a mobile, wearable terminal often to connect with a cloud-based database. The second type of device, exemplified by the Hololens seeks to augment the reality by placing virtual objects locked in space. I am sure the Hololens would like to be a wide-field model and it is narrow field at the moment because of the limitations of its current display technology

Looking forward to feedback and comments.

Everyman's VR: Bridging the Gap between $15 and $1500 for VR Goggles

There is a gap in VR pricing that could slow down the adoption of VR.

Today, you can spend about $15 and buy a Cardboard-type phone attachment to get simple VR with your phone. You can also spend $1500 and in a few months you will get a nice computer and what would be considered today a high-performance consumer VR experience, touch controller not included. As I write this, the price for an HTC Vive has not been published - perhaps it will be even higher.

$1500 is a lot of money. Maybe I just don't have $1500 available to spend. Maybe I spent $900 a year ago on a nice computer and I don't want to buy a new one. Maybe I'm running a classroom and we just can't afford a $1500 machine on every desk? Maybe I worry that VR is moving so fast that the $1500 VR system I buy in 2016 will be obsolete in 2017?

$15. $1500. Excluding Gear VR ($100 if you already have a fairly new Samsung phone), is there any good solution in-between? How can we bridge the gap to allow "Everyman's VR"?

A few things could - and perhaps should - happen:

• HMD vendors would select different trade-offs geared towards different use cases (media viewing, casual gaming) and offer them at lower price points. 
• Software platforms, like OSVR, would continue to allow games to modulate the required computing power. For instance, the Unreal Showdown demo on OSVR runs nicely on an NVIDIA 6xx card.
• Benchmark utilities would allow users to determine how fast a game can run on their PC and have the game adjust to it. Can't run Elite: Dangerous at 90 FPS? Run it at 45 FPS with time-warping or reduce the resolution and image quality.
• Eye trackers embedded inside goggles allow using foveated rendering, which is the process of doing high-quality rendering at the portion of the image you are directly looking at and lower quality rendering elsewhere. This would save GPU resources and allow running on lower-end PCs. Similarly, VR driver optimizations from NVIDIA/AMD/Intel could make better use of GPU resources.

This is where OSVR can make a big impact. I'm not focused so much on the OSVR HDK ($300 at osvr.org), but rather on the OSVR software platform. A middleware platform like OSVR allow running the same application at different performance levels on a wide range of devices, with striving towards nearly universal compatibility. If an HMD vendor knew that lots of compelling content could be immediately available to their customers if they built an OSVR plugin, many HMD variations are likely to appear.

Does this remind you of something? Think about what Android did to the phone and tablet market. It created choice. It allowed tablets and phone to appear at all kinds of performance and price specs, all running off a common software platform. We saw high-end phones like the Samsung S6 or LG G4 all the way to low-end generic phones. Could a $50 tablet run all the apps that a high-end Android tablet runs? No, but for some users the $50 tablet is perfectly adequate.

There is a wide gap between $15 and $1500 and there are many people that would like to experience VR that's better than $15 but prefer not to spend $1500. Let's work together to fill this gap and create "Everyman's VR"

Phone-based VR without the Phone

There are two main VR configurations today: PC-based VR (e.g OSVR HDK, HTC VIVE, etc.) and phone-based VR (e.g. Samsung Gear VR, Google Cardboard, etc.). Each approach carries different advantages: PC-based VR allow using high-power graphics cards; Phone-based VR is highly portable and battery-operated.

The rationale behind phone-based VR solutions is two-fold:

1. If you already have a phone, the incremental investment for a VR experience is low. Gear VR is under $100 now, and Cardboard is much cheaper. For casual VR, popping a phone in a carrier is a convenient solution.
2. Today's phones have high-resolution screens, integrated cameras, increasingly better motion sensors and many additional capabilities that are very useful for VR. Because they are mass-produced, they are very cost-effective.

However, upon closer investigation it appears that a user of phone-based VR is paying - both in cost and in weight - for phone components that are not truly necessary for VR. For instance one could argue that the following components are not truly necessary:

• Cellular connectivity
• Phone body. If the phone is permanently integrated into the cradle, one does not need the weight of the body itself.
• Touch screen
• and more...

Now that VR is gaining steam, VR devices are going to have modules and components that are designed specifically for VR instead of settling for parts from other electronics devices. For instance we'll see screens made for VR (small size, higher refresh rate, low persistence). Given this, it would not be surprising to start seeing phone vendors provide pre-integrated VR goggles that are essentially a phone without the unnecessary components coupled with a cradle/head strap. These would be lighter than separate phone + cradle and probably also less expensive than the combination.

What else would you like to see in 2016?

Beyond gaming: virtual reality helps people with vision disabilities

Over the past two years, Sensics has been working with our customer Visionize and a group of researchers from the Wilmer Eye Institute at Johns Hopkins University on applying the group's combined expertise towards creating a solution to help people with vision disabilities. The Los Angeles Times published a story today about one of the models in this line. It's a good opportunity to describe this Low Vision project in greater detail and shine the light on non-gaming application of consumer VR.

Prototype of Visionize low-vision system is used to magnify the image of the boy above the monitor. The monitor - used for illustration purposes only - shows a screencast of what the user of the system sees in each eye

Low vision is a common problem. It is estimated that there are about 2.5 million people in the United States - so over 0.75% of the population - that suffer from low vision (defined as best corrected visual acuity less than 20/60 in the better seeing eye). While low vision is typically associated with aging, there are also a large number of kids who are born with vision disabilities or develop them in their early years. Additionally, hundreds of thousands additional patients enter the low-vision population every year.

The impact of low vision ranges from difficulty in reading to difficulties in recognizing people, places and objects. Disease progression can often be controlled, but the existing damage is permanent. Macular Degeneration, a disease that destroys the area of central vision (the fovea), is the most common low vision pathology. Because the resolution at the fovea is higher than the rest of the eye, the overall visual acuity is reduced.

Optical or digital magnifiers are popular with the low vision population and can be effective for static activities such as watching TV or reading. However, they are more challenging for use in dynamic activities such as walking:

A magnifier hides part of the text

• They might be too large or cumbersome to hold
• If they magnify the entire visual field, the user loses peripheral vision in a significant way. For instance, if the person using a 5x magnifier has a total field of vision of 100 degrees and if the magnifier covered their entire visual field, then only 20 degrees of the real world would be mapped into this 100 degrees, preventing effective movement.
• If just part of the image is magnified, there is part of the scene that is completely hidden underneath the magnification, as is illustrated in the diagram on the right

To address this problem, we developed a non-linear magnification algorithm that magnifies the image at the point of interest but creates a continuous image so that nothing is lost at the edges. In the model covered in the LA Times, a Samsung Gear VR system is used. The on-board camera provides a live view of the environment and the customized algorithms perform real-time 60 FPS enhancement to present a smartly-magnified image to the user. Parameters such as size and amount of magnification of the "bubble" can be easily controlled. In some cases, these depend on the viewing conditions and in others they can be customized to the particular vision disability of the user.

More advanced models use different types of HMDs and have been tested with multiple cameras and other powreful additions. More about this as well as additional vision enhancements will perhaps be covered at a future opportunity.

An illustration of the magnified bubble can be seen in the diagram below:

and a video illustrating operating an ATM machine with the system (as well as other examples) can be seen in the Visionize site:

These days, gaming gets the majority of the press attention for virtual reality, but other applications exist. For us, the ability to work on a product that truly improves the quality of life for people with vision disabilities, is truly heart-warming.