VR and AR in 12 variations

I've been thinking about how to classify VR and AR headsets and am starting to look at them along three dimensions (no pun intended):

1. VR vs AR
2. PC-powered vs. Phone-powered vs. Self-powered. This looks at where the processing and video generation is coming from. Is it connected to a PC? Is it using a standard phone? Or does it embed processing inside the headset 
3. Wide field of view vs. Narrow FOV

This generates a total of 2 x 3 x 2 = 12 options as follows

Configuration Example and typical use 1: VR, PC-powered, Wide-field Examples: Oculus, HTC Vive, Sensics dSight, OSVR HDK. This immersive VR configuration is used in many applications, though the most popular one is gaming. One attribute that separated consumer-grade goggles like the HTC Vive from professional-grade goggles such as the Sensics dSight is pixel density: the number of pixels per degree. You can think about this as the diffence between watching a movie on a 50 inch standard-definition TV as opposed to a 50 inch HDTV. 2: VR, PC-powered, Narrow-field Example: Sensics zSight 1920. With a given number of pixels per eye, narrow-field systems allow for much higher pixel density, which allows observing fine details or very small objects. For instance, imagine that you are training to land a UAV. The first step in landing a UAV is spotting it in the sky. The higher the pixel density is, the farther out you can spot an object of a given size. The zSight 1920 has about 32 pixels/degree whereas a modern consumer goggle like the HTC Vive has less than half that. 3: VR, Phone-powered, Wide-field Examples: Samsung Gear VR, Google Cardboard, Zeiss VROne.. This configuration where the phone is inserted into some kind of holster is used for general-purpose mobile VR. The advantages of this configuration is its portability as well as its low cost - assuming you already own a compatible phone. The downside of this configuration is that the processing power of a phone is inferior to a high-end PC and thus the experience is more limited in terms of frame rate and scene complexity. Today's phones were not fully designed with VR in mind, so there are sometimes concerns about overheating and battery life. 4: VR, Phone-powered, Narrow-field Example: LG 369 VR. In this configuration, the phone is not carried on the head but rather connected via a thin wire to a smaller unit on the head. The advantage of this configuration is that it can be very lightweight and compact. Also, the phone could potentially be used as an input pad. The downside is that the phone is connected via a cable. Another downside is often the cost. Because this configuration does not use the phone screens, it needs to include its own screens that might add to the cost. Another advantage is that the phone camera can not be used for video see-through or for sensing. 5: VR, Self-powered, Wide-field Examples: Gameface Labs, Pico Neo. These configurations aim for standalone, mobile VR without requiring the mobile phone. They potentially save weight by not using unnecessary phone components such as the casing and touch screen, but would typically be more expensive than phone based VR for those users that already own the phone. They might have additional flexibility with regards to which sensors to include, camera placement and battery location. They are more difficult to upgrade relative to a phone-based VR solution, but the fact that the phone cannot be taken out might be an advantage for applications such as public VR where a fully-integrated system that cannot be easily taken apart is a plus. 6: VR, Self-powered, Narrow-field Example: Sensics SmartGoggles.  These configurations are less popular today. Even the Sensics SmartGoggles which included on-board Android processor as well as wide-area hand sensors was built with relatively narrow field of view (60 degrees) because of the components available at the time. 7: AR, PC-powered, Wide-field Example: Meta 2. In many augmented reality applications, people ask for wide field so that, for instance, a virtual object that appears overlaid on the real world does not disappear when the user looks to the side.  This configuration may end up being transient because in many cases the value of augmented reality is in being able to interact with the real world, and the user's motion when tethered to a PC is more limited. However, one might see uses in applications such as engineering workstation. 8: AR, PC-powered, Narrow-field I am not aware of good examples of this configuration. It combines the limit of narrow-field AR with the tether to the PC. 9: AR, Phone-powered, Wide-field This could become one of the standard AR configuration just like phone-powered, wide-field VR is becoming a mainstream configuration. To get there, the processing power and optics/display technology catch up with the requirements. 10: AR, Phone-powered, Narrow-field Example: Seebright. In this configuration, a phone is worn on the head and its screen becomes the display for the goggles. Semi-transparent optics combine phone-generated imagery with the real world. I believe this is primarily a transient configuration until wide-field models appear. 11: AR, Self-powered, Wide-field I am unaware of current examples of this configuration though one would assume it could be very attractive because of the mobility on one hand and the ability to interact in a wide field of view. 12: AR, Self-powered, Narrow-field Examples: Microsoft Hololens, Google glass, Vuzix M300. There are two types of devices here: one is an 'information appliance' like Google Glass, designed to provide contextually-relevant information without taking over the field of view. These configurations are very attractive in industrial settings for applications like field technicians, workers in a warehouse or even customer service representatives needing a mobile, wearable terminal often to connect with a cloud-based database. The second type of device, exemplified by the Hololens seeks to augment the reality by placing virtual objects locked in space. I am sure the Hololens would like to be a wide-field model and it is narrow field at the moment because of the limitations of its current display technology

Looking forward to feedback and comments.

Diagramming the VR Stack

While preparing for last week's Consumer Electronics Show, I created a simple diagram to explain and VR application stack and how it related to the work that we are doing with OSVR. I thought it might be interesting to share this diagram and explain its contents:

The VR Stack diagram, Copyright (c) Sensics, Inc.

At the risk of contradicting Copernicus, we see the HMD at the center. Without the display, whether VR or AR, there is not much of an experience.

The display by itself does not make the experience, just like a game console without game controllers are not very interesting. Thus, the HMD is surrounded by a variety of sensors and output devices. These are often orientation and position trackers, input devices, but could also be eye trackers, motion platforms, gesture and skeleton sensors, depth sensors and more.

Moving for a moment to the outer circle, there are games and other applications. These needs to be distributed (and often monetized), but let's leave the distribution discussion for another time.

Very often, these applications are written using graphics or game engines. They provide a wide range of services: from assets to lighting to physics and more. These engines - such as Unity, Unreal, Cryengine or other are shown in the next to last circle.

To me, what is often overlooked in VR is the need for a middleware layer - a layer that connects the various engines to the various sensors and peripherals. I submit that no one likes to write applications that work only on one device and because of that, a layer that allows a game engine to connect to many different HMDs, to many different eye trackers, to many different sensors is critical. Think about your computer at home. Does everyone use the same monitor? Keyboard? Mouse? Printer? Not everyone would want to use the same HMD. That's why middleware is required and an excellent example of middleware is the OSVR software platform supporting hundreds of devices, peripherals and game engines. All layers are important, so don't forget the middleware!

The post "Diagramming the VR Stack" first appeared on "Virtual Reality and Head Mounted Displays"