What every VR game engine needs

When game engines are used for VR, they have to include many new capabilities: stereo rendering, higher frame rate, distortion correction, latency control and more. But one topic that is often overlooked is that VR game engine also have to deal with a wide variety of VR peripherals, each with their own API.

A non-VR engine primarily interfaces with a game controller, keyboard and mouse. These are well-understood peripherals with standard software interfaces. New types of keyboards, controllers and mice do appear on the market, but they share essentially identical interfaces to pre-existing devices.

In contrast, there is a much greater variety of VR devices: HMDs, motion and position trackers, hand and finger sensors, eye trackers, body suits, locomotion devices, force feedback devices, augmented reality cameras and more. Furthermore, each class of devices does not offer a common interface: working with a Leap Motion sensor is different than a Softkinetic one or an Intel Realsense one, even though the capabilities that they provide are similar.

The result? An endless effort to keep up. Consider the diagram below, showing just a small selection of available VR engines and a small subset of VR devices

VR devices and VR engines face a constant struggle to support each other

Consider a Sixense STEM motion controller. The Sixense team would probably like to make the STEM available to a widest possible range of software engines so that a developer can use STEM regardless of their engine of choice. Same goes for every other peripheral. Conversely, an Unreal Engine developer wants to support the maximum reasonable number of devices so that the game can reach the maximum number of users. Every permutation of VR device and game engine may have value. For the device vendor, there is a need how to build, test, optimize and deploy plugins for various engines. For the engine vendor, there is constant pressure - from both game developers and hardware makers - to offer device support.

That's why every VR game engine needs a middleware abstraction layer like the OSVR SDK. OSVR factors devices into common interfaces - tracker, skeleton, imager, etc. - and then provides a standard device-independent interface to the game engine. Just like desktop scanners offer a TWAIN interface that applications can use regardless of the scanner vendor, OSVR offers optimized skeleton, eye tracker and other VR interfaces that work regardless of the underlying hardware.

VR Middleware like OSVR dramatically simplifies the device and engine support effort

Consider the advantages. With OSVR:

• A device vendor writes a single interface to OSVR and immediately gains access to high-performance plugins to a wide range of game engines. In the particular case of OSVR, these plugins are free and open-source.
• Instead of attempting to work with hundreds and vendors and devices, an engine company can work with the OSVR team to create an optimized interface for each particular type of device. As new devices of the same type enter the market, supporting this new device is as easy as writing an OSVR plugin. Engine companies do not need to go over the effort of teaching yet another hardware vendor how to work well with their engine. The effort can be focused on optimizing the OSVR driver.
• Game developers are spared the effort of selecting particular vendors. By using the OSVR plugin for their favorite game engines, a wide range of devices is supported out of the box. 
• As new devices enter the market, games do no need to recompile and redistribute. In fact, that game might not even know that it is now supporting a new hardware vendor
• OSVR interfaces are defined collectively by industry leaders that understand the power of the abstraction layer. This is done in a transparent and open-source way.
• The number of engine and device combination increases, while the number of software interfaces to be written, debugged and optimized significantly decreases.

Game engines could still use vendor-specific function calls to activate some special functions of a peripheral, but the need to so quickly decreases.

Yet another advantage is the ability to add in-line analysis plugins to the data path. For instance, often times it is desired to convert hand and finger movements into higher-level gestures. The gesture engine could be vendor-specific, but that would limit its adoption in the game. It could be game-engine specific but that would limit market adoption of gestures as a user interface method. Or, it could be an OSVR plugin that works on top of all existing hand and finger sensors (using their standardized interface) and provides a standard gesture interface to many game engines. Furthermore, this gesture engine can be written by a company or a group that are experts in gestures, instead of forcing a hardware vendor or a game engine to master yet another technology.

Whether you make engines, hardware or smart analysis software, using VR middleware such as the OSVR API deserves serious consideration.

Open Source Augmented Reality?

Illustration from Meta SpaceGlasses

Can the OSVR project be used not just for virtual reality but also for augmented reality? Absolutely! Here why:

OSVR includes two independent parts: an open-source HMD (the "Hacker Development Kit") and the OSVR framework, a free and open-source software platform that provides an easy and standard way to discover, configure and operate a wide range of virtual reality and augmented peripherals.

Examples of such peripherals could be head trackers, hand and finger sensors (like Leap Motion and SoftKinetic), gesture control devices (such as the Myo armband and the Nod ring), cameras, eye trackers and many others. It turns out that most of these devices are used not only in virtual reality applications but also in augmented reality applications and so the services that the OSVR framework provides are just as useful for AR as they are for VR.

What are these services that OSVR provides? There are many different services, but some of them are:

• Discovery. Determine what peripherals and devices are connected to the system
• Configuration and autoconfiguration. Configure each peripheral and the interconnection between them. 
• Operation. Extract data and events from the peripherals and pass it on to the application in a standardized way. Support both a state (synchronous) and an event (asynchronous) programming model or any combination thereof.
• Store and restore configuration to/from a file or the cloud. 
• Provide user parameter such as eye separation.
• Provide optimized connectors to popular engines such as Unity and Unreal.
• Support of multiple operating systems and hardware platforms.

For instance, if you are developing an application that uses the Leap Motion sensor but also want users to operate it with the SoftKinetic camera, you have two options:

Option one: do it yourself and learn the individual APIs of each of these cameras and create multiple pieces of code to support them. Then, as these APIs evolve over time, you have to continuously upgrade your application. After a while, you want to use yet a third type of camera, say the Intel RealSense, so now you need to add yet another branch of code. Of course, each of these devices may report data differently. They may use different coordinate systems, or different units. You application needs to handle all these configurations.

Option two: use OSVR. You need to learn just one API which abstracts the various types of cameras. As new cameras are added, the hardware vendor (or the OSVR community) are likely to quickly add a new plugin to OSVR, meaning that your application works the same way with just a download of a new OSVR plugin. If the community does not work fast enough, OSVR is open-source and well documented so that you can write a plugin yourself. Coordinate systems and units are consistent. You are not strongly dependent on one particular piece of hardware.

OSVR also provide a growing list of analysis plugins that latch onto the output of device plugins and provide high-value processing such as a gesture engine (converting motion into gestures), data smoothing, target recognition and more. 

Next time you think about our AR software architecture, think OSVR. 

OSVR at IEEE VR

Arles (image source: Wikipedia)

I'm heading to the city of Arles in the south of France (someone had to go, right?) to participate in the IEEE VR conference and exhibition. This would probably be my sixth or seventh IEEE VR but going to it the feeling and the goals are different.

What's unique at IEEE VR is the it is first and foremost an academic conference, not a VR exhibition. Hundreds and hundreds of researchers (many of which are Sensics customers) come to share, learn and discuss their research, experience cutting-edge demos that are not yet mature enough to show up at a GDC or CES. Because of the renewed interest in VR, I'm sure there will quite a few corporate visitors that were missing from previous years and wish to pick up trends, technologies and partners.

I am chairng a panel discussion on the resurgence of open-source VR. My co-panelists (Sebastien Kuntz from MiddleVR, Goeffrey Subileau from Dassault Systemes and Bill Sherman from the Desert Research Institute) will seek to answer several questions including:

• What’s new (relative to 1-2 years ago) in open-source and closed-source VR software
• When should I use closed-source and when should I use open-source?
• Should I contribute to open-source projects, and if so, why?
• What’s missing in current open-source VR?
• Is there an opportunity to combine open-source and closed-source frameworks

and of course we will take questions from the audience.

Later in the conference, Dr, Ryan Pavlik and I will be presenting a technical overview of OSVR, targeting both industry and academia.

When not in session, we will be demonstrating the OSVR HDK and the Sensics dSight at the exhibit area.

But what I am most interested in doing at the conference is listening. I want to hear about all the great research that is out there. I want to have in-depth conversations with people who might want to become OSVR partners, whether it is to hack the OSVR HDK, write a smart software plugin for the software platform or create some new kind of VR experience.

If you are going, look me up! If you're not, stay tuned on these pages for what I found at the show.

A brief overview of the OSVR source-code repositories

The OSVR team opened up most of the source code repositories to the public this weekend, and a few additional repositories will be opened in the coming days. Because Sensics is a founding contributor to this open-source VR project, many have asked us for a brief overview of the project.

The best place to start is osvr.github.io. If you haven't read the 'introduction to OSVR whitepaper' you might want to do so.

There are several github projects under the /osvr organization. They are as follows:

Key projects:

• OSVR-Core : this is the heart of the project. The OSVR_server executable connects the game to the OSVR hardware and software components.

Utilities:

• OSVR-Tracker-Viewer is a utility that graphically shows the position and orientation of the head and hand controllers. It is also an OSVR client, and thus an example on how to connect to and extract data from the server
• Distortionizer: a utility to estimate distortion correction parameters for various HMDs and a shader to implement the parameters estimated by the Distortionizer. OSVR has JSON descriptor files for HMDs (and many other objects) and the distortion parameters are part of that JSON file

Game engine plugins:

• OSVR-Unity includes a prefab component that can be imported into Unity. 
• OSVR-Unreal (to be released later this week) is an Unreal Engine plugin

Development tools:

• OSVR-Boxstarter is a Boxstarter install that helps quickly set up a development environment on a Windows machine
• OSVR-JSON-Editor is the source code for a tool (deployed version here) that helps create and edit the JSON descriptor files
• OSVR-JSON-Schemas is a repository for such JSON files

Plugins:

• OSVR-Oculus-Rift provides a plugin that allows using the position and orientation data of an Oculus device inside OSVR. 
• OSVR-Vuzix (to be released later this week) does the same for Vuzix headsets

Additional projects are coming. There is also a wiki page. Issues are currently tracked as part of the Github pages of the projects and we are looking to add an open-source project management tool.

OSVR (licensed under Apache 2.0 license) aims to create a multi-platform framework to detect, configure and operate a wide range of VR devices as well as to allow smart plugins that turn data into useful information. This is a big undertaking and there is a lot more work to be done to fulfill that vision, so we are looking for all the help we can get. I am super encouraged by the support and feedback we are getting from developers all over the world that believe in the open-source concept, and want to make their contributions towards moving VR forward. 

Let me know what you think. What's missing, what your priorities are and how we can get you involved. Welcome to OSVR.

GDC, OSVR and the Power of the Network

I'll be at the Game Developers Conference next week to talk OSVR with anyone interested in the open-source virtual reality movement. You are likely to find me at the Razer OSVR exhibit (booth 602, South Hall) but you can also contact me to set up a meeting.

Towards the end of the week we will open up the OSVR software source code to everyone on Github on the /osvr organization. We can't wait to help the VR community - a community of innovators, of dreamers, of artists and artisans - dive into OSVR and, together, make it even better.

The power of OSVR is the power of the network. If only two people in the world had an email address, email would be pretty useless. Now that "everyone" has an email address, it is invaluable. The more companies that connect their hardware into the OSVR software framework, the more compelling it will be for game developers to create games on top of OSVR. The more games are created, the more valuable it is to add hardware to it. The value of the framework grows exponentially with the number of products that connect to it.

That's why we are working to make plugging into OSVR software framework as easy as possible. We do this through example programs, through developer assistance, through white papers that show how to convert a VR Unity into Unity over OSVR.

We announced OSVR in January. As I write this, there are about a dozen companies making HMDs that are supporting OSVR. In our experience, it takes between a few hours to a few days to fully integrate an HMD into OSVR. Once integrated, all software that is correctly written with the OSVR framework works on such HMD. If it's that easy, why wait?

BTW, it seems that there will be new HMDs announced at GDC, and we look forward to supporting them as well. Our goal is that OSVR application developers can truly use their code with pretty much any hardware on the market.

We see multiple eye tracking vendors, multiple 3D audio vendors, multiple hand and finger tracking solutions, all coming into the OSVR network. We've added about 50 partners this year. The more, the merrier, because we want to give consumers the choice of selecting the best hardware and software combination for their experience. There are numerous Android devices out there - different screen sizes, wide range of cameras, wide range of CPUs, an endless selection of unique features. Why should VR be different? Why would you want a 'one size fits all' solution?

Of course if your type of device has not been integrated into OSVR yet, it's going to be a bit more work as we would need to work together to define the interface and define it in such a way that it is not unique to your device but rather covers a wide range of similar devices. If you made the first printer that integrated into Windows, you had to work a little harder, but once a print layer services was defined for Windows, it was easy to connect other printers. The OSVR team wants to help in defining the interface and doing everything we can from the OSVR side to make such integration straightforward.

Have a new device or software package that you want to integrate into OSVR? Come talk to us.

The power of OSVR is the power of the community. How many people would we need to hire if we wanted to do OSVR completely in-house and get all the features that are truly desired by the community? 10 people? 100 people? 1000 people? Even if we had 1000 employees working on this, would we have features out quickly enough to satisfy everyone's different set of priorities? Probably not.

That's why OSVR is open. Want to connect to a custom sensor development board that OSVR does not support at the moment? Download the example on how to write a tracker plug-in and do it yourself. Want to support a game engine that is not currently supported? Follow the example of a community member that connected OSVR to a new game engine. Think you have a good 'time warp' rendering algorithm or a new SLAM algorithm or can improve system performance? Show us (and the world) how good you are!

We'll be putting up a Wiki page with some suggestions for new features and improvements, but let's drive the platform forward together.

There is power in being open. Aside from open-sourcing the OSVR hardware and software, we are building upon other open-source projects so that OSVR advances as they advance:

• OSVR builds upon some aspects of VRPN (see the list of supported devices on the main VRPN page), so that we can leverage many of the existing device drivers. As an aside, OSVR adds a descriptor file to each device so that the capabilities of the device are well-understood by the OSVR application, something that was missing from the original VRPN implementation.
• The OSVR imager class builds upon OpenCV which provides an incredible amount of computer vision and image processing algorithms, some highly optimized for particular GPUs.
• Other libraries such as Eigen and Boost, so that we don't have to reinvent the wheel

See you at GDC. Come see our demos, but more importantly, come talk to us about how we can work together. See you in San Francisco!

The post "GDC, OSVR and the Power of the Network" first appeared on the VRguy's blog at www.vrguy.net 

Connecting a Smartphone to your VR Headset: How and Why?

If you have the right combination of a smartphone, an adapter and a VR headset, you can use the smartphone to drive the headset. Let's look at how it's done and why you might want to do it.

How?

Many phones the ability to output a copy of their screen to an external monitor. Two prevalent standards to do so are MHL (Mobile High-definition Link) and SlimPort. Both provide a copy of the phone display to an external HDMI connector. For this purpose of driving a VR headset, both should be fine, and the choice depends on which method your phone supports. For example, for an LG G3 phone, this SlimPort adapter worked well for us.

Now that you have this in HDMI format, typically at 1920x1080, you'll want a VR headset that supports this resolution. This is a bit more tricky than it sounds because several 1080p headsets expect 1080x1920 portrait mode video format and thus are incompatible with the smartphone output. However, headsets such as the OSVR HDK and the professional Sensics dSight and zSight 1920 can accept standard 1080p landscape-mode video and display the phone output right away.

This approach is not limited to phones. Tablet such as the Google Nexus 7 also features a SlimPort output.

Why?

Obviously, phones or tablets have access to content that would be interesting to watch using a VR headset. For instance, www.youtube.com/3d has plenty of side-by-side movies that are suitable for a VR headset.

Headsets that accept standard 1920x1080 video typically have on-board video processing. This can also be used to turn a regular image into a side-by-side image by replicating it across both sides of the display. This can be very useful to experience non-3D content or standard applications.

One could also imagine a game being run on the phone and the headset being used as a display device. A Bluetooth motion tracker could be installed on the headset and communicate with the phone, or if you have the appropriate software installed on the phone (such as the OSVR software framework), you could communicate with the headset's USB tracker.

The alternative, of course, is to wear the phone on your head with the appropriate adapter - whether the Samsung Gear VR, Google Cardboard, Zeiss VR One and so forth.

What are the advantages of wearing the phone on your head?

• No cables required.
• No need to purchase SlimPort or MHL adapter.
• Can use phone camera within application.

What are advantages of using the MHL or SlimPort method?

• Can be used with any compatible phone or tablet
• Reduces weight - no need to wear the battery, the phone case or other unnecessary components on the head. This is especially valuable with a tablet.
• Allows using the phone's touch screen.
• With the right headset, can also experience non-3D content or regular applications.
• Can quickly connect and disconnect if required.

It's worth a try, in my opinion

Building Wireless VR Goggles

Wireless is good. Most people prefer cordless or mobile phones over corded ones; a wireless network connection over a wired one; wireless game controllers; wireless speakers; wireless charging. You get the point.

Would a wireless goggle - one that does not require cables to connect to a PC - be desirable? I certainly think so. With a wireless goggle, you could:

• Use the goggle in the living room even if the computer is on your desk
• Have multiple people use multiple goggles in the same space - such as an arcade - without tripping over each other's wires
• Avoid the risk of wrapping the goggle's umbilical cord around you as you turn
• Have greater freedom of movement

While wireless solutions already exist for professional-grade goggles (see this 2012 video of the Sensics zSight to the right), let's examine what it would take to do this for a consumer-grade product such as the OSVR HDK (Hacker Developer Kit)

For a wireless solution, one would need to consider three key components: video transmission (how you get the video to the goggles), wireless tracking (how the PC gets information about where the user is) and power.

Video Transmission

Not all wireless video solutions are created the same, because not all aim to solve the same problem. Here is what's important to look for:

• Low latency. Streaming a movie or a sports event to your TV does not require tight control over latency. If the movie is shown with a 1-second delay, that is perfectly acceptable. Streaming a game to a goggle with a 1-second delay is completely unacceptable. Sometimes, the need to control latency also dictates whether compression can be used. Compressed video saves on bandwidth but takes time to compress at the transmitter and decompress at the receiver.
• No line-of-sight requirement. Some wireless video solutions require there the transmitter can "see" the receiver, which is typically called 'line of sight'. Those solutions typically don't work for VR goggles because the whole premise of wireless goggles is to allow the user to move around as well as turn. Such turning will inadvertently result in breaking the line of sight from transmitter to receiver and thus dropping the connection. Other use cases, such as having the PC in one room and using the goggles in another probably don't even have line of sight to begin with. This requirement that the wireless signal can go through objects or walls also influences the wireless transmission frequency. Higher frequency transmission (e.g. 60 GHz) results in shorter wavelength which in turn results in greater difficulty in going through walls or humans. Lower frequency bands such as 5 GHz does a better job in penetrating through obstacles and is thus more suitable for wireless goggles.
• Support the right resolution. Wireless video solutions were originally designed with home entertainment in mind, and thus focus on supporting standard home video resolutions of 1080p (1920x1080) or 720p (1280x720). However, many consumer goggles use smartphone displays that have a native resolution of 1080x1920 - portrait mode - as opposed to the traditional 1920x1080 landscape mode. Many wireless video links do not support 1080x1920. One solution, that is now available as an option with the OSVR HDK is to use on-goggle FPGA to perform the video rotation, so that the wireless link still carries the standard 1920x1080 resolution but it is rotated in real-time to 1080x1920. This enables using wireless video links with such products (note: see the 'no free lunch' section below)
• Ability to support multiple independent video links. While this is not required if there is just one video link from the PC to the goggles, it might be required if 1) there are multiple goggles in the same space, and they each want to operate independently; 2) there is a need to carry video, such as from a video camera, from the goggles back to the PC (see this wireless camera) or 3) if the goggles require multiple HD1080p signals such as those goggles that have dual screens

Tracking

Goggles are interactive and thus very often require head orientation and sometimes head position tracking to be sent back to the controlling application. Thus, cutting the video cable is not enough and one needs to find a solution for wireless tracking.

Some wireless transmission technologies - such as WHDI - support an in-band downstream data link. If the transmission of video is considered 'upstream' - from video transmitter to video receiver - a downstream data downlink is sending information from the video receiver (where the goggles are) to the video transmitter (where the PC is). These data downlinks can often carry USB HID (Human Interface Device) messages such as those coming from game controllers and provide a reasonable bandwidth of about 100 kbps. If on-board orientation tracking is formatted into a compatible message, head tracking can be carried to the PC on the same link as the video. This is how tracking is performed in the 2012 zSight video above. The downside of this approach is the update rate. Assuming 60 FPS video, WHDI transmits the downstream data during the blanking periods inbetween frames. Thus, USB HID transmission is also provided at 60 Hz, which is sometimes not interactive enough.

Another method is to use or embed a wireless tracker such as the YEI 3-space sensor. Adding an out-of-band data link ((not on the same link as the video transmission) to a tracker can provide high-rate tracking information without needing a cable.

Other methods are to use trackers that have some kind of base station, such as the Sixense STEM. In this case, the tracker base is connected to the PC. An advantage of this approach is that beyond the head, other parts of the body can also be tracked, as is also the case with the PrioVR suit.

Power

Cutting the cord means also providing local power to the goggles as well as other local consumers of energy - wireless receiver, wireless tracker, on-board camera where applicable and so forth. Including the goggle and wireless receiver, one could expect a power draw of 5-10 Watts (1-2 Amps on +5V). This can typically be satisfied with a 1 or 2 high-current battery such as those being marketed to charge phones and tablets. These batteries are rated in mAh (milliampere-hour) or Ah (ampere-hour). An Ampere-hour means that the battery can theoretically provide 1 ampere of current for an hour. For instance, a 10000 mAh battery that outputs +5V can theoretically provide 10 Watts (2 Amps x 5V) for 5 hours. A typical AA battery provides 2-3Ah. A typical external battery for smartphones can provide 10Ah and a typical car battery can provide about 50Ah (though it would be quite heavy). However, because of many factors - such as the drop in voltage when the battery drains - these figures are a someone optimistic. However, one can often plan for 1-2 hours between charges on an external smartphone battery.

There's no free lunch

What are the downsides of a wireless VR solution?

• It costs more. One would need to factor in the price of the wireless video link, a potential price increase in the tracking costs, and the costs of a battery
• There may be a price to pay in video latency. If you use an on-board FPGA to rotate the image from 1920x1080 to 1080x1920, you add 1 frame's worth of latency, or 16mSec. This is because you have the store the entire image in memory before you can start outputting the rotated image.
• You are often limited to 60 FPS video. Because most wireless video solutions were designed for home entertainment, they provide 1080p @ 60 FPS. Newer solutions that aim to provide higher resolutions might be able to address this issue.
• You have to carry the battery and the receiver. This is can be done in a beltpack or small backpack.
• You have to recharge or change the battery from time to time.

What about using a phone for your HMD?

One way to completely overcome this issue is not use a PC. Phone-based HMDs such as Google Cardboard perform all the processing and display on the local phone, thus eliminating the need for wireless video transmission. However, one could probably safely assume that a powerful PC would always have more computing power, more graphics power and access to a wider range of peripherals than a phone, and thus some PC-based experiences could not be replicated on a phone.

Should you do it?

Having discussed the advantages and disadvantages, should you use a wireless HMD? At the very least, you will want to experience it. Having no cables can be quite liberating and is certainly worth a try. Let me know what you think!

The post "Building Wireless VR Goggles" first appeared on the VRguy's blog at www.vrguy.net