Embedded OSVR and what it can do for Phones and Game Consoles

What do you do when you need to add a VR peripheral like an eye tracker or a motion suit to a phone or a game console? Tear your hair out, probably.

There are multiple reasons why this is not easy. Depending on the particular setup, common reasons are:

• Closed programming environment that prevents you from working with an unknown peripheral, without going through a tedious approval process.
• Not enough processing power to service the peripheral without negatively impacting the performance of the game.
• No physical connectivity: lack of available USB ports.
• Lack of device drivers supporting the particular peripheral.
• No appetite to connecting additional wires to an otherwise portable or wireless system.

The OSVR software framework might be able to help. OSVR has two parts: the "OSVR Server" and "OSVR client". The Server processes data as it comes from the various peripherals and sensors and then presents it to Client in a device-agnostic format. For instance, once an eye tracker interface has been defined, the server can translate the vendor-specific implementation of eye tracking data (such as for an SMI eye tracker) into the universal device-independent reports. The Client is part of the game or VR application that uses this data.

In many cases, the Server and Client will reside on the same CPU (or sometimes even in the same process), and that is the standard deployment for a PC environment. However, the Client and Server can also be connected through a network interface such as WiFi or Bluetooth. In fact, a Client could connect to multiple servers, some residing locally, and some residing elsewhere over a network.

What does this mean? It means that you can connect VR peripherals to one CPU and run the game on another CPU.

For instance:

• Game (including OSVR Client) runs on phone. Peripherals (positional tracking, for example) including OSVR server runs on PC. Game receives 60 FPS updates from PC via Wifi or Bluetooth. Since PC does all the heavy lifting, the 60 FPS position updates are very short and low-bandwidth. 
• Game runs on XBOX, eye tracking runs on external CPU (such as super-compact single-board computers from Gumstix or Variscite) which uses the OSVR server to provide the universal eye tracking interface. By the way, these single-board computers could run OSVR using the operating system of your choice - Linux, Android, Windows Embedded) regardless of what operating systems runs with the client. 

My guess is that we will see additional examples of this over the coming months.

What would you use this for?

Advancing VR support for Mac

Given that Oculus/Facebook mothballed support for VR on Mac, the OSVR team at Sensics decided to step up and fill that void.

Once successful, this will provide an active path to continue to work with Oculus (and many other HMDs) on Mac. It will also pave the way to make use of the many capabilities of OSVR including positional and predictive tracking, device-independent integration for cameras, gestures, skeleton, locomotion devices and more.

We are seeking community assistance in making this happen faster. Below are short descriptions of what we know and what we are seeking. Comments, feedback and offers to assist will be most appreciated. 

What we know 

• OSVR is built to be multi-platform. We have OSVR working on Windows, Android and Linux and have had some reports of success in working on Mac. 
• We have OSVR building on both Linux (with Clang and GCC) and Windows. All the commits must pass testing on the CI for Linux and Windows. 
• OSVR supports multiple HMDs including those made by Oculus, OSVR, Sensics, Vuzix. Vive support is coming, as well as better integration with Cardboard. Support includes display parameters, distortion. Asynchronous time warp supported on Windows/NVIDIA environment but we are working to expand this to AMD/Intel as well as other platforms. 
• OSVR has been integrated into WebVR. Bugzilla patch has been submitted and reviewed by the Mozilla team. WebVR demos work on top of OSVR-supported HMDs. This means that OSVR work will also help support WebVR on Mac and on other platforms
• We know OSVR was successfully built on Mac last December (prior to public release), and all external dependencies have Mac support, so we should have a good idea of the missing pieces.

Status of Facebook/Oculus support in OSVR

Display support for Oculus in OSVR is already entirely independent of Oculus SDK. It uses distortion parameters from OSVR Distortionizer and our own distortion shader, and display descriptor. So, OSVR support for Oculus on Mac comes down to sensor access (and sensor fusion, as in the Oculus case, the SDK or runtime does the IMU fusion). 

Currently, the OSVR-Oculus plugin master branch uses a VRPN driver written to access the Rift SDK, which is built against a 0.4.x release, and fully functional and multi-platform. For instance, here is a demo of OSVR running Rift on Linux. Here is another demo showing the OSVR Palace demo running on Unity over Linux with OSVR/Oculus:

A branch contains a direct OSVR PluginKit driver (no VRPN involved) that builds against a newer Rift SDK, but there are bugs in it, so it is unfinished. If looking to handle Rift on "officially unsupported" (by Facebook) platforms, including Mac and Linux, will want to develop an OSVR PluginKit driver using an open-source driver stack. 

There are several, though some of them claim a non-copyleft license while incorporating some well-known GPL sensor fusion, so problematic that way. OpenHMD appears intended to be a broader approach than just an open-source sensor access for Oculus, but is BSL1.0 licensed and includes sensor fusion, so could serve the purpose. There are almost surely others and we would be happy to receive referrals.

Contributions/assistance wanted

Starting with the OSVR codebase (osvr.github.io): 

• Set up homebrew builds for libfunctionality and jsoncpp (if needed) 
• Potential implementation (libltdl or other) needed in libfunctionality (which is a simple wrapper for loading plugins at runtime) 
• Set up homebrew build for OSVR-Core (though it won't build fully at first). Known implementation details needed: 
• usbserialenum - could just use dummy impl right now 
• plugin search path code (in PluginHost) 
• Provide input on best open-source library to access Rift sensors 

 Comments, feedback and offers to assist will be most appreciated. Let's do this together!