Suffering, Art and VR Standards

Think about a great work of art: a classic book, a timeless painting, a symphonic masterpiece. What's common to many of these creations?

They were all the result of great suffering.

Tolstoy, Van Gogh, Mozart - they did not have easy lives. Many of the greats suffered from oppression, mental or physical illness, or hunger.


If you don't have drama in your life, how could you summon drama for your art?

People ask me "what made you want to work on VR standards?" My answer: it's the suffering.

No, not my personal suffering. I'm no Amaedus or have never considered cutting off my earlobe to express love.

But in many years of working with customers on their VR systems, I saw a lot of technical suffering:

The suffering of integrators that need to chase the latest API again and again. That don't know if the equipment they design for today will be available to buy in a year.

The suffering of device manufacturers that need just one more driver to support them.

The suffering of end-users that wonder if today's software will work on tomorrow's devices.

That's why we need efforts like OSVR or OpenXR to make it easy for everyone to work together. It wouldn't be as timeless or profound as "War and Peace", but it will help a lot of people.

Virtual Reality Standards: too early or long overdue?

This article originally appeared on Mar 22nd at ReadWrite

You bought a new printer for the office. You unpack and connect it to your PC. You install its demonstration software and see the printer works well. Then, an unfortunate surprise: your word processor cannot work with this printer. You'll need to wait until the maker of the word processor releases a new version. When will this version be available? Should you return the printer? Should you change to a different word processor?

If this sounds like the 1980's, it's not too far from where VR is today. VR programs are often hard-coded to one set of hardware devices. Only a particular HMD, coupled with its tracking system and controller will work. Every device manufacturer has a different API. If you want to use this VR experience with a different set of hardware, you might not be in luck. At best, you'll need another version. Worst case, you just won't be able to do it. The problem of different vendors having different APIs is often called 'API fragmentation'

VR standards can help solve this fragmentation problem. In the PC world there are a few basic device types: keyboard, mouse, printer, scanner, and so forth. Likewise, basic VR device types include an HMD, tracker, controller, and a few others. A standard way for a program to interface with these VR devices would help solve this problem.

If software could work across different hardware combinations, almost everyone would benefit:

• Consumers could mix and match devices to their liking. I may have a Dell computer but I don't always want a Dell printer to go with it. Furthermore, consumers would be confident that their investments are future-proof. A 2017 game would likely work with 2018 or 2019 hardware. 
• Game publishers and other experience creators would have a larger addressable market. Today, they hard-code their game to a particular set of hardware devices. Tomorrow, they would support any device that has a conforming 'driver'. 
• Manufacturers that support a standard would have their devices work with lots of content. This would allow even small players to enter the market, and promote innovation. 

Some claim it is too early for a VR standard. VR is new, they say. Let a couple of years pass and then we will know what to standardize. Yet while consumer VR is new, VR has been in academia and industry for decades. Labs and factories deployed HMDs, head and eye trackers, and controllers for many years. Such devices were expensive and perhaps more clunky, but still performed the same functions. Software frameworks like UNC's VRPN (http://www.vrpn.org) provided device independent access for many years.

The resistance to a standard sometimes stems from the competitive strategy of a company. A vendor relying on a 'walled garden' approach often wishes to control the entire stack. The ability to swap out hardware, or use a different app store might be not what they had in mind.

In VR, there are often two standard interfaces that need to defined. The first is the device interface. This defines how to configure devices of particular type and how to extract data from them. Printers have different capabilities but share the same basic functions. The same is true for VR devices. The second standard interface is the application interface. It describes how an application or a game engine renders its content and get data. Inbetween the applications and devices there is often a middleware layer. That middleware is the software intermediary between applications and devices.

One effort that adapts this approach is OSVR. Started by Sensics and Razer, it is an open-source software platform for VR. OSVR implements both a device interface layer as well as an application layer. OSVR supports over 200 devices, and most of the OSVR code is free and open-source.

Another effort is OpenVR which is an open API (though not open source) from Valve. Building on the success of SteamVR, OpenVR allows HMDs to work with SteamVR content. There is some compatibility between these efforts. An OSVR plugin for OpenVR, allowing OSVr devices to work with SteamVR content.

In January, the Khronos group (known for OpenGL standards) launched a new VR initiative. The initiative, called OpenXR, brings together a wide range of companies. Industry leaders including Google, Oculus, Valve, Sensics and Samsung are part of this effort. OpenXR aims to combine lessons learned from building OSVR, OpenVR and proprietary APIs. It aims to create both a device interface as well as application interface. It is unclear how soon this effort will mature. Khronos standards take an average of 18 months. It is also unclear what capabilities will be part of the first standard. What is clear is that these companies felt enough pain to want to work on standards.

I am encouraged that so many participants are coming together to work on a standard. Other interested parties are also invited to contribute. Standards are sometimes boring, but they are important. They will make the consumer experience better and promote innovation.

This article originally appeared on Mar 22nd at ReadWrite

Why did Sensics launch the OSVR Store?

Last week, the OSVR Store came on-line. It offers a range of OSVR-related products, services, accessories and components. It also contains useful information, most of it adopted from this blog.

But why did the Sensics team launch it?

The first answer that comes to mind is “to make money”. That’s an obvious reason, as Sensics is a for-profit company. We invest a lot in developing OSVR and would love to see returns on our investments.

But that’s not the only reason, nor perhaps the most important one. Here are some others.

We wanted the OSVR Store to be helpful to the VR enthusiast and hacker. That’s why we offer components: optics, tracking boards from various vendors, IR camera. More components are coming. Some will use those to upgrade an existing system, others to build a new one.

We wanted a place for hardware developers, a platform to market their innovations. If you make something OSVR-related, we invite you to sell it on the OSVR Store. It can be an OSVR-supported HMD. It can be an accessory or component that can help OSVR users. It can even be OSVR-related services. We strive to offer fair and simple terms. If you can build it, we can help you promote it. Drop us a note at hello@osvrstore.com to get started.

To me, OSVR has always been about choice. About democratizing VR. Not forcing users to buy everything from the same vendor. Encouraging applications to run on many devices. Support more than one operating system.

The OSVR Store is one more way to give everyone choice. Check it out.

OSVR - a Look Ahead

Introduction

OSVR is an open source software platform and VR goggle. Sensics and Razer launched OSVR 18 months ago with the intent of democratizing VR. We wanted to provide an open alternative to walled-garden, single-device approaches.

It turns out that others share this vision. We saw exponential growth in participation in OSVR. Acer, NVIDIA, Valve, Ubisoft, Leap Motion and many others joined the ecosystem. The OSVR goggle – called the Hacker Development Kit – has seen several major hardware improvements. The founding team and many other contributors expanded the functionality of the OSVR software.

I’d like to describe how I hope to see OSVR develop given past and present industry trends.

Increased Device Diversity leads to more Choices for Customers

Trends

An avalanche of new virtual reality devices arrived. We see goggles, motion trackers, haptics, eye trackers, motion chairs and body suits. There is no slowdown in sight: many new devices will launch in the coming months. What is common to all these devices? They need software: game engine plugins, compatible content and software utilities. For device manufacturers, this software is not a core competency but ‘a necessary evil’. Without software, these new devices are almost useless.

At the same time, content providers realize it’s best not to limit the their content to one device. The VR market is too small for that. The more devices you support, the largest your addressable market becomes.

With such rapid innovation, what was the best VR system six months ago is anything but that today. The dream VR system might be a goggle from one vendor, input devices from another and tracking from a third. Wait another six months and you’ll want something else. Does everything need to come from the same vendor? Maybe not. The lessons of home electronics apply to VR: you don’t need a single vendor to make all your devices.

This ‘mix and match’ ability is even more critical for enterprise customers. VR arcades, for instance, might use custom hardware or professional tracking systems. They want a software environment that is flexible and extensible. They want an environment that supports ‘off-the-shelf’ products yet extends for ‘custom’ designs.

OSVR Implications

OSVR already supports hundreds devices. The up-to-date list is here: http://osvr.github.io/compatibility/ . Every month, device vendors, VR enthusiasts and the core OSVR team add new devices. Most OSVR plugins (extension modules) are open-sourced. Thus, it is often possible to use an existing plugin as baseline for a new one. With every new device, we come closer towards achieving universal device support.

A key OSVR goal is to create abstract device interfaces. This allows applications to work without regards to the particular device or technology choice. For example, head tracking can come from optical trackers or inertial ones. The option of a a “mix and match” approach overcomes the risk of a single vendor lock-in. You don’t change your word processor when you buy a new printer. Likewise, you shouldn’t have to change your applications when you get a new VR device.

We try to make it easy to add OSVR support to any device. We worked with several goggle manufactures to create plugins for their products. Others did this work themselves. Once such a plugin is ready, customers instantly gains access to all OSVR content. Many game engines – such as Unity, Unreal and SteamVR- immediately support it.

The same is also true for input and output peripherals such as eye trackers and haptic devices. If developers use an API from one peripheral vendor, they need to learn a new API for each new device. If developers use the OSVR API, they don’t need to bother with vendor-specific interfaces.

I would love to see more enhancements to the abstract OSVR interfaces. They should reflect new capabilities, support new devices and integrate smart plugins.

More People Exposed to more VR Applications in More Places

Trends

Just a few years ago, the biggest VR-centric conference of the year had 500 attendees. Most attendees had advanced computer science degrees. My company was one of about 10 presenting vendors. Today, you can experience a VR demo at a Best Buy. You can use a VR device on a roller coaster. With a $10 investment, you can turn your phone into a simple VR device.

In the past, to set up a VR system you had to be a geek with plenty of time. Now, ordinary people expect to do it with ease.

More than ever, businesses are experimenting with adopting VR. Applications that have always been the subject of dreams of are becoming practical. We see entertainment, therapy, home improvement, tourism, meditation, design and many other applications.

These businesses are discovering that different applications have different hardware and software requirements. A treadmill at home is not going to survive the intensive use at a gym. Likewise, a VR device designed for home use is not suitable for use in a high-traffic shopping mall. The computing and packaging requirements for these applications are different from use to use. Some accept a high-end gaming PC, while others prefer inexpensive Android machines. I expect to see the full gamut of hardware platforms and a wide variety of cost and packaging options.

OSVR Implications

“Any customer can have a car painted any color that he wants so long as it is black”, said Henry Ford. I’d like to see a different approach, one that encourages variety and customization.

On the hardware side, Sensics is designing many products that use OSVR components. For instance, our “Goggles for public VR” use OSVR parts in an amusement park goggle. We also help other companies use OSVR components inside their own packages. For those that want to design their own hardware, the OSVR goggle is a good reference design.

On the software side, I would like to see OSVR expand to support more platforms. I’d like to see better Mac support and more complete coverage of Android and Linux platforms. I’d like to see VR work well on mid-range PCs and not limited to the newest graphics cards. This will lower the barriers to experience good VR and bring more people into the fold. I’d like to see device-specific optimizations to make the most of available capabilities. The OpenCV image processing library has optimizations for many processors. OSVR could follow a similar path.

Additionally, it is important to automate or at least simplify the end-user experience. Make it as close to plug-and-play as possible . The task of identifying available devices and configuring them should be quick and simple.

Simplicity is not limited to configuration. We’d like to see easier ways to choose, buy and deploy software.

Reducing Latency is Becoming Complex

Trends

Presence in VR requires low latency, and reducing latency is not easy. Low latency is also not the result of one single technique. Instead, many methods work together to achieve the desired result. Asynchronous time warp modifies the image just before sending it to the display. Predictive tracking lowers perceived latency by estimating future orientation. Direct mode bypasses the operating system. Foveated rendering reduces render complexity by understanding eye position. Render masking removes pixels from hidden areas in the image.

If this sounds complex, it is just the beginning. One needs to measure optical distortion and correct it in real-time. Frame rates continue to increase, thus lowering the available time to render a frame. Engines can optimize rendering by using similarities between the left- and right-eye images. Techniques that used to be exotic are now becoming mainstream.

A handful of companies have the money and people to master all these techniques. Most other organizations prefer to focus on their core competencies. What should they do?

OSVR implications

A key goal of OSVR is to “make hard things easy without making easy things hard”. The OSVR Render Manager examplifies this. OSVR makes these latency-reduction methods available to everyone. We work with graphics vendors to achieve direct mode through their API. We work with game engines to provide native integration of OSVR into their code.

I expect the OSVR community to continue to keep track of the state of the art, and improve the code-base. Developers using OSVR can focus away from the plumbing of rendering. OSVR will continue to allow developers to focus on great experiences.

The Peripherals are Coming

Trends

A PC is useful with a mouse and keyboard. Likewise, A goggle is useful with a head tracker. A PC is better when adding a printer, a high-quality microphone and a scanner. A goggle is better with an eye tracker, a hand controller and a haptic device. VR peripherals increase immersion and bring more senses into play.

In a PC environment, there are many ways to achieve the same task. You select an option using the mouse, the keyboard, by touching the screen, or even with your voice. In VR, you can do this with a hand gesture, with a head nod or by pressing a button. Applications want to focus on what you want to do rather than how you express your wishes.

More peripherals mean more configurations. If you are in a car racing experience, you’d love to use a rumble chair if you have it. Even though Rumble chairs are not commonplace, there are several types of them. Applications need to be able to sense what peripherals are available and make use of them.

Even a fundamental capability like tracking will have many variants. Maybe you have a wireless goggle that allows you to roam around. Maybe you sit in front of a desk with limited space. Maybe you have room to reach forward with your hands. Maybe you are on a train and can’t do so. Applications can’t assume just one configuration.

OSVR implications

OSVR embeds Virtual Reality Peripheral Network (VRPN), an established open-source library. Supporting many devices and focusing on the what, not the how is in our DNA.

I expect OSVR to continue to improve its support for new devices. We might need to enhance the generic eye tracker interface as eye trackers become more common. We will need to look for common characteristics of haptics devices. We might even be able to standardize how vendors specify optical distortion.

This is a community effort, not handed down from some elder council in an imperial palace. I would love to see working groups formed to address areas of common interest.

Turning Data into Information

Trends

A stream of XYZ hand coordinates is useful. Knowing that this stream represents a ‘figure 8’ is more useful. Smart software can turn data into higher-level information. Augmented reality tools detect objects in video feeds. Eye tracking software converts eye images into gaze direction. Hand tracking software converts hand position into gestures.

Analyzing real-time data gets us closer to understanding emotion and intent. In turn, applications that make use if this information can become more compelling. A game can use gaze direction to improve the quality of interaction with a virtual character. Monitoring body vitals can help achieve the desire level of relaxation or excitement.

As users experience this enhanced interaction, they will demand more of it.

OSVR Implications

Desktop applications don’t have code to detect a mouse double-click. They rely on the operating system to convert mouse data into the double-click event. OSVR needs to provide applications with both low-level data and high-level information.

In “OSVR speak”, an analysis plugin is the software that converts data into information. While early OSVR work focused on lower-level tasks, several analysis plugins are already available. For example, DAQRI integrated a plugin that detects objects in a video stream.

I expect many more plugins will become available. The open OSVR architecture opens plugin development to everyone. If you are an eye tracking expert, you can add an eye tracking plugin. If you have code that detects gestures, it is easy to connect it to OSVR. One might also expect a plugin marketplace, like an asset store, to help find and deploy plugins.

Augmenting Reality

Market trends

Most existing consumer-level devices are virtual reality devices. Google Glass has not been as successful as hoped. Magic Leap is not commercial yet. Microsoft Hololens kits are shipping to developers, but are not priced for consumers yet.

With time, augmented-reality headsets will become consumer products. AR products share many of the needs of their VR cousins. They need abstract interfaces. They need to turn data into information. They need high-performance rendering and flexible sensing.

OSVR Implications

The OSVR architecture supports AR just as it supports VR. Because AR and VR have so much in common, many components are already in place.

AR devices are less likely to tether to a Windows PC. The multi-platform and multi-OS capabilities of OSVR will be an advantage. Wherever possible, I hope to continue and see a consistent cross-platform API for OSVR. This will allow developers to tailor deployment options to the customer needs.

Summary

We designed OSVR to provide universal connectivity between engines and devices. OSVR makes hard things easy so developers can focus on fantastic experiences, not plumbing. It is open so that the rate of innovation is not constrained by a single company. I expect it to be invaluable for many years to come. Please join the OSVR team and myself for this exciting journey.

To learn more about our work in OSVR, please visit this page

This post was written by Yuval Boger, CEO of Sensics and co-founder of OSVR. Yuval and his team designed the OSVR software platform and built key parts of the OSVR offering.

Action Items from the OSVR Software Developer Survey

A couple of weeks ago, we surveyed the OSVR community for what they would like the core software development team to focus on. You can see the questions and answers here

Based on this input, the Sensics development team met and decided to focus on the following items:

1. Smoother end-user experience
This includes:

• 1-click installer for both end-user and developer
• Lightweight software that will detect when a new hardware is connected and help with the process of obtaining and installing the relevant drivers. 
• A graphical configurator to help the end-user select the HMD, input and output devices as well as configure key parameters for each

2. Continue to add device support. An immediate focus is to add support for the HTC Vive so that an end-user can obtain software that was written on the OSVR framework that - using a simple configurator - decide what to run it on. For instance 1) HTC Vive; 2) Oculus + hand controller; 3) OSVR HDK and more.

3. Make it easier for the OSVR community to contribute to the platform by listing key development priorities as well as high-level directions on how to perform certain development tasks.

Keep an eye on the progress of the OSVR software platform in the coming weeks.

Thank you for your feedback!

Vision Summit 2016: Using OSVR to Support (practically) Any Device in AR/VR

I delivered this presentation last week at the Vision 2016 Summit

Full video:

Slides only:

Using OSVR to Support (practically) Any Device in VR/AR from Yuval Boger

The key point in the presentation is that no one wants to write AR/VR applications that work only on one device. To put a positive frame on it, the ability of applications to work across a wide range of displays, inputs and output devices is valuable to practically to everyone:

• Content providers want their applications to be used on the widest range of possible devices.
• Makers of VR displays, input or output devices don't want to settle for a few pieces of content that are written specifically for then; they want access to a wide range of content
• Consumers want 2016 applications to work on 2017 hardware without having to buy upgrades or wonder if their new hardware will ever be supported

OSVR achieves just that - it allows runtime choice of what input, output and display devices to use and the presentation illustrates this. OSVR supports numerous devices today, with new devices being added every week.

Got new or custom HMD? Need direct render, time warping, distortion correction and game engine integration?

Google Concept from Sensics

Let's assume you built a new HMD. You've done all the hardware work - figured out the optics and display setup, created driver boards, integrated a motion tracker and perhaps an eye tracker, worked out the ergonomics issues. It's a lot of work and you should be commended on doing it.

How do you get software support for it? Specifically, you probably want to:
- Model the field of view and allow users to correct any optical or color distortion
- Obtain support for direct render and asynchronous time warp
- Get Unity and Unreal drivers for your trackers
- Get some demo content and some cool games running on it
- and, in general, make it easy for others to support your new creation.

 A solution to consider is OSVR. By integrating your new HMD into the open-source OSVR framework, you can get all that done (and more) very quickly. Get low-latency rendering for your HMD; correct distortion in one of several possible ways; support for many game engines; debug and demonstration software.

OSVR already supports many devices (full list here) and based on the work team Sensics did with several HMD vendors over the last few months, we put up documentation on how to add an HMD to OSVR. You can find it here and it is part of the official OSVR developer documentation repository.

Let me know what you think and how we can make it easier to get this done.

Everyman's VR: Bridging the Gap between $15 and $1500 for VR Goggles

There is a gap in VR pricing that could slow down the adoption of VR.

Today, you can spend about $15 and buy a Cardboard-type phone attachment to get simple VR with your phone. You can also spend $1500 and in a few months you will get a nice computer and what would be considered today a high-performance consumer VR experience, touch controller not included. As I write this, the price for an HTC Vive has not been published - perhaps it will be even higher.

$1500 is a lot of money. Maybe I just don't have $1500 available to spend. Maybe I spent $900 a year ago on a nice computer and I don't want to buy a new one. Maybe I'm running a classroom and we just can't afford a $1500 machine on every desk? Maybe I worry that VR is moving so fast that the $1500 VR system I buy in 2016 will be obsolete in 2017?

$15. $1500. Excluding Gear VR ($100 if you already have a fairly new Samsung phone), is there any good solution in-between? How can we bridge the gap to allow "Everyman's VR"?

A few things could - and perhaps should - happen:

• HMD vendors would select different trade-offs geared towards different use cases (media viewing, casual gaming) and offer them at lower price points. 
• Software platforms, like OSVR, would continue to allow games to modulate the required computing power. For instance, the Unreal Showdown demo on OSVR runs nicely on an NVIDIA 6xx card.
• Benchmark utilities would allow users to determine how fast a game can run on their PC and have the game adjust to it. Can't run Elite: Dangerous at 90 FPS? Run it at 45 FPS with time-warping or reduce the resolution and image quality.
• Eye trackers embedded inside goggles allow using foveated rendering, which is the process of doing high-quality rendering at the portion of the image you are directly looking at and lower quality rendering elsewhere. This would save GPU resources and allow running on lower-end PCs. Similarly, VR driver optimizations from NVIDIA/AMD/Intel could make better use of GPU resources.

This is where OSVR can make a big impact. I'm not focused so much on the OSVR HDK ($300 at osvr.org), but rather on the OSVR software platform. A middleware platform like OSVR allow running the same application at different performance levels on a wide range of devices, with striving towards nearly universal compatibility. If an HMD vendor knew that lots of compelling content could be immediately available to their customers if they built an OSVR plugin, many HMD variations are likely to appear.

Does this remind you of something? Think about what Android did to the phone and tablet market. It created choice. It allowed tablets and phone to appear at all kinds of performance and price specs, all running off a common software platform. We saw high-end phones like the Samsung S6 or LG G4 all the way to low-end generic phones. Could a $50 tablet run all the apps that a high-end Android tablet runs? No, but for some users the $50 tablet is perfectly adequate.

There is a wide gap between $15 and $1500 and there are many people that would like to experience VR that's better than $15 but prefer not to spend $1500. Let's work together to fill this gap and create "Everyman's VR"

Diagramming the VR Stack

While preparing for last week's Consumer Electronics Show, I created a simple diagram to explain and VR application stack and how it related to the work that we are doing with OSVR. I thought it might be interesting to share this diagram and explain its contents:

The VR Stack diagram, Copyright (c) Sensics, Inc.

At the risk of contradicting Copernicus, we see the HMD at the center. Without the display, whether VR or AR, there is not much of an experience.

The display by itself does not make the experience, just like a game console without game controllers are not very interesting. Thus, the HMD is surrounded by a variety of sensors and output devices. These are often orientation and position trackers, input devices, but could also be eye trackers, motion platforms, gesture and skeleton sensors, depth sensors and more.

Moving for a moment to the outer circle, there are games and other applications. These needs to be distributed (and often monetized), but let's leave the distribution discussion for another time.

Very often, these applications are written using graphics or game engines. They provide a wide range of services: from assets to lighting to physics and more. These engines - such as Unity, Unreal, Cryengine or other are shown in the next to last circle.

To me, what is often overlooked in VR is the need for a middleware layer - a layer that connects the various engines to the various sensors and peripherals. I submit that no one likes to write applications that work only on one device and because of that, a layer that allows a game engine to connect to many different HMDs, to many different eye trackers, to many different sensors is critical. Think about your computer at home. Does everyone use the same monitor? Keyboard? Mouse? Printer? Not everyone would want to use the same HMD. That's why middleware is required and an excellent example of middleware is the OSVR software platform supporting hundreds of devices, peripherals and game engines. All layers are important, so don't forget the middleware!

The post "Diagramming the VR Stack" first appeared on "Virtual Reality and Head Mounted Displays"

Phone-based VR without the Phone

There are two main VR configurations today: PC-based VR (e.g OSVR HDK, HTC VIVE, etc.) and phone-based VR (e.g. Samsung Gear VR, Google Cardboard, etc.). Each approach carries different advantages: PC-based VR allow using high-power graphics cards; Phone-based VR is highly portable and battery-operated.

The rationale behind phone-based VR solutions is two-fold:

1. If you already have a phone, the incremental investment for a VR experience is low. Gear VR is under $100 now, and Cardboard is much cheaper. For casual VR, popping a phone in a carrier is a convenient solution.
2. Today's phones have high-resolution screens, integrated cameras, increasingly better motion sensors and many additional capabilities that are very useful for VR. Because they are mass-produced, they are very cost-effective.

However, upon closer investigation it appears that a user of phone-based VR is paying - both in cost and in weight - for phone components that are not truly necessary for VR. For instance one could argue that the following components are not truly necessary:

• Cellular connectivity
• Phone body. If the phone is permanently integrated into the cradle, one does not need the weight of the body itself.
• Touch screen
• and more...

Now that VR is gaining steam, VR devices are going to have modules and components that are designed specifically for VR instead of settling for parts from other electronics devices. For instance we'll see screens made for VR (small size, higher refresh rate, low persistence). Given this, it would not be surprising to start seeing phone vendors provide pre-integrated VR goggles that are essentially a phone without the unnecessary components coupled with a cradle/head strap. These would be lighter than separate phone + cradle and probably also less expensive than the combination.

What else would you like to see in 2016?

Using VR (and OSVR) in roller coaster rides and other out-of-home applications

Ubisoft Rabbids demo at E3 2015

We've seen a lot of interest in using virtual reality for theme park and other "out of home" applications. To me, this interest is very justified. Virtual reality allows creating new types of experiences. It is much cheaper for an operator of a roller coaster, for instance, to have visitors use VR to upgrade a ride as opposed to building an entire new track. VR allows changing the experience from ride to ride as well as to modulate the intensity of the experience.

Similarly, the notion of a VR cafe has also been raised. Just like in the 1990's when high-speed Internet access was not prevalent, a VR cafe would allow casual VR users to experience gaming VR without having to purchase a VR headset and a high-performance computer for home use.

There are several common requirements to all these use cases:

• VR goggles need to be rugged enough to withstand heavy use, accidental drops or the occasional teenager that tries to take it apart.
• Nearly-universal fit is needed so that little or no adjustment is required for a good VR experience. Some attractions will have an operator that can offer some assistance, but maintaining the throughput of an attraction is an important requirements.
• Need to be able to quickly clean and sanitize goggles between users. This might use a disposable face mask, or a way to quickly wipe down the headset.

In additional, some applications also require:

• Integration of the VR goggles into an existing frame (e.g. racing helmet for a racing simulator)
• Cable management
• Integration of other peripherals such a multi-person position tracking

Consumer VR headsets are not immediately suitable for these applications, though can definitely be used to prototype the initial design. On the other hand, building a completely new HMD and then creating a high-performance rendering infrastructure for it is also an expensive and time-consuming endeavor.

What to do? One option is to use OSVR for either the hardware and/or software portions of this effort.

Because OSVR hardware is open-source and designed to be hacked and changed, it is possible to take existing OSVR components - for instance the display, electronics and optics - and then package them as required to address the particular needs of the attraction.

Similarly, the OSVR software framework provides high-performance rendering and plugins for many game engines across a wide variety of operating systems, HMDs and peripherals. OSVR can also be extended to new types of HMDs if custom hardware is created.

My company can also help with creating semi-custom designs primarily based on pre-existing building blocks and with optimizing a software infrastructure to support a particular set of hardware peripherals

On a personal note, I had a chance to try a pre-production version of the VR ride at Europapark in Germany and it was quite an experience. During my visit, I tried the same ride twice: once with a track being shown in the virtual world (giving some hint of what turn or roll will happen next) and the other without it (making me feel like a pinball in space).

Out-of-home VR is becoming possible and cost effective, and that's an exciting development both for theme park owners as well as for showcasing VR to the public.

I'll be at the IAAPA show later this month in Orlando. If you want to meet and discuss some opportunities, drop me a note.

The Video Processing FPGA inside the OSVR HDK and the Sensics dSight

Now that OSVR hacker developer kits are in the wild, @skyworxx Mark Schramm has posted some teardown photos of the 1.2 version of the HDK. He notes the FPGA on the board, and I thought I'd take the opportunity to explain what the FPGA can do in the HDK, and in it's high-end brother the dSight

The FPGA and the DRAM chips next to it can perform real-time video processing, giving hackers the ability to experiment with hardware-level transformations that do not require cooperation from the video-generating device or application.

Some of the options that are currently implemented:
1. Pass-through. This is the simplest mode and it does not involve the DRAMs. 1080x1920 video from the HDMI receiver is essentially passed through to the MIPI interface that drives the OLED screen.

2. Image rotation. This mode allows real-time 90 degree rotation, so that standard landscape mode 1920x1080 video can be presented in the 1080x1920 display. To do this, a full video frame is stored in the DRAM chips while the previous frame is sent to the display. This 90 degree rotation does cost you 1 frame of latency, but can be very useful in some of the following scenarios:

• Video coming from a 1920x1080 source such as as DVD player or in "replicated desktop" mode.
• Video coming over a low-latency wireless link. These links primarily support 1920x1080 today and not the native 1080x1920 mode.

3. Conversion of full screen to side-by-side mode. Ever saw a desktop in the HMD and found yourself squinting to see each half at a time. When this conversion mode is enabled, the video signal is converted into two identical copies of the signal, that can then be shown on both eyes at the same time. Control over this mode is via a command via the HID feature interface or with a simple utility.

Additional modes that are not yet implemented but can be implemented by the community:

• Real-time distortion correction. If you have a non-cooperative video source or just prefer to use your GPU for something else, real-time distortion correction in the FPGA can be useful.
• Resolution up-scaling: converting from lower resolution into the full resolution of the HDK
• Color enhancements (e.g. gamma, HSI improvements)
• Rearrange the video signal. One cool application that we saw from one of our partners is rearranging a 1080x1920 on the output of a GPU to reformat it into a non-legible 1920x1080 image, send that over the low-latency wireless video link and then use the FPGA to unscramble the image. This allows wireless video transmission without "paying" the 1-frame latency penalty.

If a manufacturer was very cost-conscious, they probably would not include the FPGA in the design, but as a hacker developer kit, we think it's an excellent exploratory option.

What could you do with it?

Embedded OSVR and what it can do for Phones and Game Consoles

What do you do when you need to add a VR peripheral like an eye tracker or a motion suit to a phone or a game console? Tear your hair out, probably.

There are multiple reasons why this is not easy. Depending on the particular setup, common reasons are:

• Closed programming environment that prevents you from working with an unknown peripheral, without going through a tedious approval process.
• Not enough processing power to service the peripheral without negatively impacting the performance of the game.
• No physical connectivity: lack of available USB ports.
• Lack of device drivers supporting the particular peripheral.
• No appetite to connecting additional wires to an otherwise portable or wireless system.

The OSVR software framework might be able to help. OSVR has two parts: the "OSVR Server" and "OSVR client". The Server processes data as it comes from the various peripherals and sensors and then presents it to Client in a device-agnostic format. For instance, once an eye tracker interface has been defined, the server can translate the vendor-specific implementation of eye tracking data (such as for an SMI eye tracker) into the universal device-independent reports. The Client is part of the game or VR application that uses this data.

In many cases, the Server and Client will reside on the same CPU (or sometimes even in the same process), and that is the standard deployment for a PC environment. However, the Client and Server can also be connected through a network interface such as WiFi or Bluetooth. In fact, a Client could connect to multiple servers, some residing locally, and some residing elsewhere over a network.

What does this mean? It means that you can connect VR peripherals to one CPU and run the game on another CPU.

For instance:

• Game (including OSVR Client) runs on phone. Peripherals (positional tracking, for example) including OSVR server runs on PC. Game receives 60 FPS updates from PC via Wifi or Bluetooth. Since PC does all the heavy lifting, the 60 FPS position updates are very short and low-bandwidth. 
• Game runs on XBOX, eye tracking runs on external CPU (such as super-compact single-board computers from Gumstix or Variscite) which uses the OSVR server to provide the universal eye tracking interface. By the way, these single-board computers could run OSVR using the operating system of your choice - Linux, Android, Windows Embedded) regardless of what operating systems runs with the client. 

My guess is that we will see additional examples of this over the coming months.

What would you use this for?

Advancing VR support for Mac

Given that Oculus/Facebook mothballed support for VR on Mac, the OSVR team at Sensics decided to step up and fill that void.

Once successful, this will provide an active path to continue to work with Oculus (and many other HMDs) on Mac. It will also pave the way to make use of the many capabilities of OSVR including positional and predictive tracking, device-independent integration for cameras, gestures, skeleton, locomotion devices and more.

We are seeking community assistance in making this happen faster. Below are short descriptions of what we know and what we are seeking. Comments, feedback and offers to assist will be most appreciated. 

What we know 

• OSVR is built to be multi-platform. We have OSVR working on Windows, Android and Linux and have had some reports of success in working on Mac. 
• We have OSVR building on both Linux (with Clang and GCC) and Windows. All the commits must pass testing on the CI for Linux and Windows. 
• OSVR supports multiple HMDs including those made by Oculus, OSVR, Sensics, Vuzix. Vive support is coming, as well as better integration with Cardboard. Support includes display parameters, distortion. Asynchronous time warp supported on Windows/NVIDIA environment but we are working to expand this to AMD/Intel as well as other platforms. 
• OSVR has been integrated into WebVR. Bugzilla patch has been submitted and reviewed by the Mozilla team. WebVR demos work on top of OSVR-supported HMDs. This means that OSVR work will also help support WebVR on Mac and on other platforms
• We know OSVR was successfully built on Mac last December (prior to public release), and all external dependencies have Mac support, so we should have a good idea of the missing pieces.

Status of Facebook/Oculus support in OSVR

Display support for Oculus in OSVR is already entirely independent of Oculus SDK. It uses distortion parameters from OSVR Distortionizer and our own distortion shader, and display descriptor. So, OSVR support for Oculus on Mac comes down to sensor access (and sensor fusion, as in the Oculus case, the SDK or runtime does the IMU fusion). 

Currently, the OSVR-Oculus plugin master branch uses a VRPN driver written to access the Rift SDK, which is built against a 0.4.x release, and fully functional and multi-platform. For instance, here is a demo of OSVR running Rift on Linux. Here is another demo showing the OSVR Palace demo running on Unity over Linux with OSVR/Oculus:

A branch contains a direct OSVR PluginKit driver (no VRPN involved) that builds against a newer Rift SDK, but there are bugs in it, so it is unfinished. If looking to handle Rift on "officially unsupported" (by Facebook) platforms, including Mac and Linux, will want to develop an OSVR PluginKit driver using an open-source driver stack. 

There are several, though some of them claim a non-copyleft license while incorporating some well-known GPL sensor fusion, so problematic that way. OpenHMD appears intended to be a broader approach than just an open-source sensor access for Oculus, but is BSL1.0 licensed and includes sensor fusion, so could serve the purpose. There are almost surely others and we would be happy to receive referrals.

Contributions/assistance wanted

Starting with the OSVR codebase (osvr.github.io): 

• Set up homebrew builds for libfunctionality and jsoncpp (if needed) 
• Potential implementation (libltdl or other) needed in libfunctionality (which is a simple wrapper for loading plugins at runtime) 
• Set up homebrew build for OSVR-Core (though it won't build fully at first). Known implementation details needed: 
• usbserialenum - could just use dummy impl right now 
• plugin search path code (in PluginHost) 
• Provide input on best open-source library to access Rift sensors 

 Comments, feedback and offers to assist will be most appreciated. Let's do this together!

OSVR Roadmap: Creating an Ecosystem of Interoperable VR Hardware and Software

OSVR (Open Source Virtual Reality) aims to create an open and universal standard for the discovery, configuration, and operation of VR/AR devices. Created by Sensics and Razer, OSVR includes two independent components: 1) the OSVR software platform and 2) the open-source Hacker Development Kit VR headset.

Since the OSVR launch in January this year, nearly 250 organizations including Intel, NVIDIA, Xilinx, Ubisoft, Leap Motion, and many others have joined the OSVR ecosystem. Concurrent with the expansion of the OSVR community, the capabilities of the software platform have grown by leaps and bounds.

The Sensics team architected the OSVR software platform and is its official maintainer. Below, I describe my personal perspective on the road ahead for the OSVR software along several paths: interfaces and devices, game engines, low-latency rendering, operating systems, utilities, and high-level processing.

The OSVR HDK is modular with open-source design plans

The Big Picture

The goal of the OSVR software framework is to make it easy to create compelling, high-performance VR/AR applications that:

• Work on as many VR/AR displays and peripherals as possible.
• Support even those devices that were not available at the time the application was created. Just like you don’t need to upgrade your word processor when you buy a new printer, you should not have to upgrade your game when a new HMD becomes available.
• If desired, can run on a wide range of operating systems and computing platforms.
• Take advantage of the unique features of individual devices, as opposed to reaching for the ‘lowest common denominator’.
• Are not locked into a particular device, peripheral, development environment, programming language or app store.

Directory of OSVR projects

The OSVR framework aims to achieve these goals while open-sourcing the vast majority of the software in order to:

• Encourage participation from the broader VR/AR community.
• Provide adopters the security and peace of mind that they desire.
• Accelerate the pace of development by including a wide community of contributors.
• Allow adopters to customize the platform to their specific needs.

Last, OSVR leverages existing open-source projects (OpenCV, CMake, VRPN) and is designed with a modular, plugin-based architecture so that:

• Participants can opt to keep modules closed-sourced so as to protect IP.
• Adopters can choose a small footprint deployment by choosing only the modules they need.
• Functionality such as support for new devices can be added after the fact.

Interfaces and Devices

Overview

In “OSVR speak”, an interface is a pipe for data of a certain type (“interface class”). Devices are said to expose interfaces to the OSVR core, and in turn, to an application. Such interfaces include tracker (provides position, orientation, or full pose), button, analog (axis of a joystick, analog trigger value), eye tracker (gaze direction), gesture, imager (single image or image stream), locomotion (for omnidirectional treadmills), skeleton (bone structure) and display (output device).

A single physical device may provide data in more than one interface class, just like a multi-function printer might look to an operating system as a printer, a scanner, and a fax. For instance, here are some interfaces exposed by popular devices:



Imagine that you are developing a viewer for 360 degree videos and you want to allow the users interact with it using gestures. With OSVR, because you work through a universal device-independent gesture interface, you can use any device that has an OSVR plugin. In the above example, these devices would be products from Leap Motion, NOD, YEI and Softkinetic. All these devices would expose gestures to the app in a standardized way. Contrast this approach with the hassle of having to integrate every one of these devices individually. Moreover, when new devices such as the Intel RealSense camera get an OSVR plugin that includes a gesture interface, they work immediately work with your app without having to change a single line of code.

Current Status

To date, the OSVR team has primarily focused on creating the various types of interfaces, connecting popular devices that illustrate that these interfaces work, and creating simulation plugins that allow developers to work with simulated data in lieu of using a real device. Aside from native OSVR plugin, OSVR also inherits support for about 100 devices from VRPN, a popular open-source project.

Future Plans

In the coming months, I believe that the OSVR team, OSVR hardware partners, and other OSVR contributors will significantly expand the number of devices supported by OSVR. Specifically, I am aware of plans and work in progress to support Intel Realsense, Tobii eye trackers, NOD devices, Leap Motion camera, HTC Vive and others.

Game Engines

Overview

OSVR performs a very important service for both game engines as well as for hardware devices that wish to be supported by these engines.

Figure 1 – Without OSVR: each device needs multiple plugins

The problem is that there are many graphics and game engines (Unity, Unreal, Crytek, VBS, Mongame, OpenVR, Unigine, WebVR, etc.) and numerous hardware devices. If each hardware vendor wants to be supported by each engine, a lot of drivers will need to be written and many person-years will be spent in integration and optimization. Moreover, this “many-to-many” connectivity (seen in Figure 1) puts a lot of stress on engine developers as they need to continuously educate and support hardware developers as well as relentlessly work to keep up with the latest hardware and driver versions.

Figure 2 – With OSVR: a single OSVR plugin makes it easy to support many engines

In contrast, when using OSVR, a game engine needs to have a single integration with OSVR to support all OSVR-supported hardware. The OSVR game engine integration provides a harmonized, standardized way to connect to all devices that support the relevant OSVR interfaces: whether tracker, gesture, skeleton, eye trackers or others. The OSVR team and community can work with each engine developer to create an optimized integration, so there is no need for hardware developers to learn and re-learn the intricacies of each engine. If a new hardware device comes on the market, the hardware manufacturer (or an interested member of the OSVR community) can create a device plugin for OSVR, thus automatically achieving connectivity into all OSVR-supported game engines. Because OSVR is open-source and plugin-based, the availability of such plugin does not depend on the priorities of Sensics or any other company. A developer can simply start from an existing open-source plugin and quickly modify it for the specific API of the new device. The result is illustrated in Figure 2.

Current Status

OSVR is integrated Unity, Unreal Engine, and Monogame

Future Plans

• A plugin for OpenVR (Valve) is in beta and should be completed soon. This will allow OSVR-supported displays and devices to be used with OpenVR/SteamVR games, though subject to the limitations of the OpenVR API.
• An OSVR backend for WebVR in Mozilla Firefox has been submitted to the Mozilla project and we expect it will become part of the Firefox WebVR nightly build very soon.
• A team of students is working on a Blender Game Engine plugin. As part of this effort, they are creating a Python wrapper for the OSVR Core client functionality, which should allow easy integration into other engines such as WorldViz Vizard.
• Possible integrations into CryEngine, Bohemia VBS3 and others are being discussed.

Due to conscious design decisions made when developing the client (game/application) API, because there are multiple language wrappers available for the native C OSVR API (C++, .Net and soon Python), and because existing integrations are open-sourced, it is easy to integrate OSVR into additional engines, including internal engines and custom applications.

Low-latency Rendering

Overview

The elapsed time from sensing to rendering—sometimes called ‘motion to photon’ latency—has been the subject of intense scrutiny towards creating more comfortable and immersive experiences. Latency comes from multiple sources including: how often do the sensors generate data? How quickly does that data get to the application? Are data points extrapolated into the future to create ‘predictive’ tracking? How quickly can the application render the image through the graphics stack? Can the application make ‘time warping’ corrections at the last instant?

Current Status

OSVR systematically addresses these points as follows:

• Data rate from sensors: The OSVR HDK provides 100 Hz positional data and 400 Hz “sensor-fused” orientation data.
• Speed in which data reaches the application: OSVR integrates with ETW (Event Tracing for Windows) which is a powerful tool for latency and performance analysis (See http://osvr.github.io/presentations/20150901-Intro-ETW-OSVR/for a brief tutorial). ETW helps optimize the full software stack—from the game to the sensors—towards achieving optimal performance.
• Predictive tracking: OSVR currently includes predictive orientation tracking. When using the OSVR HDK, this is derived from angular velocity reports that are provided at 400Hz to the controlling PC. When using HMDs that do not provide angular velocity, the velocity can be extracted from consecutive yaw/pitch/roll reports. Predictive tracking looks 16 milliseconds into the future and reduces the apparent latency.
• Direct render: The Sensics/OSVR Render Manager (supported on Windows/NVIDIA platforms and includes a Unity Plugin) provides optimal low-latency rendering on any OSVR-supported device. It includes:

1. Direct to HMD Mode: Enable an application to treat VR Headsets as head mounted displays that are accessible only to VR applications, bypassing rendering delays typical for Windows displays. Direct to HMD Mode is available for both Direct3D and OpenGL applications.
2. Front-Buffer Rendering: Renders directly to the front buffer to reduce latency.

• Time warping: OSVR includes Asynchronous Time Warp as part of the Render Manager. It reduces latency by making just-in-time adjustments to the rendered image based on the latest head orientation after scene rendering but before sending the pixels to the display. It includes texture overfill on all borders for both eyes and supports all translations and rotations, given an approximate depth to apply to objects in the image.

Future Plans

The OSVR team is working to expand the range of supported “Render Manager” graphics cards to include AMD and Intel. We are also looking to add Render Manager capabilities on Android and other non-Windows platforms by collaborating with graphics vendors for these platforms.

With regards to the Render Manager itself, in the very near future, the following enhancements will be released:

• Integrated Distortion Correction: handling the per-color distortion found in some HMDs requires post-rendering distortion. Today, OSVR performs this on the client/application side, but moving distortion correction into the Render Manager provides additional benefits. The same buffer-overfill rendering used in Asynchronous Time Warp will provide additional image regions for rendering.
• High-Priority Rendering: increasing the priority of the rendering thread associated with the final pixel scan-out ensures that every frame is displayed on time.
• Time Tracking: indicating to the application what time the future frame will be displayed lets it render the appropriate scene. This also enables the Render Manager to do predictive tracking when producing the rendering transformations and asynchronous time warp. The system also reports the time taken by previous rendering cycles, informing application when to simplify the scene to maintain an optimal update rate.
• Add Render Manager support in additional engines.
• Use ETW to perform additional engine and plugin optimizations.
• Design of a plugin-based API for advanced rendering capabilities. This would allow open-source release and public development of a cross-platform Render Manager stack with the ability to load vendor-specific code (which may be closed-source if required by those vendors).

Operating Systems

Overview

VR/AR products come in many shapes and forms and reside on several different computing platforms. The obvious examples are PC-based VR, which typically uses Windows, and mobile VR, which typically uses Android phones. The goal of OSVR is to both support a wide range of operating systems as well as provide a consistent programming interface regardless of the particular operating system. In an ideal case, OS-specific details (such as graphics drivers, file system details) as they relate to creating the VR/AR experience are all abstracted by OSVR.

Current Status

OSVR currently supports the following operating systems:

• Windows, including Windows-specific support for direct rendering.
• Android, including the ability to use internal sensors, camera and other on-board peripherals. Currently, this uses the CrystaX NDK to build native-code applications. Unity/Android also works on top of OSVR/Android.
• Linux: The OSVR engine has complete support for Linux and the code is tested on Linux before Windows binaries are released. Unity/Linux should be possible but has not been tested yet.

Future Plans

• Include Android and possibly Linux binaries in Unity plugin releases.
• Add OSX support.
• Add iOS support.
• Add RenderManager support to other platforms, working with platform vendors and manufacturers of graphics chips as required.
• Validate correct operation of additional game engines for non-Windows operating systems.
• Add plugins for OS-specific/platform-specific peripherals and capabilities.

Utilities

Overview

OSVR utilities are standalone programs that are perform useful functions in support of OSVR development and deployment. True to the OSVR philosophy, OSVR utilities are also open-sourced.

Current Status

The following utilities currently exist:

• Distortionizer: The distortionizer is an interactive tool that helps determine the RGB optical distortion parameters of a display. Sometimes, these parameters are provided by the optical design team. Other times, they need to be measured. The output of the distortionizer is a set of distortion correction coefficients that automatically feeds into the Render Manager.
• Latency test: This combines open-source hardware (based on low-cost Arduino components) and open-source software to provide end-to-end latency measurements.
• Tracker Viewer: a graphical utility to dynamically display position and orientation of several tracked objects.

Future Plans

I am aware of several additional utilities under development:

• Windows installer for the runtime components of OSVR.
• Interactive configuration utility to allow configuring eye separation, height and other user-specific parameters.

High-level Processing

Overview

High-level processing modules (Analysis Plugins in “OSVR speak”) are software modules that convert data into higher-level information. For instance, a gesture engine plugin can convert a stream of XYZ coordinates to a recognized gesture; an eye tracker plugin can take live feed from a camera pointed at the eye and provide real-time gaze direction; a sensor-fusion plugin can mesh data various tracker interfaces into more accurate reports.

Current Status

The OSVR team and community have primarily been focused on building the lower-level infrastructure before adding optional processing layers on top of them. At present, two analysis plugins exist:

• 1-Euro filter: This is a data smoothing filter that, as currently applied, improves stability of the output from a Razer Hydra.
• Predictive tracker: Estimates head orientation into the near future as a method to reduce perceived latency.

Future Plans

A unified API for easily developing analysis plugins and allowing their configuration is in progress. Several analysis plugins are under consideration and in various stages of design:
Sensor fusion: to combine the output from multiple sensors into more accurate information.
Augmented reality: to allow detecting objects in the scene.
Eye tracking: to convert pupil camera images into pupil coordinates.

We are working to simplify the process of writing analysis plugins, to provide open-source examples and are very open to community contributions.

Summary

OSVR, like VR in general, is a work in progress. I am very pleased with the quality of partners, the breadth of functionality, and the level of community involvement that we have seen in the eight months since launching OSVR. Having said that, there is still plenty of work to be done: whether in supporting new devices, supporting a wider range of engines and development environments, in making devices more useful by creating analysis plugins, or in providing high-performance rendering across multiple platforms. I am excited to have played a role in building OSVR and look forward to the months and years ahead.

The post "OSVR Roadmap: Creating an Ecosystem of Interoperable VR Hardware and Software" by vrguy was first published in RoadToVR

Using OSVR Software in Professional-Grade Simulators

A few weeks ago, I met with representatives from a large multi-national defense contractor. They are looking for high-performance HMDs for a new simulator and wanted to explore some of the higher-end Sensics products.

After reviewing the HMDs, the conversation turned to software. As is often the case, this company uses a special-purpose engine (as opposed to a game engine like Unity or Unreal). The precise tracking hardware for their simulator is still in flux, and could also change between the development and deployment phase. The HMD hardware could also evolve over time.

How can they mitigate development costs in spite of the changing hardware? The easy answer: use OSVR software.

The OSVR software framework supports many HMDs and dozens and dozens of tracking systems and input devices. These are all presented to the application software under one common software interface. For software that works on top of OSVR, supporting a different hardware configuration is as simple as changing a couple of configuration files. No need to recompile the application. Integrating OSVR into a custom game engine is also fairly simple because OSVR:

• Includes multiple language bindings
• Supports asynchronous (callback) as well as synchronous (blocking) function calls
• Provides open-source examples on how to integrate OSVR into other game engines such as Unity, Unreal, SteamVR and Monogame.
• Makes it easy to define new devices if required

Because OSVR is built on top of VRPN, a de-facto industrial standard in controlling VR devices, it enjoys a very wide range of common as well as esoteric tracking systems. Being free and open source, you can't beat it's price.

We're hoping to deliver HMDs to this exciting simulator, but regardless of what HMD is chosen, I think OSVR software should be strongly considered as a device-independent framework for the simulator.

What every VR game engine needs

When game engines are used for VR, they have to include many new capabilities: stereo rendering, higher frame rate, distortion correction, latency control and more. But one topic that is often overlooked is that VR game engine also have to deal with a wide variety of VR peripherals, each with their own API.

A non-VR engine primarily interfaces with a game controller, keyboard and mouse. These are well-understood peripherals with standard software interfaces. New types of keyboards, controllers and mice do appear on the market, but they share essentially identical interfaces to pre-existing devices.

In contrast, there is a much greater variety of VR devices: HMDs, motion and position trackers, hand and finger sensors, eye trackers, body suits, locomotion devices, force feedback devices, augmented reality cameras and more. Furthermore, each class of devices does not offer a common interface: working with a Leap Motion sensor is different than a Softkinetic one or an Intel Realsense one, even though the capabilities that they provide are similar.

The result? An endless effort to keep up. Consider the diagram below, showing just a small selection of available VR engines and a small subset of VR devices

VR devices and VR engines face a constant struggle to support each other

Consider a Sixense STEM motion controller. The Sixense team would probably like to make the STEM available to a widest possible range of software engines so that a developer can use STEM regardless of their engine of choice. Same goes for every other peripheral. Conversely, an Unreal Engine developer wants to support the maximum reasonable number of devices so that the game can reach the maximum number of users. Every permutation of VR device and game engine may have value. For the device vendor, there is a need how to build, test, optimize and deploy plugins for various engines. For the engine vendor, there is constant pressure - from both game developers and hardware makers - to offer device support.

That's why every VR game engine needs a middleware abstraction layer like the OSVR SDK. OSVR factors devices into common interfaces - tracker, skeleton, imager, etc. - and then provides a standard device-independent interface to the game engine. Just like desktop scanners offer a TWAIN interface that applications can use regardless of the scanner vendor, OSVR offers optimized skeleton, eye tracker and other VR interfaces that work regardless of the underlying hardware.

VR Middleware like OSVR dramatically simplifies the device and engine support effort

Consider the advantages. With OSVR:

• A device vendor writes a single interface to OSVR and immediately gains access to high-performance plugins to a wide range of game engines. In the particular case of OSVR, these plugins are free and open-source.
• Instead of attempting to work with hundreds and vendors and devices, an engine company can work with the OSVR team to create an optimized interface for each particular type of device. As new devices of the same type enter the market, supporting this new device is as easy as writing an OSVR plugin. Engine companies do not need to go over the effort of teaching yet another hardware vendor how to work well with their engine. The effort can be focused on optimizing the OSVR driver.
• Game developers are spared the effort of selecting particular vendors. By using the OSVR plugin for their favorite game engines, a wide range of devices is supported out of the box. 
• As new devices enter the market, games do no need to recompile and redistribute. In fact, that game might not even know that it is now supporting a new hardware vendor
• OSVR interfaces are defined collectively by industry leaders that understand the power of the abstraction layer. This is done in a transparent and open-source way.
• The number of engine and device combination increases, while the number of software interfaces to be written, debugged and optimized significantly decreases.

Game engines could still use vendor-specific function calls to activate some special functions of a peripheral, but the need to so quickly decreases.

Yet another advantage is the ability to add in-line analysis plugins to the data path. For instance, often times it is desired to convert hand and finger movements into higher-level gestures. The gesture engine could be vendor-specific, but that would limit its adoption in the game. It could be game-engine specific but that would limit market adoption of gestures as a user interface method. Or, it could be an OSVR plugin that works on top of all existing hand and finger sensors (using their standardized interface) and provides a standard gesture interface to many game engines. Furthermore, this gesture engine can be written by a company or a group that are experts in gestures, instead of forcing a hardware vendor or a game engine to master yet another technology.

Whether you make engines, hardware or smart analysis software, using VR middleware such as the OSVR API deserves serious consideration.

Open Source Augmented Reality?

Illustration from Meta SpaceGlasses

Can the OSVR project be used not just for virtual reality but also for augmented reality? Absolutely! Here why:

OSVR includes two independent parts: an open-source HMD (the "Hacker Development Kit") and the OSVR framework, a free and open-source software platform that provides an easy and standard way to discover, configure and operate a wide range of virtual reality and augmented peripherals.

Examples of such peripherals could be head trackers, hand and finger sensors (like Leap Motion and SoftKinetic), gesture control devices (such as the Myo armband and the Nod ring), cameras, eye trackers and many others. It turns out that most of these devices are used not only in virtual reality applications but also in augmented reality applications and so the services that the OSVR framework provides are just as useful for AR as they are for VR.

What are these services that OSVR provides? There are many different services, but some of them are:

• Discovery. Determine what peripherals and devices are connected to the system
• Configuration and autoconfiguration. Configure each peripheral and the interconnection between them. 
• Operation. Extract data and events from the peripherals and pass it on to the application in a standardized way. Support both a state (synchronous) and an event (asynchronous) programming model or any combination thereof.
• Store and restore configuration to/from a file or the cloud. 
• Provide user parameter such as eye separation.
• Provide optimized connectors to popular engines such as Unity and Unreal.
• Support of multiple operating systems and hardware platforms.

For instance, if you are developing an application that uses the Leap Motion sensor but also want users to operate it with the SoftKinetic camera, you have two options:

Option one: do it yourself and learn the individual APIs of each of these cameras and create multiple pieces of code to support them. Then, as these APIs evolve over time, you have to continuously upgrade your application. After a while, you want to use yet a third type of camera, say the Intel RealSense, so now you need to add yet another branch of code. Of course, each of these devices may report data differently. They may use different coordinate systems, or different units. You application needs to handle all these configurations.

Option two: use OSVR. You need to learn just one API which abstracts the various types of cameras. As new cameras are added, the hardware vendor (or the OSVR community) are likely to quickly add a new plugin to OSVR, meaning that your application works the same way with just a download of a new OSVR plugin. If the community does not work fast enough, OSVR is open-source and well documented so that you can write a plugin yourself. Coordinate systems and units are consistent. You are not strongly dependent on one particular piece of hardware.

OSVR also provide a growing list of analysis plugins that latch onto the output of device plugins and provide high-value processing such as a gesture engine (converting motion into gestures), data smoothing, target recognition and more. 

Next time you think about our AR software architecture, think OSVR. 

A brief overview of the OSVR source-code repositories

The OSVR team opened up most of the source code repositories to the public this weekend, and a few additional repositories will be opened in the coming days. Because Sensics is a founding contributor to this open-source VR project, many have asked us for a brief overview of the project.

The best place to start is osvr.github.io. If you haven't read the 'introduction to OSVR whitepaper' you might want to do so.

There are several github projects under the /osvr organization. They are as follows:

Key projects:

• OSVR-Core : this is the heart of the project. The OSVR_server executable connects the game to the OSVR hardware and software components.

Utilities:

• OSVR-Tracker-Viewer is a utility that graphically shows the position and orientation of the head and hand controllers. It is also an OSVR client, and thus an example on how to connect to and extract data from the server
• Distortionizer: a utility to estimate distortion correction parameters for various HMDs and a shader to implement the parameters estimated by the Distortionizer. OSVR has JSON descriptor files for HMDs (and many other objects) and the distortion parameters are part of that JSON file

Game engine plugins:

• OSVR-Unity includes a prefab component that can be imported into Unity. 
• OSVR-Unreal (to be released later this week) is an Unreal Engine plugin

Development tools:

• OSVR-Boxstarter is a Boxstarter install that helps quickly set up a development environment on a Windows machine
• OSVR-JSON-Editor is the source code for a tool (deployed version here) that helps create and edit the JSON descriptor files
• OSVR-JSON-Schemas is a repository for such JSON files

Plugins:

• OSVR-Oculus-Rift provides a plugin that allows using the position and orientation data of an Oculus device inside OSVR. 
• OSVR-Vuzix (to be released later this week) does the same for Vuzix headsets

Additional projects are coming. There is also a wiki page. Issues are currently tracked as part of the Github pages of the projects and we are looking to add an open-source project management tool.

OSVR (licensed under Apache 2.0 license) aims to create a multi-platform framework to detect, configure and operate a wide range of VR devices as well as to allow smart plugins that turn data into useful information. This is a big undertaking and there is a lot more work to be done to fulfill that vision, so we are looking for all the help we can get. I am super encouraged by the support and feedback we are getting from developers all over the world that believe in the open-source concept, and want to make their contributions towards moving VR forward. 

Let me know what you think. What's missing, what your priorities are and how we can get you involved. Welcome to OSVR.