The Video Processing FPGA inside the OSVR HDK and the Sensics dSight

Now that OSVR hacker developer kits are in the wild, @skyworxx Mark Schramm has posted some teardown photos of the 1.2 version of the HDK. He notes the FPGA on the board, and I thought I'd take the opportunity to explain what the FPGA can do in the HDK, and in it's high-end brother the dSight

The FPGA and the DRAM chips next to it can perform real-time video processing, giving hackers the ability to experiment with hardware-level transformations that do not require cooperation from the video-generating device or application.

Some of the options that are currently implemented:
1. Pass-through. This is the simplest mode and it does not involve the DRAMs. 1080x1920 video from the HDMI receiver is essentially passed through to the MIPI interface that drives the OLED screen.

2. Image rotation. This mode allows real-time 90 degree rotation, so that standard landscape mode 1920x1080 video can be presented in the 1080x1920 display. To do this, a full video frame is stored in the DRAM chips while the previous frame is sent to the display. This 90 degree rotation does cost you 1 frame of latency, but can be very useful in some of the following scenarios:

• Video coming from a 1920x1080 source such as as DVD player or in "replicated desktop" mode.
• Video coming over a low-latency wireless link. These links primarily support 1920x1080 today and not the native 1080x1920 mode.

3. Conversion of full screen to side-by-side mode. Ever saw a desktop in the HMD and found yourself squinting to see each half at a time. When this conversion mode is enabled, the video signal is converted into two identical copies of the signal, that can then be shown on both eyes at the same time. Control over this mode is via a command via the HID feature interface or with a simple utility.

Additional modes that are not yet implemented but can be implemented by the community:

• Real-time distortion correction. If you have a non-cooperative video source or just prefer to use your GPU for something else, real-time distortion correction in the FPGA can be useful.
• Resolution up-scaling: converting from lower resolution into the full resolution of the HDK
• Color enhancements (e.g. gamma, HSI improvements)
• Rearrange the video signal. One cool application that we saw from one of our partners is rearranging a 1080x1920 on the output of a GPU to reformat it into a non-legible 1920x1080 image, send that over the low-latency wireless video link and then use the FPGA to unscramble the image. This allows wireless video transmission without "paying" the 1-frame latency penalty.

If a manufacturer was very cost-conscious, they probably would not include the FPGA in the design, but as a hacker developer kit, we think it's an excellent exploratory option.

What could you do with it?

Embedded OSVR and what it can do for Phones and Game Consoles

What do you do when you need to add a VR peripheral like an eye tracker or a motion suit to a phone or a game console? Tear your hair out, probably.

There are multiple reasons why this is not easy. Depending on the particular setup, common reasons are:

• Closed programming environment that prevents you from working with an unknown peripheral, without going through a tedious approval process.
• Not enough processing power to service the peripheral without negatively impacting the performance of the game.
• No physical connectivity: lack of available USB ports.
• Lack of device drivers supporting the particular peripheral.
• No appetite to connecting additional wires to an otherwise portable or wireless system.

The OSVR software framework might be able to help. OSVR has two parts: the "OSVR Server" and "OSVR client". The Server processes data as it comes from the various peripherals and sensors and then presents it to Client in a device-agnostic format. For instance, once an eye tracker interface has been defined, the server can translate the vendor-specific implementation of eye tracking data (such as for an SMI eye tracker) into the universal device-independent reports. The Client is part of the game or VR application that uses this data.

In many cases, the Server and Client will reside on the same CPU (or sometimes even in the same process), and that is the standard deployment for a PC environment. However, the Client and Server can also be connected through a network interface such as WiFi or Bluetooth. In fact, a Client could connect to multiple servers, some residing locally, and some residing elsewhere over a network.

What does this mean? It means that you can connect VR peripherals to one CPU and run the game on another CPU.

For instance:

• Game (including OSVR Client) runs on phone. Peripherals (positional tracking, for example) including OSVR server runs on PC. Game receives 60 FPS updates from PC via Wifi or Bluetooth. Since PC does all the heavy lifting, the 60 FPS position updates are very short and low-bandwidth. 
• Game runs on XBOX, eye tracking runs on external CPU (such as super-compact single-board computers from Gumstix or Variscite) which uses the OSVR server to provide the universal eye tracking interface. By the way, these single-board computers could run OSVR using the operating system of your choice - Linux, Android, Windows Embedded) regardless of what operating systems runs with the client. 

My guess is that we will see additional examples of this over the coming months.

What would you use this for?