The Sensory Explosion

At last week's SIGGRAPH conference, I had the pleasure of contributing a "Sensory Explosion" presentation.on the "Sight, Sounds and Sensors" panel.

Below are the key slides and some speaker notes. Enjoy!

The panel featured several contributors:

• Cynthia Traeger. Pacific
• Mark Mine, Walt Disney Imagineering
• Chris Mayhew, Vision III Imaging, Inc.
• Becky Oh, PNI Sensor Corporation
• Myself

Sensics has been doing VR for a long time. Historically, it has mostly focused on enterprise applications (government, academia, corporate) and it is considering how to best leverage its technologies and know-how into larger markets such as medical devices and consumer VR.

Traditionally, head-mounted displays had three key components: a display (or multiple displays), adaptation optics and most often an orientation sensor. Most of the efforts were focused on increasing resolution, improving field of view and designing better optics. The orientation sensor was necessary, but not the critical component.

Recently, we are seeing the evolution of the HMD as a sensory platform. On top of the traditional core, we see the emergence of new types of sensors: position trackers, eye tracking, cameras (either for augmented reality and/or depth sensing), biometric sensors, haptic feedback, sensors that perform real-time determination of hand and finger position and more. Increasingly, innovation is shifting to how to make these sensors deliver maximum performance,  lightest weight (after all, they are on the head), and utmost power-efficiency (both for heat dissipation reasons as well as battery life for portable systems)

Above and beyond these on-board sensors, VR applications can now access sensors that are external to the HMD platform. For instance, most users carry a phone that has its own set of sensors such as a GPS. Some might wear a fitness or be in a room where a Kinect or some other camera can provide additional information. These sensors pose an opportunity for application developers to know even more about what the user is doing.

Integrating all these sensors can be pretty complex pretty quickly. Above is a block diagram of the SmartGoggles(tm) prototype that was done by Sensics a few years ago. These days, there is a much greater variety of sensors, so what can be done about them?

I feel that getting a handle on the explosion of sensors requires a few things:
1. A way to abstract sensors, just like VRPN abstracted motion trackers.
2. A standardized way to discover which sensors are connected to the system.
3. An easy way to configure all these sensors, as well as store the configuration for quick retrieval
4. A way to map the various sensor events into high-level application events. Just like you might change the mapping of a the buttons on a gamepad, you should be able to decide what impact does a particular gesture, for instance, have on the application.

But beyond this "plumbing", what is really needed is a way to figure out the context of the user, to turn data from various sensors into higher-level information. For instance: turn the motion data from two hands into the realization that the user is clapping, or determine that a user is sitting down, or is excited, or happy or exhausted.

We live in exciting times with significant developments in display technologies, goggles and sensors. I look forward to seeing what the future holds, as well as to make my contribution to shaping it.

Positional tracking: "Outside-in" vs. "Inside-out"

Optical positional tracking for goggles uses a camera (or cameras) and a known set of markers to determine the position of the camera relative to the markers. Positional tracking can be done using the visible spectrum but is more commonly done using infra-red markers and a camera that is sensitive to IR light.

There are two main options:

• Inside-out tracking: where the camera is based on the goggles and the IR markers are placed in stationary locations (e.g. on the computer monitor, on the wall, etc.)
• Outside-in tracking: where the camera is placed in a stationary location and the IR markers are placed on the goggles.

In both cases, the targets sometimes flash in a way that is synchronized with the camera. This allows reducing power consumption for the targets and helps reduce tracking noise from IR sources that are not the targets.

Sensics dSight panoramic HMD with IR targets for "outside in" tracking

How do these approaches compare?

• Tracking volume: in both cases, at least some of the targets need to be visible to the camera. When the user rotates the head, an "inside-out" system needs targets that are physically far apart. If the targets are, for instance, placed on the bezel of a notebook PC, it is easy to see how head rotation could easily take these targets out of the field of view of the camera. A wider lens could be used in the camera, but this would reduce the tracking precision as each camera pixel would now cover a greater physical space in the world. In the "outside-in" system, targets could be placed on most sides of the goggle, allowing reasonably large rotation while still having targets visible on the camera. Advantage: outside-in
• Tracking inside an entire room: if we want to allow mobility within a room, an 'inside-out' system would require additional markers on the walls, whereas an 'outside-in' system would require additional cameras. Both systems would require room calibration to make sure the target and/or cameras are placed in a known position. Additional cameras require additional processing power. Slight advantage: inside-out
• Where is data being processed? In "inside-out' tracking, the camera data is either processed on the goggle or the camera is connected to a computer that is either carried by the user or stationary and connected via a wire. In 'outside-in' tracking, the data is processed on a computer that could be stationary. Advantage: outside-in
• Can this be used with a wireless goggle? If the goggle is not tethered to a computer, "inside-out" tracking requires that the data is either processed locally or that the camera signal is sent wirelessly to a base station. In contrast, an 'outside-in' approach does not require wireless data of the camera. At most, a synchronization signal can be sent to the IR LEDs to make sure they flash in sync with the camera. Advantage: outside-in
• Ability to combine with augmented reality system. Sometimes, the goggle will already have an on-board camera (or cameras) for the purpose of augmented reality and/or 3D reconstruction. In that case, using the same camera for positional tracking may have some cost advantages if positional tracking can be used with visible targets or if the camera already has IR sensitivity. Advantage: inside-out

Note: tracking accuracy is also an important comparison parameter, but this is more difficult to generically compare across both approaches. Very often, accurate tracking is achieved not just through the camera data but also by integrating ("sensor fusion") rotational data and linear acceleration data from on-board sensors. The sensors used and the quality of the sensor fusion algorithm would determine which approach is better.

Bottom line: for most applications, an 'outside-in' approach would be better, and thus we expect to see a greater number of 'outside-in' solutions on the market.

What has been your experience?


For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here

Wearable but Intrusive?

From: Universitat Pompeu Fabra, Barcelona

At last month's E3 show, I had the opportunity to experience an early version of ControlVR, dubbed a "wearable controller" by its creators.

ControlVR is comprised of a set of sensors that attach to the torso, arms and fingers. Once properly calibrated, they can provide fine-grained reporting of the position of each finger and use this information in a variety of virtual reality applications.

For a prototype, the system worked very well, and I am sure it will improve both in looks and performance as it heads towards a production model.

My concern, though, will users really be willing to a wear such device for a extended periods of time. Such a design is relatively intrusive - takes a while to strap on as well as touches the user in so many points. This is not specific to ControlVR. The same could be said of the PrioVR suit or similar devices.

To answer the question, one needs to consider the benefits of such devices vs. the alternatives. I am sold on the benefits of finger tracking, or full-body tracking for that matter. But, there are alternatives (Kinect, Leap Motion, others) that do not require so many touch points on the user and deliver, in my opinion, similar benefits in most situations.

A virtual reality goggle is also intrusive. It touches your face, potentially messes your hair and sometimes requires you to remove glasses, just to name a few. However, there is no cost-effective alternative to achieving the immersion and portability of goggles these days. Whether that is also the case for devices such as ControlVR and PrioVR - I am not sure.

What do you think?

Playing with the Pros

Source: DontWorry.TV

They say that a common dream is to be able to fly, or sing. I'm sure virtual reality can help with the flying part.

Kids who play sports want to play - at least once - in a professional game. "Bend it like Beckham" or play with LeBron.

180 cm (5'11") in height, sometimes overweight, I am not tall enough to dunk nor light enough to fly.

So there had to be another way of playing with the pros.

For the next week, I'll be taking a short break from VR activities. My violin in hand, I'll be dedicating most of my time to practicing with like-minded individuals and members of the Baltimore Symphony Orchesta towards a concert next weekend at Symphony Hall. Conducted by the BSO's music director Marin Alsop, the concert will feature wonderful (and wonderfully difficult to play) pieces by Dvorak, Debussy, Tchaikovsky, Mahler and Berlioz. Below are a recording of two parts from these pieces.

I'll be back next week with more E3 thoughts, positional tracking information and more. But for now, let the music begin.

Creating VR games and software designed for cross-headset compatibility

With multiple consumer VR headsets now on the horizon, will it be difficult for developers to support multiple VR headsets from a single application, rather than creating custom distributables for each individual headset?

Interested? Read my guest blog entry on RoadToVR to find out.

An overview of positional tracking technologies for VR

Photo Credit: 
Saad Faruque via Compfight

Positional tracking is very important towards achieving immersion and presence in virtual reality. Whether it is the head, arms, fingers or objects (such as a weapon), positional tracking can deliver multiple benefits:

• Change the viewpoint of the user to reflect actions such as jumping, ducking or leaning forward.
• Show hands and other objects in the displayed image. A common complaint of users in virtual reality is that they can't see their hands.
• Connect the physical and virtual world. For instance, by detecting hand position, a software program can implement the option to move virtual objects by touching them.
• Detect gestures. By analyzing position over time, a gesture can be detected. For instance, a user might draw the number "8" in air and have the software detect it.

There are several methods of tracking position and I felt it is worthwhile to describe some of them. This post focuses on tracking for virtual reality applications, so we will not look at vehicle tracking, tracking of firemen in buildings and so forth. In no particular order, here are some of the popular tracking methods include magnetic, inertial, optical and acoustic tracking as well as hybrid tracking that combines multiple methods.

I've described these tracking methods as well as others such as depth map in a guest blog post at RoadToVR. Please click here to read that post on the RoadToVR site.


For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here

New iPhone utility app simplifies VR calculations

A few months ago, I realized we go back again and again to the same Excel spreadsheets for some VR calculations. For instance, if the field of view is 90 degree diagonal and the aspect ratio is 16:9, what is the horizontal field of view? If using a goggle is like watching a 70 inch TV from 6 feet, what does that say about the field of view?

To help with this, the Sensics team decided to create a simple iPhone app (Android version coming soon) that helps with these calculations. This app is now available as a free download from iTunes. Think about it as public service for the VR community.


In it's current beta form, the app provides several useful conversions:

• TV screen size to/from goggle field of view
• Screen size and aspect ratio to/from field of view
• Quaternion to/from Euler angles

The app also includes useful reference info as well as a way to ask the VR experts at Sensics.

Some of this math already appeared on this blog such as here and here, but it is now available in your pocket.

What additional calculators would you find useful? What changes would you like to see in the app? Any bugs that need to be fixed? Let me know.