Notes from the Zero Latency Free-Roam VR Gameplay

I spent this past weekend in Australia working with Sensics customer Zero Latency towards their upcoming VR deployment at SEGA's Joypolis park in Tokyo. As part of the visit, I had the chance to go through the Zero Latency "Zombie Outbreak" experience and I thought I would share some notes from it. Zero Latency has been running this experience for quite some time and have had nearly 10,000 paying customers do it. The experience is about 1-hour long including about 10 minutes of pre-game briefing and equipment setup, 45 minutes of play and 5 minutes to take the equipment off and get the space ready for the next group. There are 6 customer slots per hour and everyone plays together in the same space at the same time. To date, Zero Latency has opened this to customers for about 29 hours a week - mostly on weekends - but will now be adding weeknights for a total of 40 game hours per week. A ticket costs 88 Australian dollars (about 75 US dollars) and there is typically a 6-week waiting list to get in. The experience is located in the Zero Latency office, a converted warehouse in the north side of Melbourne Australia. Most of the warehouse is taken up by the rectangular game space, about 15 x 25 meters (50 x 80 feet), or 375 m² (4000 sq ft) to be precise. The rest of the warehouse is used for two floors engineering and administrative offices.  One can peak through the office windows at the customers playing and during the day you can constantly hear the shouts of excitement, squeals of joy and screams of horror coming from the game space.

I had a chance to go through the game twice: once with a group of Zero Latency employees before the space was opened to customers, and once as the 6th man of a 5-person group of paying customers late night. Once customers come in they are greeted by a 'game master' that provides a pre-mission briefing, explains the rules and provides explanation on the gaming gun. The gun can switch between a semi-automatic rifle and a shotgun. It has a trigger, a button to switch modes, a reload button and a pump to load bullets into the shotgun and load grenades when in rifle mode. I found the gun to be comfortable and balanced, and it seems that is has undergone many iterations before arriving in the current form. Players wear a backpack that includes a lightweight Alienware portable computer, a battery and a control box. The HMD and the gun have lighted spheres on them - reminiscent of the PlayStation Move - that are used to track the players and the weapons throughout the space. Players also wear Razer headsets that provide two-way audio so that players can easily communicate with each other as well as hear instructions from the game master.

The game starts with a few minutes of acclimation where players walk across the space to virtual shooting range and spend a couple of minutes getting comfortable with operating their weapons. The game then starts. It is essentially a simple game - players fight their way through the space while shooting zombies and other menacing characters, some of which shoot back at you. Every few minutes, players switch scenes by going through an elevator or teleportation waypoints, circles on the ground where each of the six players has to stand before the next scene can be reached. Sometimes you fight in an urban setting, sometimes on a rooftop, inside a cafeteria and so forth. Zombies can be killed by a direct shot to the head or multiple shots to the body. The players can also be killed, but then return to the game after about 10 seconds of appearing as a 'ghost'.  Game 'power ups' are sometimes found through the space. For instance, during my gameplay I found an AK-47 assault rifle and later a heavy machine gun. At the end of the game, each player is shown their score and ranking, where the score is calculated based on the number of kills and the number of player deaths. That score sheet is emailed to players and is available for later viewing on the Web. The graphics are fine and an attacking zombie is quite compelling when it is right in your face, but the things I truly found compelling in the game are not so much the graphics and gameplay but rather a few other things:

• Free-roam VR is great. The large space offers fantastic freedom of movement. You can see players move throughout the space, duck to take cover, turn around quickly with no hesitation at all. This generates an excellent feeling of immersion. You can truly feel that you could hide behind corners or walk anywhere with no apparent limitations. Of course, every space has physical limitations and Zero Latency has implemented a system where if you get too close to a player or a wall, something like a radar appears on your screen showing you at the center and the obstacles (players, walls) on it so that you know how to avoid them. If you get too close, the game pauses until you are farther away. This felt very natural. Throughout nearly two hours of active gameplay I think I brushed once or twice against another player but no more than that, even though players were in close proximity. Immersion is such that players don't notice people that are not players around them. In the current Zero Latency office, the bathroom for the office (the "Loo" in "Australian") is right across from the playing space so to get there you can either take a detour walking alongside the walls or go straight through the playing area where the players couldn't care less because they don't even know that you are walking by.

• The social aspect is very compelling. This game is not about 6 individuals playing separately in a space. It is about 6 players acting as a team within the space. You can definitely hear "you take the right corridor and I'll take the left", or "watch your back" or "I need some help here!" shouts from one player to another. Players that work individually have little chance to stop the zombie invasion coming from all directions, but playing together gives you that chance.
• Tracking - for both the head and the weapon - are very smooth to the point where you don't think about it. Because multiple players are tracked in the space you can see their avatars around you (sometimes with name tags). The graphics of players walking in the game need some work in my opinion, but you can clearly see where everyone is and what they are doing.
• 45 minutes of game play go by very quickly and the game masters control the pace very well. As you can imagine, some groups take longer than others to get to the next waypoints, and the game uses waiting for elevators or helicopters as a way to condense or extend the total time. For instance, once you arrive in the cafeteria a sign shows up that the elevator will arrive in 100 seconds. I would imagine that if a group arrived earlier, they would have to wait longer for the elevator or if a group took more time, they would wait less.

The space itself is essentially empty save for the overhead tracking cameras. Thus, the same space the tracking system can be used for many different experiences. Unfortunately, we had to work from time to time and I did not have a chance to try some of the newer experiences that Zero Latency is working on, especially since the space was occupied by customers most of the time. I'm certainly looking forward to coming there again and continue to save the world.

Understanding Predictive Tracking

Image source: Adrian Boeing blog

In the context of AR and VR systems, predictive tracking refers to the process of predicting the future orientation and/or position of an object or body part. For instance, one might want to predict the orientation of the head or the position of the hand.

Why is predictive tracking useful?

One common use of predictive tracking is to reduce the apparent "motion to photon" latency, meaning the time between movement and when that movement is reflected in the drawn scene. Since there is some delay between movement and an updated display (more on the sources of that delay below), using an estimated future orientation and position as the data used in updating the display, could shorten that perceived latency.

While a lot of attention has been focused on predictive tracking in virtual reality applications, it is also very important in augmented reality. For instance, if you are displaying a graphical overlay to appear on top of a physical object that you see with an augmented reality goggles, it is important that the overlay stays on the object even when you rotate your head. The object might be recognized with a camera, but it takes time for the camera to capture the frame, for a processor to determine where the object is in the frame and for a graphics chip to render the new overlay. By using predictive tracking, you can get better apparent registration between the overlay and the physical object.

How does it work? 

If you saw a car travelling at a constant speed and you wanted to predict where that car will be one second in the future, you could probably make a fairly accurate prediction. You know the current position of the car, you might know (or can estimate) the current velocity, and thus you can extrapolate the position into the near future.

Of course if you compare your prediction with where the car actually is in one second, your prediction is unlikely to be 100% accurate every time: the car might change direction or speed during that time. The farther out you are trying to predict, the less accurate your prediction will be: predicting where the car will be in one second is likely much more accurate than predicting where it will be in one minute.

The more you know about the car and its behavior, the better chance you have of making an accurate prediction. For instance, if you were able to measure not only the velocity but also the acceleration, you can make a more accurate prediction.

If you have additional information about the behavior of the tracked body, this can also improve prediction accuracy. For instance, when doing head tracking, understand how fast the head can possibly rotate and what are common rotation speeds, can improve the tracking model. Similarly, if you are doing eye tracking, you can use the eye tracking information to anticipate head movements as discussed in this post

Sources of latency

The desired to perform predictive tracking comes from having some latency between actual movement and displaying an image that reflects that movement. Latency can come from multiple sources, such as:

• Sensing delays. The sensors (e.g. gyroscope) may be bandwidth-limited and do not instantaneously report orientation or position changes. Similarly, camera-based sensors may exhibit delay between when the pixel on the camera sensor receives light from the tracked object to that frame being ready to be sent to the host processor.
• Processing delays. Sensors are often combined using some kind of sensor fusion algorithm, and executing this algorithm can add latency.
• Data smoothing. Sensor data is sometimes noisy and to avoid erroneous jitter, software or hardware-based low pass algorithms are executed.
• Transmission delays. For example, if orientation sensing is done using a USB-connected device, there is some non-zero time between the data available to be ready by the host processor and the time data transfer over USB is completed.
• Rendering delays. When rendering a non-trivial scene, it takes some time to have the image ready to be sent to the display device.
• Frame rate delays. If a display is operating at 100 Hz, for instance, there is a 10 mSec time between successive frames. Information that is not precisely current to when a particular pixel is drawn may need to wait until the next time that pixel is drawn on the display.

Some of these delays are very small, but unfortunately all of them add up and predictive tracking, along with other techniques such as time warping, are helpful in reducing the apparent latency.

How much to track into the future?

In two words: it depends. You will want to estimate the end-the-end latency of your system as a starting point and then optimize them to your liking.

It may be that you will need to predict several timepoints into the future at any given time. Here are some examples why this may be required:

• There are objects with different end-to-end delays. For instance, a hand tracked with a camera may be have different latency than a head tracker, but both need to be drawn in sync in the same scene, so predictive tracking with different 'look ahead' times will be used.
• In configurations where a single screen - such as a cell phone screen - is used to provide imagery to both eyes, it is often the case that the image for one eye appears with a delay of half a frame (e.g. half of 1/60 seconds, or approx 8 mSec) relative to the other eye. In this case, it is best to use predictive tracking that looks ahead 8 mSec more for that delayed half of the screen.

Common prediction algorithms

Here is some sampling of predictive tracking algorithms:

• Dead reckoning. This is a very simple algorithm: if the position and velocity (or angular position and angular velocity) is known at a given time, the predicted position assumes that the last know position and velocity are correct and the velocity remains the same. For instance, if the last known position is 100 units and the last known velocity is 10 units/sec, then the predicted position 10 mSec (0.01 seconds) into the future is 100 + 10 x 0.01 = 100.1. While this is very simple to compute, it assumes that the last position and velocity are accurate (e.g. not subject to any measurement noise) and that the velocity is constant. Both these assumptions are often incorrect.
• Kalman predictor. This is based on a popular Kalman filter that is used to reduce sensor noise in systems where there exists a mathematical model of the system's operation. See here for more detailed explanation of the Kalman filter.
• Alpha-beta-gamma. The ABG predictor is closely related to the Kalman predictor, but is less general and has simpler math, which we can explain here at a high level. ABG tries to continuously estimate both velocity and acceleration and use them in prediction. Because the estimates take into account actual data, they provide some measurement noise reduction. Configuring the parameters (alpha, beta and gamma) provide the ability to emphasize responsiveness as opposed to noise reduction. If you'd like to follow the math, here it goes:

Summary

Predictive tracking is a useful and commonly-used technique for reducing apparent latency. It offers simple or sophisticated implementations, requires some thought and analysis, but it is well worth it.

Positional tracking: "Outside-in" vs. "Inside-out"

Optical positional tracking for goggles uses a camera (or cameras) and a known set of markers to determine the position of the camera relative to the markers. Positional tracking can be done using the visible spectrum but is more commonly done using infra-red markers and a camera that is sensitive to IR light.

There are two main options:

• Inside-out tracking: where the camera is based on the goggles and the IR markers are placed in stationary locations (e.g. on the computer monitor, on the wall, etc.)
• Outside-in tracking: where the camera is placed in a stationary location and the IR markers are placed on the goggles.

In both cases, the targets sometimes flash in a way that is synchronized with the camera. This allows reducing power consumption for the targets and helps reduce tracking noise from IR sources that are not the targets.

Sensics dSight panoramic HMD with IR targets for "outside in" tracking

How do these approaches compare?

• Tracking volume: in both cases, at least some of the targets need to be visible to the camera. When the user rotates the head, an "inside-out" system needs targets that are physically far apart. If the targets are, for instance, placed on the bezel of a notebook PC, it is easy to see how head rotation could easily take these targets out of the field of view of the camera. A wider lens could be used in the camera, but this would reduce the tracking precision as each camera pixel would now cover a greater physical space in the world. In the "outside-in" system, targets could be placed on most sides of the goggle, allowing reasonably large rotation while still having targets visible on the camera. Advantage: outside-in
• Tracking inside an entire room: if we want to allow mobility within a room, an 'inside-out' system would require additional markers on the walls, whereas an 'outside-in' system would require additional cameras. Both systems would require room calibration to make sure the target and/or cameras are placed in a known position. Additional cameras require additional processing power. Slight advantage: inside-out
• Where is data being processed? In "inside-out' tracking, the camera data is either processed on the goggle or the camera is connected to a computer that is either carried by the user or stationary and connected via a wire. In 'outside-in' tracking, the data is processed on a computer that could be stationary. Advantage: outside-in
• Can this be used with a wireless goggle? If the goggle is not tethered to a computer, "inside-out" tracking requires that the data is either processed locally or that the camera signal is sent wirelessly to a base station. In contrast, an 'outside-in' approach does not require wireless data of the camera. At most, a synchronization signal can be sent to the IR LEDs to make sure they flash in sync with the camera. Advantage: outside-in
• Ability to combine with augmented reality system. Sometimes, the goggle will already have an on-board camera (or cameras) for the purpose of augmented reality and/or 3D reconstruction. In that case, using the same camera for positional tracking may have some cost advantages if positional tracking can be used with visible targets or if the camera already has IR sensitivity. Advantage: inside-out

Note: tracking accuracy is also an important comparison parameter, but this is more difficult to generically compare across both approaches. Very often, accurate tracking is achieved not just through the camera data but also by integrating ("sensor fusion") rotational data and linear acceleration data from on-board sensors. The sensors used and the quality of the sensor fusion algorithm would determine which approach is better.

Bottom line: for most applications, an 'outside-in' approach would be better, and thus we expect to see a greater number of 'outside-in' solutions on the market.

What has been your experience?


For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here

An overview of positional tracking technologies for VR

Photo Credit: 
Saad Faruque via Compfight

Positional tracking is very important towards achieving immersion and presence in virtual reality. Whether it is the head, arms, fingers or objects (such as a weapon), positional tracking can deliver multiple benefits:

• Change the viewpoint of the user to reflect actions such as jumping, ducking or leaning forward.
• Show hands and other objects in the displayed image. A common complaint of users in virtual reality is that they can't see their hands.
• Connect the physical and virtual world. For instance, by detecting hand position, a software program can implement the option to move virtual objects by touching them.
• Detect gestures. By analyzing position over time, a gesture can be detected. For instance, a user might draw the number "8" in air and have the software detect it.

There are several methods of tracking position and I felt it is worthwhile to describe some of them. This post focuses on tracking for virtual reality applications, so we will not look at vehicle tracking, tracking of firemen in buildings and so forth. In no particular order, here are some of the popular tracking methods include magnetic, inertial, optical and acoustic tracking as well as hybrid tracking that combines multiple methods.

I've described these tracking methods as well as others such as depth map in a guest blog post at RoadToVR. Please click here to read that post on the RoadToVR site.


For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here

Linear Acceleration and Motion Trackers, a follow-up

After the post What you should know about head trackers I was asked to comment about linear movement in motion trackers.

Since inertial motion trackers include an accelerometer, it can measure the magnitude and direction of linear acceleration. Most of those that took high-school physics remember that the velocity is the derivative of position and that acceleration is the derivative of velocity. If so, can we integrate (sum up) linear acceleration readings to obtain velocity and then integrate velocity to obtain the X/Y/Z position of the head?

Theoretically yes, but practically that is not possible because of the accumulation of error. Every linear acceleration reading is slightly inaccurate, whether because of sensor error or of insufficient precision in reporting the acceleration reading. Because we have inaccurate acceleration, we will get inaccurate velocity from it, and because we have inaccurate velocity, we will get inaccurate position. In real life, we will see two problems in position reading: drift and repeatability.

The drift problem will present itself as a changing reading of the position even if the motion tracker is perfectly still. The repeatability problem will manifest itself as a changed reading when returning to the same position. For instance, assuming you are standing with an HMD and the motion tracker is 1.8m (6 feet) off the ground. Now, crouch and stand up several times, returning to the same position. Most likely, after many times of doing this, the X/Y/Z reading of the motion tracker will be substantially different than when you started.

Does measuring linear acceleration in the motion tracker is of any use other than for sensor fusion? Of course. Linear acceleration can give you a good sense of what the user is doing in the short-term: jump, duck, juke left/right, lunge forward/back. All of these can be very useful for gaming or other interactive experience, as long as you are not assuming that linear acceleration gives you accurate position information.

Comment? Question? Correction? Write a comment and I will try to address it.

Have a VR-related topic you'd like me to discuss? Write a comment and I'll consider it for a future post.


For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here

What you should know about Head Trackers

In this post, I will cover useful information about motion trackers in the context of head-mounted displays: how they work, what features are important, and what you want to think about when integrating them.

It used to be that you'd have to pay $2000 or more for a accurate head tracking, but these days good trackers come included inside many professional HMDs such as the Sensics zSight and even in some of the recent consumer goggle entries.

Head tracker basics

All head trackers measure rotational orientation - yaw, pitch and roll.

Head Yaw, Pitch and Roll (from www.resourceonbalance.com)

Yaw is the side to side movement as in looking left and right. Pitch would be the motion when you look up or down. And roll is tilting the head side to side.

This post deals with inertial head-trackers as opposed to head-tracking systems that use markers, cameras, ultrasound or other methods. Inertial head trackers are the most popular type of trackers in HMDs 

Most inertial head trackers include three sensors: accelerometer, gyroscope and magnetometer. The accelerometer measures the linear acceleration along three axes, including the earth's gravitational force, thus providing information regarding both the magnitude and direction of acceleration. The gyroscope measures angular velocity. The magnetometer measures the strength and direction of a magnetic field, most notably the earth's magnetic field. When all three sensors are used, and each provides measurement in three directions, the resulting head tracker is typically referred to as a '9-axis' or '9-sensor' module. Vendors sometimes refer to this as a '9 degree of freedom' module but this is often confusing and inaccurate.

What role do the vendors of head tracker modules play?

xSight HMD with IntertialLabs head tracker

Companies that make head trackers - Intersense, IntertialLabs, Hillcrest Labs just to name a few - don't make the sensors themselves. The accelerometers, gyroscopes and magnetometers are from other vendors such as Panasonic, Robert Bosch, InvenSense, Kionix, Seiko Epson, STMicroelectronics, Honeywell and Analog Devices. What the head tracker companies do with these sensors are several things:

• Sensor Fusion. This is arguable the most important function, involving combining the data from the various sensors into output that is more accurate than each of the sensors. For example, the accelerometer senses gravity and thus is very helpful in determining what the vertical axis (up/down) is. However, if the module is in motion, other non-gravity accelerations can cause error in the axis determination. The module can query the gyroscope to see if there is angular velocity and determine the direction of gravity only when the module is not rotating. Similarly, the magnetometer can help calibrate the reading of the gyroscope and align yaw readings with the magnetic north. Module vendors often have their own "sensor fusion" algorithms and that is a key part of their intellectual property.
• Calibration. The individual sensors might not be linear or might output a 'non-zero' reading when they are in complete rest. Calibration parameters can be different from one sensor vendor to another and between two sensors from the same vendor. Calibration parameters can also be temperature-dependent so calibration ends up being an important part of combining sensors into a head tracking module.
• Data smoothing and filtering. Sensor output is noisy. If the heading from a motion tracker module is used to change the viewpoint in a virtual world, sensor noise gets translated into jitter or other undesired image artifacts. Data smoothing could be as simple as averaging samples over a sliding window or as sophisticated as Kalman Filter algorithms. Data smoothing can also be contextual - when the sensor is approaching rest, one kind of data smoothing may be used whereas when the sensor is moving quickly, different data smoothing might be applicable. More on data smoothing below as we discuss performance.
• Predictive tracking. In an effort to reduce the apparent response time of a tracker (more on that below), some trackers including algorithms that try to predict the orientation of the head a short time into the future. For instance, the algorithm could simplistically say "I see that the head rotated left at a constant rate of 10 degrees/sec for the last couple of seconds, so I could estimate where the head will be 20-30 mSec into the future".
• Format translation. Orientation data can be consumed in various formats by the application: Euler angles, Quaternions, Rotational Matrices and more. An on-board processor on the module often provides the host computer with this data in the most convenient format. Aside from the actual content data, modules can offer the data in various physical layer and low-level protocols such as USB, RS-232/RS-485, SPI and more.
• Additional processing. Some modules go as far as including an on-board gesture engine that can analyze movements and report gestures when recognized. Those modules that perform this kind of additional processing on-board help reduce computational load on the host computers.

Key performance parameters

There are several key performance parameters that differentiate between the various tracker modules.

• Refresh rate. The rate - number of times of second - that a tracker can report new orientation data has come into the spotlight in the last year with the increased focus on virtual reality gaming. The quicker the game is able to sense the motion of the user, the lower the delay will be between motion and screen updates. However, one must pay attention to what data is received in each cycle - is it raw or smoothed/filtered. The sensors on a tracker module can report raw data very quickly, but if the tracker module does not provide smoothing and filtering, the host computer will have to do that. For instance, a tracker module might provide 1000 updates per second of raw data, but 250 updates per second of processed, smoothed data. The choice of the physical interface between the tracker module and the host can impact the refresh rate. For instance, using an RS-232-based protocol would likely limit the number of messages that can be received per second as a result of the baud rate and message length.
• Resolution. A low resolution tracker module would create a noticeable distraction in the visual experience, especially when moving the head in a low speed. As an example, if a tracker only reported yaw readings in 1 degree resolution, rotating the head slowly in a head-tracked environment would result in a 'ratchet' effect where the image stays static in spite of head movement and then makes a large, noticeable jump once the tracker sensed a different position. This is particularly noticeable in images that simulate narrow field of view such as when looking through a rifle sight. A good tracker would provide resolutions that are better than 0.1 degrees in all directions.
• Accuracy. Resolution aside, how much is the tracker reading correlated with actual yaw/pitch/roll? Accuracy matters when trying to match tracker orientation with a real-world object. For instance, an completely inaccurate yaw heading would make navigation practically impossible. Another example might be in augmented reality applications where the application needs to place a graphical overlay on a particular point in real space. Accuracy also matters when using two or more orientation orientation trackers in the same application. For instance, if a soldier is wearing a tracked HMD and is  aiming a simulated weapon that is also tracked, matching the orientation of the HMD with the weapon is very important for training effectiveness.
• Repeatability. This shows whether returning to the same orientation heading in the physical world also equates to obtaining the same readings from the tracker module.

Compass Interference

• Magnetic field compensation. Have you ever seen the compass interference sign on an iPhone navigation app? This happens because other metal objects in the area cause confusion with the built-in compass/magnetometer, resulting in unstable heading reading. In an HMD context, this could be a problem in certain usage scenarios. I remember doing a demo at USC a few years ago and we set up the demo right underneath a large metal sculpture hanging from the ceiling. It took us awhile to understand why tracking was completely off! While in our case we could move away from the sculpture, a soldier that uses an HMD to train inside a tank cannot move away from the tank. In this case, some vendors can ignore the magnetometer, which precludes the tracker module from understanding where 'true north' is but provides a reasonably stable but slowly drifting reading. Other more sophisticated solutions include manual or automatic algorithms to sense and compensate for the surrounding metal objects.

Trackers modules for head tracking vs. tracker modules for other uses

Orientation trackers can be used for many applications that are not HMDs: remote control devices, smart phones and more. Even though the underlying technology is the same, HMD applications often need a tweaking of tracker parameters such as smoothing. Several years ago, my company started using a tracker that was originally made for a 3D pointer. When such a pointer is placed on a desk, it is not being used for pointing and the original firmware used this state to reset various tracker parameters and turn the tracker off to save energy. However, when an HMD moving very slowly, it may be that the user is trying to focus on a particular feature in the virtual world. Resetting tracker parameters - thus causing yaw/pitch/roll correction - is the wrong thing to do. We worked with the tracker vendor to correct this in a special firmware build, but the lesson is that one needs to investigate the tracker performance settings and make sure that they are optimized for head-mounted use.

UPDATE: see also our follow-on post Linear acceleration and motion trackers

Comment? Question? Correction? Write a comment and I will try to address it.

Have a VR-related topic you'd like me to discuss? Write a comment and I'll consider it for a future post.

For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here