Comprehensive hand and finger tracking? Try sensor fusion

A wide variety of applications could benefit from hand and finger tracking, but the performance requirements are quite diverse.

• For some applications, understanding the position of the hand as a point in space would be sufficient. Virtual boxing, for instance.
• For some, 360 degree tracking of the hands would be useful. A throwing motion of a football quarterback starts from behind the head. 
• Those applications needing finger tracking can often do with the fingers in front of the body, though if the finger sensor is mounted on the head, its field of vision might not be enough of the head is turned one way - say left - while the tracked hand goes right.

I think the solution will be some hybrid of the various technologies. Whether it is something like the Sixense STEM, YEI PrioVR, structured light technologies (like the one implemented in the Kinect), Time of Flight (inside the new Kinect), these technologies would need to be combined for a truly effective solution.

Progress in Hand and Body Tracking

I continue to believe that putting a display on the head, as great as that display might be, is not enough for a truly compelling virtual reality interaction, and that hand and finger tracking is a critical missing component for the VR experience.

Two new Kickstarter projects provide a step in the right direction, each taking a different approach:

Approach 1: PrioVR from YEI

PrioVR from YEI Technology has launched on Kickstarter earlier this month. It uses a connected network of high-quality orientation sensors to determine the orientation of various body parts. By logically connecting these sensors to a skeletal model of the body, the system can also determine the body posture and position of hands, feet, elbows, etc. This facilitates both motion capture as well as real-time reporting of body position and orientation into a game engine.

Motion Capture studio from YEI technology

One of the things that I like about the PrioVR system is that it is completely portable. It does not require a stationary sensor (e.g. Kinect), it does not require the person to be facing towards a particular direction and can really be taken anywhere, assuming you are willing to walk around with the sensors strapped to the body. The system does assume a wireless link between the central station on the body and a computer, but this works over fairly substantial distances. Additionally, if the computing device is portable, one could imagine a simple wired connection to it for enhanced portability.

The fidelity of the model is dependent on many parameters, including:

• The number of sensors that are being used. For instance, if a sensor is placed on the back, this sensor can be used to determine the rotation of the body and also help in determining the XYZ position of head (leaning forward will be registered in the system and through the skeletal model can be used to estimate the height of the head). However, if another sensor is placed on the lower back, the combination of these two sensors can be used to determine if the person has turned or is twisting the back.
• Calibration accuracy. In the YEI model, sensors are attached to the body using elastic straps. It is easy to see how a strap might be rotated so that, for instance, an arm sensor is not parallel to the ground even when the arms are. To avoid errors, a quick calibration might be required at the beginning of a session. 
• Accuracy of skeletal model. If the model assumes a certain distance from shoulder to elbow, but the actual distance is different that what is assumed, one could see how the hand position might be impacted by this skeletal error.

One wonders if this system does not produce 'too much information' relative to what is required. For instance, while it may be nice to understand if the arm is bent and exactly at what direction, is that information really required for a game that only cares about the hand position?

Approach 2: STEM from Sixense

The STEM system is scheduled to launch on Kickstarter later this month. It is an enhanced version of the current Razer Hydra in the sense that it adds wireless controllers as well as additional tracking points.

The STEM system uses a base station that helps track the XYZ position of various sensors/endpoints. A typical use case would be to track both hands when the user is holding a wireless controller as well as to track additional devices (head, weapon, lower body) if sensing modules are placed on it. To some extent, this is a simpler and more direct method than the PrioVR solution. With STEM, if you want to know the position of the hand, you just put a sensor on the hand. With PrioVR, if you want to know the position of the hand, you have to deduce it from the orientation of the various bones that make up the arm as well as knowledge about the upper and lower body. At the same time, it provides fewer data points about the exact posture and perhaps is more limited in the number of simultaneous sensors.

I have not had a chance yet to thoroughly evaluate the accuracy and response time of the STEM system yet.

Sixense STEM controller

Once the basic position and orientation data is presented to the application from either the PrioVR or STEM sensors, there is still an opportunity for a higher level of processing and analysis. For instance, additional software layers can determine gestures or hand signals. If more processing can be done in a middleware software layer, less processing will be required by the games and other applications to take advantage of these new sensors.

Another open question for me is the applicability to multi-person scenarios, assuming more than one 'instrumented' person in the same space. How many of these devices can be used in the same room without cross-interference.

Having said all that, I am excited by both these products. They are very welcome steps in the right direction towards enhancing and potentially revolutionizing the user experience in virtual reality.

Overcoming Optical Distortion

In a previous post, we discussed what optical distortion is and why it is important. In this post, we will discuss ways to correct or overcome distortion.

jenny downing via Compfight cc

There are four main options:

1. Do nothing and let users live with the distortion. In our experience, geometrical distortion of less than 5-7% is acceptable for mainstream professional applications. One of our competitors in the professional market made a nice living for several years by selling an HMD that had close to 15% distortion. However, some customers felt that it was good enough and that the HMD had other things going for it, such as low contrast and high power consumption. For gaming, it may be that larger distortion is also acceptable.

2. Improve the optical design. Lower distortion can certainly be a design goal. However, the TANSTAAFL principle holds ("There ain't no such thing as a free lunch", as popularized by Robert Heinlein) and to get lower distortion, you'd typically have to agree to relax other requirements such as weight, material selection, eye relief, number of elements, cost, transmissivity or others. Even for a standard eMagin SXGA display, my company has found that different customers seek different sets of requirements, which is why we offer two different OEM modules for this display, one design with lower weight and the other with lower distortion and generally higher performance.

3. Fix it in the GPU. The Graphics Processing Unit (GPU) of a modern graphics card or the GPU embedded inside many ARM chips are capable of performing geometry remapping to accommodate the geometrical distortion in the optics in a process called texture mapping. The upside of this approach is that it does not increase the direct system cost. The downside is that it requires modifying the program generating the content. If the content comes from a source that you have less control of (such as a camera, or a game that you previously purchased), you are unable to correct for the distortion.

4. Correct it in the goggle electronics. One could construct high-speed electronics (such as these) that perform real-time distortion correction for a known distortion function. This adds cost and complexity to the system but can work with any content regardless of where it came from. Done correctly, it need not add significant latency to the video signal.

Additional options, often application-dependent, also exist. For instance, we have several customers that use the goggles to present visual stimuli to their subjects. If the stimuli are simple such as a moving dot on the screen, the program generating them can take into account the distortion function while generating the stimuli, thus correcting for the distortion without help from the GPU



For additional VR tutorials on this blog, click here
Expert interviews and tutorials can also be found on the Sensics Insight page here.