Tuesday, September 12, 2017

Mozart meets Virtual Reality

I'm a VR professional. I'm also an amateur violinist. Not too shabby, and getting better with practice.
Once a year, I take a week off to play with the wonderful musicians of the Baltimore Symphony Orchestra. Alongside the pros and other amateurs, we practice and perform great classical music works.

That week got me thinking about VR can help various aspects of the performing arts.
One key area is audience engagement. The performance feels much different on stage than off it. VR can put the audience in places that money can't buy. Just in front of the conductor. In the middle of the violin section. At the back of the stage where the percussion players are. The audience can experience the excitement of music making from within.

Indeed, several major orchestras are already experimenting with VR. The Los Angeles Philharmonic recording Beethoven's Fifth Symphony in 360-degrees. That recording is free to download.  The Philharmonia Orchestra of London has made similar recordings.  The Baltimore Symphony Orchestra shared an open rehearsal.

Critics are applauding this inside view into the music. Audiences are getting a unique immersive experience. Most major orchestras as struggling with balancing their budgets. Engaging young audiences with a VR experience can sell tickets and attract new followers. One day, one could imagine a completely virtual experience. A music lover in Iowa could attend a Berlin Philharmonic concert without international travel.

After all, Movies evolved from just filming a stage play to many cameras with movement. Why should attending a concert stay the same for hundreds of years?

Another area where VR can be useful is performance anxiety. Musicians get nervous in performances, just like some grade school students. If a musician cannot perform on stage at the same level that she performed in a rehearsal, that is a problem.

There are many techniques to battle performance anxiety. Books such as "The inner game of tennis" help overcome self-doubt and nervousness. Presenters like Noa Kageyama of "The Bulletproof Musician" teach other methods. Some musicians medicate themselves before a high-stakes performance. Virtual reality is already used today to help overcome fear of public speaking. It is easy to envision extending this to performance anxiety. Just like VR can place you on stage in a large conference, it can place you on the virtual Carnegie Hall stage.

One critical performer that gets the least practice time is the orchestra's conductor. This is particularly true for young conductors. Without a permanent position with an orchestra, "podium time" is scarce. Conductors end up conducting their CD players or TV sets in preparation for a real orchestra. Imagine using a VR headset with a hand gesture sensor for conducting practice. It could be the next best alternative to the real experience.

This might not be the most popular use of VR, but certainly one that I can't wait to try myself.

Wednesday, September 6, 2017

VRguy podcast episode: Kevin Williams discussing trends in out-of-home VR

Kevin Williams of KWP consulting and I spoke about out-of-home VR, including:

  • Longevity of VR cafe model
  • Single-experience vs configurable-experience sites
  • How much money would people pay for VR experience
  • and more

you can also compare this to what he said in late 2015

Monday, August 28, 2017

VRguy Podcast Episode: Jason Jerald, Principal Consultant at NextGen Interactions

Jason and I talk about VR sickness, precision input, and design tradeoffs in VR interactions. We have a particularly interesting discussion about pen input in VR

Listen to the Podcast or read the transcript at this link 

Friday, August 25, 2017

Four myths are blocking real, needed VR standards

Neil Trevett, VP Developer Ecosystem at NVIDIA and President of the Khronos Group collaborated with me on a VentureBeat article discussing the four myths blocking real, needed VR standards.

The four myths we discuss are:
  1. It’s too early for standards.
  2. Standards stifle innovation.
  3. Consumers won’t be impacted if standards are not enacted.
  4. There are too many cooks developing standards.
You can read the full article here.

Monday, August 21, 2017

Sunday, June 11, 2017

How does eye tracking work?

Eye tracking could become a standard peripheral in VR/AR headsets. Tracking gaze direction can deliver many benefits. Foveated rendering, for instance, optimizes GPU resources by using eye tracking data. Higher-resolution images at shown at the central vision area and lower-resolution outside it. Understanding gaze direction can lead to more natural interaction. Additionally, People with certain disabilities can use their eyes instead of their hands. Eye tracking can detect concussions in athletes and can even help people see better. Eye tracking can help advertisers understand what interests customers.

Eye tracking is complex. Scientists and vendors have spent many year perfecting algorithms and techniques.

But how does it work? Let's look at a high-level overview.

Most eye tracking systems use a camera pointing at the eye and infrared (IR) light. IR illuminates the eye and a camera sensitive to IR analyzes the reflections. The wavelength of the light is often 850 nanometers. It is just outside the visible spectrum of 390 to 700 nanometers. The eye can't detect the illumination but the camera can.

We see the world when our retinal detects light entering through the pupil. IR light also enters the eye through this pupil. Outside the pupil area, light does not enter the eye. Instead, it reflects back towards the camera. Thus, the camera sees the pupil as a dark area - no reflection - whereas the rest of the eye is brighter. This is "dark pupil eye tracking". If the IR light source is near the optical axis, it can reflect from the back of the eye. In this case, the pupil appears bright. This is "bright pupil eye tracking". It is like the "red eye" effect when using flash photography. Whether we use dark or bright pupil, the key point is that the pupil looks different than the rest of the eye.

The image captured by the camera is then processed to determine the location of the pupil. This allows estimating the direction of gaze from the observed eye. Processing is sometimes done on a PC, phone or other connected processor. Other vendors developed special-purpose chips that offload the processing from the main CPU. If eye tracking cameras observe both eyes, one can combine the gaze readings from both eyes. This allows estimating of the fixation point of the user in real or virtual 3D space.

There are other eye tracking approaches that are less popular. For instance, some have tried to detect movements of the eye muscles. This method provides high-speed data but is less accurate than camera-based tracking.

How often should we calculate the gaze direction? The eyes have several types of movements. Saccadic movements are fast and happen when we need to shift gaze from one area to another. Vergence shifts are small movements the help in depth perception. They aim to get the image of an object to appear on corresponding spots on both retinas. Smooth pursuit is how we move when we track a moving object. To track saccadic movements, one needs to track the eye hundreds of time per second. But, saccadic movements do not provide gaze direction. Thus, they are interesting to research applications but not to mass-market eye tracking. Vergence and smooth pursuit movements are slower. Tens of samples per second are often enough. Since Many VR applications want to have the freshest data, there is a trend to track the eyes at the VR frame rate.

Eye tracking systems need to compensate for movements of the camera relative to the eye. For instance, a head-mounted display can slide and shift relative to the eyes. One popular technique is to use reflections of the light source from the cornea. These reflections are called Purkinje reflections. They change little during eye rotation and can serve as an anchor for the algorithm. Other algorithms try to identify the corners of the eye as an anchor point.

There are other variables that an algorithm needs to compensate for. The eye is not a perfect sphere. Some people have bulging eyes and others have inset eyes. The location of the eye relative to the camera is not constant between users. These and other variables are often addressed during a calibration procedure. Simple calibration presents a cross on the screen at a known location and asks the user to fixate on it. By repeating this for a few locations, the algorithm calibrates the tracker to a user.

Beyond the algorithm, the optical system of the tracker presents extra challenges. It aims to be lightweight. It tries to avoid needs constraints on the optics used to present the actual VR/AR image to the user. It needs to work with a wide range of facial structures. For a discussion on optical configurations for eye tracking, please see here.

Eye trackers used to be expensive. This was not the result of expensive components, but rather of a limited market. When only researchers bought eye trackers, companies charged more to cover their R&D expenses. As eye trackers move into mainstream, eye trackers will become inexpensive.

Monday, May 22, 2017

Understanding Relative Illumination

Relative illumination in the context of optical design is the phenomena of image roll-off (e.g. reduction) towards the edge of an eyepiece. This manifests in an image that is brighter at the center of eyepiece relative to the edge of the eyepiece.

Relative illumination is usually shown in a graph such as the one below

This particular graph is from an eyepiece with 60-degree horizontal field of view designed by Sensics. The graph shows how the illumination changes from the center of the lens, e.g. 0, to the edge of the lens, e.g. 30 degrees. The Y axis shows the relative illumination where the center illumination is defined as "1". In this particular eyepiece, the illumination at the edge is just over 70% of the illumination at the center.

This effect can also be viewed in simulations. The first image below shows a simulated image through this eyepiece when ignoring the impact of relative illumination:

Simulated image while ignoring the effect of relative illumination

The second image shows the impact of relative illumination which can be seen at the edges

Simulated image with relative illumination
Relative illumination is perfectly normal and to be expected. It exists in practically every eyepiece and every sensor. It is often the result of vignetting - some light rays coming from the display through the eyepiece to the eye that are blocked by some mechanical feature of the eyepiece. This can be an internal mechanical structure or simply the edge of a particular lens. Light rays from the edge of the display are easier to block and thus typically suffer more vignetting.

When we look at an optical design, we look to see that the relative illumination graph is monotonic, e.g. always decreasing. A non-monotonic curve (e.g. a sudden increase followed by a decrease) would manifest itself as a bright ring in the image, and this is usually not desired.