VRguy podcast: Dr. Daniel Laby discusses Sports and Performance Vision

My guest today is Dr. Daniel Laby, of Sports Vision. Dr Laby began his work in sports vision more than two decades ago with the MLB’s Los Angeles Dodgers. He has also been responsible for the visual performance of the New York Mets and St Louis Cardinals, and currently works with the Boston Red Sox, Cleveland Indians, Houston Astros, Tampa Bay Rays and Chicago Cubs. Dr. Laby spent three seasons working with the NBA’s Boston Celtics as well as the NHL’s Los Angeles Kings. He also worked with the US Olympic team prior to the Beijing Olympic Games in 2008 and attended the games with the team.

Dr Laby has been fortunate to have contributed to an MLB American League Championship team as well as 4 World Series Championship teams.

Listen in here or subscribe on iTunes.

VRGuy podcast episode: Thomas Wagner on VR Roller Coasters

Thomas Wagner is the CEO of VR Coaster. VR Coaster’s technology helps park operators convert traditional roller coasters into VR Coasters. Thomas and I discuss lessons learned from dozens of VR rides installed around the world.

Listen to the podcast or read the transcript here

VR's Dirty Secret

I traveled a lot in the past few weeks. All the hotels I stayed in had something in common: they had clean towels and fresh bed sheets.

But what if they didn’t? If the towels were dirty or the sheets looked like they were slept in, I would had complained.

But what if the hotel said: you are right: these are the old sheets, but we vacuumed the bed; these are the towels that the previous guest used, but we sprayed some Fabreze on them. Would I be happy? Of course not.

How about VR? Using a VR headset in an arcade after a stranger sweated in it for 30 minutes does not sound so appealing either. I’m not particularly interested in sampling the sweat, makeup, lice or whatever else the previous user had.

Hygiene in VR, and particularly in public installations, is an important issue that needs to be addressed. To me, that is VR’s dirty secret.

Mozart meets Virtual Reality

I'm a VR professional. I'm also an amateur violinist. Not too shabby, and getting better with practice.
Once a year, I take a week off to play with the wonderful musicians of the Baltimore Symphony Orchestra. Alongside the pros and other amateurs, we practice and perform great classical music works.

That week got me thinking about VR can help various aspects of the performing arts.
One key area is audience engagement. The performance feels much different on stage than off it. VR can put the audience in places that money can't buy. Just in front of the conductor. In the middle of the violin section. At the back of the stage where the percussion players are. The audience can experience the excitement of music making from within.

Indeed, several major orchestras are already experimenting with VR. The Los Angeles Philharmonic recording Beethoven's Fifth Symphony in 360-degrees. That recording is free to download.  The Philharmonia Orchestra of London has made similar recordings.  The Baltimore Symphony Orchestra shared an open rehearsal.

Critics are applauding this inside view into the music. Audiences are getting a unique immersive experience. Most major orchestras as struggling with balancing their budgets. Engaging young audiences with a VR experience can sell tickets and attract new followers. One day, one could imagine a completely virtual experience. A music lover in Iowa could attend a Berlin Philharmonic concert without international travel.

After all, Movies evolved from just filming a stage play to many cameras with movement. Why should attending a concert stay the same for hundreds of years?

Another area where VR can be useful is performance anxiety. Musicians get nervous in performances, just like some grade school students. If a musician cannot perform on stage at the same level that she performed in a rehearsal, that is a problem.

There are many techniques to battle performance anxiety. Books such as "The inner game of tennis" help overcome self-doubt and nervousness. Presenters like Noa Kageyama of "The Bulletproof Musician" teach other methods. Some musicians medicate themselves before a high-stakes performance. Virtual reality is already used today to help overcome fear of public speaking. It is easy to envision extending this to performance anxiety. Just like VR can place you on stage in a large conference, it can place you on the virtual Carnegie Hall stage.

One critical performer that gets the least practice time is the orchestra's conductor. This is particularly true for young conductors. Without a permanent position with an orchestra, "podium time" is scarce. Conductors end up conducting their CD players or TV sets in preparation for a real orchestra. Imagine using a VR headset with a hand gesture sensor for conducting practice. It could be the next best alternative to the real experience.

This might not be the most popular use of VR, but certainly one that I can't wait to try myself.

VRguy podcast episode: Kevin Williams discussing trends in out-of-home VR

Kevin Williams of KWP consulting and I spoke about out-of-home VR, including:

• Longevity of VR cafe model
• Single-experience vs configurable-experience sites
• How much money would people pay for VR experience
• and more

Listen to the Podcast or read the transcript here

you can also compare this to what he said in late 2015

VRguy Podcast Episode: Jason Jerald, Principal Consultant at NextGen Interactions

Jason and I talk about VR sickness, precision input, and design tradeoffs in VR interactions. We have a particularly interesting discussion about pen input in VR

Listen to the Podcast or read the transcript at this link 

Four myths are blocking real, needed VR standards

Neil Trevett, VP Developer Ecosystem at NVIDIA and President of the Khronos Group collaborated with me on a VentureBeat article discussing the four myths blocking real, needed VR standards.

The four myths we discuss are:

1. It’s too early for standards.
2. Standards stifle innovation.
3. Consumers won’t be impacted if standards are not enacted.
4. There are too many cooks developing standards.

You can read the full article here.

New VRguy podcast episode: Nick Whiting, technical director of AR/VR/XR at Epic Games

Listen to my conversation with Nick or read the transcript here.

We talk about Robo Recall, the unique challenges of VR and AR for game engines, what Epic learned from enterprise customers and much more.

Other recent interesting episodes:

Neil Trevett, VP at NVIDIA and President of the Khronos Group

Wanda Meloni, CEO and Principal Analyst at M2 Advisory

How does eye tracking work?

Eye tracking could become a standard peripheral in VR/AR headsets. Tracking gaze direction can deliver many benefits. Foveated rendering, for instance, optimizes GPU resources by using eye tracking data. Higher-resolution images at shown at the central vision area and lower-resolution outside it. Understanding gaze direction can lead to more natural interaction. Additionally, People with certain disabilities can use their eyes instead of their hands. Eye tracking can detect concussions in athletes and can even help people see better. Eye tracking can help advertisers understand what interests customers.

Eye tracking is complex. Scientists and vendors have spent many year perfecting algorithms and techniques.

But how does it work? Let's look at a high-level overview.

Most eye tracking systems use a camera pointing at the eye and infrared (IR) light. IR illuminates the eye and a camera sensitive to IR analyzes the reflections. The wavelength of the light is often 850 nanometers. It is just outside the visible spectrum of 390 to 700 nanometers. The eye can't detect the illumination but the camera can.

We see the world when our retinal detects light entering through the pupil. IR light also enters the eye through this pupil. Outside the pupil area, light does not enter the eye. Instead, it reflects back towards the camera. Thus, the camera sees the pupil as a dark area - no reflection - whereas the rest of the eye is brighter. This is "dark pupil eye tracking". If the IR light source is near the optical axis, it can reflect from the back of the eye. In this case, the pupil appears bright. This is "bright pupil eye tracking". It is like the "red eye" effect when using flash photography. Whether we use dark or bright pupil, the key point is that the pupil looks different than the rest of the eye.

The image captured by the camera is then processed to determine the location of the pupil. This allows estimating the direction of gaze from the observed eye. Processing is sometimes done on a PC, phone or other connected processor. Other vendors developed special-purpose chips that offload the processing from the main CPU. If eye tracking cameras observe both eyes, one can combine the gaze readings from both eyes. This allows estimating of the fixation point of the user in real or virtual 3D space.

There are other eye tracking approaches that are less popular. For instance, some have tried to detect movements of the eye muscles. This method provides high-speed data but is less accurate than camera-based tracking.

How often should we calculate the gaze direction? The eyes have several types of movements. Saccadic movements are fast and happen when we need to shift gaze from one area to another. Vergence shifts are small movements the help in depth perception. They aim to get the image of an object to appear on corresponding spots on both retinas. Smooth pursuit is how we move when we track a moving object. To track saccadic movements, one needs to track the eye hundreds of time per second. But, saccadic movements do not provide gaze direction. Thus, they are interesting to research applications but not to mass-market eye tracking. Vergence and smooth pursuit movements are slower. Tens of samples per second are often enough. Since Many VR applications want to have the freshest data, there is a trend to track the eyes at the VR frame rate.

Eye tracking systems need to compensate for movements of the camera relative to the eye. For instance, a head-mounted display can slide and shift relative to the eyes. One popular technique is to use reflections of the light source from the cornea. These reflections are called Purkinje reflections. They change little during eye rotation and can serve as an anchor for the algorithm. Other algorithms try to identify the corners of the eye as an anchor point.

There are other variables that an algorithm needs to compensate for. The eye is not a perfect sphere. Some people have bulging eyes and others have inset eyes. The location of the eye relative to the camera is not constant between users. These and other variables are often addressed during a calibration procedure. Simple calibration presents a cross on the screen at a known location and asks the user to fixate on it. By repeating this for a few locations, the algorithm calibrates the tracker to a user.

Beyond the algorithm, the optical system of the tracker presents extra challenges. It aims to be lightweight. It tries to avoid needs constraints on the optics used to present the actual VR/AR image to the user. It needs to work with a wide range of facial structures. For a discussion on optical configurations for eye tracking, please see here.

Eye trackers used to be expensive. This was not the result of expensive components, but rather of a limited market. When only researchers bought eye trackers, companies charged more to cover their R&D expenses. As eye trackers move into mainstream, eye trackers will become inexpensive.

Understanding Relative Illumination

Relative illumination in the context of optical design is the phenomena of image roll-off (e.g. reduction) towards the edge of an eyepiece. This manifests in an image that is brighter at the center of eyepiece relative to the edge of the eyepiece.

Relative illumination is usually shown in a graph such as the one below

This particular graph is from an eyepiece with 60-degree horizontal field of view designed by Sensics. The graph shows how the illumination changes from the center of the lens, e.g. 0, to the edge of the lens, e.g. 30 degrees. The Y axis shows the relative illumination where the center illumination is defined as "1". In this particular eyepiece, the illumination at the edge is just over 70% of the illumination at the center.

This effect can also be viewed in simulations. The first image below shows a simulated image through this eyepiece when ignoring the impact of relative illumination:

Simulated image while ignoring the effect of relative illumination

The second image shows the impact of relative illumination which can be seen at the edges

Simulated image with relative illumination

Relative illumination is perfectly normal and to be expected. It exists in practically every eyepiece and every sensor. It is often the result of vignetting - some light rays coming from the display through the eyepiece to the eye that are blocked by some mechanical feature of the eyepiece. This can be an internal mechanical structure or simply the edge of a particular lens. Light rays from the edge of the display are easier to block and thus typically suffer more vignetting.

When we look at an optical design, we look to see that the relative illumination graph is monotonic, e.g. always decreasing. A non-monotonic curve (e.g. a sudden increase followed by a decrease) would manifest itself as a bright ring in the image, and this is usually not desired.

A Visit to the IMAX VR Center

A few weeks ago, I was in Los Angeles and decided to visit the newly-opened IMAX VR center. I went there as a regular paying customer - not some "behind the scene tour" - to see and learn. I've experienced Zero Latency, The Void and many others, so could not resist trying IMAX.

The lobby of the attraction is reminiscent of a small movie theater lobby. Vertically-oriented monitors on the walls announce the available VR experiences. A small reception area sells $10 tickets for the attractions. A display shows the available time slots for each 10-minute experience. After purchasing the tickets, a friend and I were asked to wait for our scheduled time. When the time came, an attendant escorted us to the VR area.

If I remember correctly, there were eight available experiences. Seven of them were based on the HTC VIVE. One - the John Wick Chronicles - was showing on Starbreeze headset. The HTC VIVE experiences did not appear to to be specially-made for this venue. For instance, one experience was Trials on Tatooine which can be freely downloaded from the Steam store. I think people come to movie theaters for an experience that they can't get at home. One would expect VR to be the same.

I have an HTC Vive at home (as well as many other headsets) at home. Using them is part of my job. However, most folks don't have easy access to PC-based VR equipment. For now, stock experiences might be just fine to get people exposed to VR.

Inside to the VR area, Each headset was in a space separated by low walls, a bit like an open space in an office. Headset cables were tied from the ceiling. HTC VIVE units had a leather face mask which is probably easier to clean. An operator administered each experience - one operator per headset. . Operators were friendly and enthusiastic about the VR equipment. I think their enthusiasm was contagious, which was nice.

Speaking of contagious, the operators told me that they wipe the face masks between users. Masks also get replaced every couple of weeks. I was told that visitors did not often complain about wearing a VR goggle that was used by many people before them.

I couldn't help but wonder about the economics. 15-minute timeslots: 10 minutes of usage plus some time to get people in and out of the experience. $40 an hour per station. One full-time operator per station. Now add rent, equipment, content fees, ticket sales, credit card fees, etc. Can you make money? Maybe making money is not the goal in this first location. Instead, the goal could be to have a "concept store" towards inclusion at the lobby of regular movie theaters.

Since I don't have a Starbreeze headset at home, I opted for the John Wick experience. It's a shooter game that encourages you to move in a space while holding a weapon. As expected, Virtual soldiers try to kill you. The headset was fairly light and the weapon comfortable. The experience was immersive though both the image and graphics quality could have been better. I can see why a person with little VR experience could enjoy these 10 minutes.

My friend did not have many VR experiences before this visit. He chose 'Trials on Tatooine" which he seemed to thoroughly like.

In all - a nice start to what can be the next big thing in entertainment.

Have you tried IMAX VR too? What did you think?

The Unique Requirements of Public VR

A good treadmill for home use costs around $1000. A treadmill for use in a health club could be ten times more expensive. Why would club owners agree to pay so much more? Because they understand that the home equipment would not withstand the heavy use in a club.

The same is true for VR goggles. Goggles for home use are not suitable for sustained use in arcades and amusement parks. Both VR vendors and attraction operators need to understand and address these differences.

Durability is one key issue. A goggle in an amusement park is subject to both accidental and intentional abuse. A kid might try to see if a lens can pop out. Someone else might wish to take a component as a souvenir.

Hygiene is also important. VR experiences can be intense, and parks can be in hot and humid areas. Whether it is sweat, suntan lotion or something else, a guest does not wish to wear a soaked or dirty goggle. This is true not only for the face mask, but for any built-in headphones.

Sweat and humidity might also cause fogging on the lenses. If a guest has to take off the goggles to defog them, the break in VR immersion degrades the experience.

Depending on the attraction, visitors can be of all ages. Good physical fit is important, regardless if the visitor is a small kit or a large man. Goggles must accommodate a wide range of head sizes, and eye separation. Eyeglasses also present a design challenge. Visitors prefer to keep them on, so goggle vendors try to make room for them.

Beyond the customer-facing experience, there are important operational considerations. Attractions make money by providing a unique experience to a large number of visitors. Every minute spent by a guest adjusting straps in a roller-coaster, is a minute lost. Some have reported a 30% decrease in guest throughput after upgrading a roller coaster to VR. A larger park crew assisting guests or maintaining headsets means larger operational costs.

There are several different types of public VR experiences, each with unique challenge. A free-roam experience (e.g. Zero Latency) needs to address backpack PCs and controllers. Themed experiences such as The Void have accessories that supplant the story. These accessories have many of the same challenges. A small attraction in a shopping mall cannot afford a large operating crew. A VR roller coaster might need a chin strap to keep the goggles on the head. If a VR roller coaster relies on standard phones, they might overheat or need to recharge often.

Often, the first instinct of those building VR attractions is to do everything. They might try to build their own goggles, create content, or even a tracking system. Over time, they focus on their core competencies, bringing external vendors for everything else.

The first generation of these solutions show the immense promise of public VR. VR Coaster, for instance, has deployed GearVR-based roller coaster experiences in over 20 parks. These use a special tracking system to determine the position of each car at any time. The Void and Zero Latency use backpack PCs to allow guests to explore unique spaces. Talon Simulations provides flight and driving simulations in malls. IMAX opened a VR center where guests can try a variety of 10-minute VR experiences. Most of these first-generation solutions use consumer-grade hardware. Operators realize that many problems still need solutions. At the same time, consumers learn what they like and dislike in current solutions.

HMD vendors are also rising to the challenge. ImmersiON-VRelia has developed phone-based goggles that feature a reinforced plastic construction. Sensics released goggles with detachable face mask to address both hygiene and operational efficiency.

What's missing? Low-latency wireless video solutions to get rid of cables. Faster ways to adjust goggles for guest. Better methods to clean goggles between guests. Phones the don't overheat. Multi-user experiences with a stronger social aspect. The passage of time to see what works and what doesn't. VR standards to help integrate new devices into compelling experiences.

I am excited what public VR experiences could be. My excitement comes both from a user standpoint - these experiences are fun! - but also from a problem-solver view - the problems are challenging.

Try these experiences next time you can. Going to a movie theater provides a different experience than watching at home. Going through a good public VR experience is beyond what VR at home provides.

Suffering, Art and VR Standards

Think about a great work of art: a classic book, a timeless painting, a symphonic masterpiece. What's common to many of these creations?

They were all the result of great suffering.

Tolstoy, Van Gogh, Mozart - they did not have easy lives. Many of the greats suffered from oppression, mental or physical illness, or hunger.


If you don't have drama in your life, how could you summon drama for your art?

People ask me "what made you want to work on VR standards?" My answer: it's the suffering.

No, not my personal suffering. I'm no Amaedus or have never considered cutting off my earlobe to express love.

But in many years of working with customers on their VR systems, I saw a lot of technical suffering:

The suffering of integrators that need to chase the latest API again and again. That don't know if the equipment they design for today will be available to buy in a year.

The suffering of device manufacturers that need just one more driver to support them.

The suffering of end-users that wonder if today's software will work on tomorrow's devices.

That's why we need efforts like OSVR or OpenXR to make it easy for everyone to work together. It wouldn't be as timeless or profound as "War and Peace", but it will help a lot of people.

Unholy Alliance

A few weeks ago, we were approached by a VR porn site seeking a partnership.

It is no secret that adult entertainment is an early adopter of many new technologies and virtual reality is no exception.

We respectfully declined as we decided a long time ago that we won't participate in this market.

Besides, I'm no longer as good looking as I used to be.

Virtual Reality Standards: too early or long overdue?

This article originally appeared on Mar 22nd at ReadWrite

You bought a new printer for the office. You unpack and connect it to your PC. You install its demonstration software and see the printer works well. Then, an unfortunate surprise: your word processor cannot work with this printer. You'll need to wait until the maker of the word processor releases a new version. When will this version be available? Should you return the printer? Should you change to a different word processor?

If this sounds like the 1980's, it's not too far from where VR is today. VR programs are often hard-coded to one set of hardware devices. Only a particular HMD, coupled with its tracking system and controller will work. Every device manufacturer has a different API. If you want to use this VR experience with a different set of hardware, you might not be in luck. At best, you'll need another version. Worst case, you just won't be able to do it. The problem of different vendors having different APIs is often called 'API fragmentation'

VR standards can help solve this fragmentation problem. In the PC world there are a few basic device types: keyboard, mouse, printer, scanner, and so forth. Likewise, basic VR device types include an HMD, tracker, controller, and a few others. A standard way for a program to interface with these VR devices would help solve this problem.

If software could work across different hardware combinations, almost everyone would benefit:

• Consumers could mix and match devices to their liking. I may have a Dell computer but I don't always want a Dell printer to go with it. Furthermore, consumers would be confident that their investments are future-proof. A 2017 game would likely work with 2018 or 2019 hardware. 
• Game publishers and other experience creators would have a larger addressable market. Today, they hard-code their game to a particular set of hardware devices. Tomorrow, they would support any device that has a conforming 'driver'. 
• Manufacturers that support a standard would have their devices work with lots of content. This would allow even small players to enter the market, and promote innovation. 

Some claim it is too early for a VR standard. VR is new, they say. Let a couple of years pass and then we will know what to standardize. Yet while consumer VR is new, VR has been in academia and industry for decades. Labs and factories deployed HMDs, head and eye trackers, and controllers for many years. Such devices were expensive and perhaps more clunky, but still performed the same functions. Software frameworks like UNC's VRPN (http://www.vrpn.org) provided device independent access for many years.

The resistance to a standard sometimes stems from the competitive strategy of a company. A vendor relying on a 'walled garden' approach often wishes to control the entire stack. The ability to swap out hardware, or use a different app store might be not what they had in mind.

In VR, there are often two standard interfaces that need to defined. The first is the device interface. This defines how to configure devices of particular type and how to extract data from them. Printers have different capabilities but share the same basic functions. The same is true for VR devices. The second standard interface is the application interface. It describes how an application or a game engine renders its content and get data. Inbetween the applications and devices there is often a middleware layer. That middleware is the software intermediary between applications and devices.

One effort that adapts this approach is OSVR. Started by Sensics and Razer, it is an open-source software platform for VR. OSVR implements both a device interface layer as well as an application layer. OSVR supports over 200 devices, and most of the OSVR code is free and open-source.

Another effort is OpenVR which is an open API (though not open source) from Valve. Building on the success of SteamVR, OpenVR allows HMDs to work with SteamVR content. There is some compatibility between these efforts. An OSVR plugin for OpenVR, allowing OSVr devices to work with SteamVR content.

In January, the Khronos group (known for OpenGL standards) launched a new VR initiative. The initiative, called OpenXR, brings together a wide range of companies. Industry leaders including Google, Oculus, Valve, Sensics and Samsung are part of this effort. OpenXR aims to combine lessons learned from building OSVR, OpenVR and proprietary APIs. It aims to create both a device interface as well as application interface. It is unclear how soon this effort will mature. Khronos standards take an average of 18 months. It is also unclear what capabilities will be part of the first standard. What is clear is that these companies felt enough pain to want to work on standards.

I am encouraged that so many participants are coming together to work on a standard. Other interested parties are also invited to contribute. Standards are sometimes boring, but they are important. They will make the consumer experience better and promote innovation.

This article originally appeared on Mar 22nd at ReadWrite