Volkswagen uses virtual reality in interactive exhibit at shopping malls

My company's customer, Maniak Experiencial, delivered an interactive exhibit for Volkswagen. In this exhibit, a futuristic car was placed in shopping malls, and treated visitors to an exciting driving simulator using the zSight HMD

Check out the many images and additional details here

"Resolutions" for the New Year

Happy New Year!

This post is not going to be about the "less coffee; more exercise" type of resolutions for the new year. Instead, let's discuss display resolutions and how they are shaping up for 2014.

A few years ago, Apple introduced the "Retina Display" as a marketing term. The thought was that the resolution of the display is so high that when held in the typical viewing distance, the pixel density is practically as high as can be distinguished by the naked eye, and thus increasing the resolution for that viewing distance won't generate any benefits.

Let's run through the numbers. 25cm (10 inches) is considered to be closest comfortable viewing distance for most people. The visual acuity of the eye - assuming 20/20 vision - is considered to be 1 arcmin/pixel or approximately 60 pixels per degree. At a distance of 25 cm, 1 degree takes up 25*tan(1 degree) or about 0.44 cm. Thus, if 60 pixels take up 0.44 cm, 1 inch would require 60*2.54/0.44 = 346 pixels/inch, sometimes referred to as 346 DPI (dots per inch) or PPI (pixels per inch) at this distance of 25 cm . Indeed, Apple has 326 DPI in the iPhone 4, so pretty close.

Some have argued that Apple's claim is misleading and that a display needs to have at 477 DPI to truly have eye-limiting resolution. So, when new 5" 1080p flat panel displays - such as those from Sharp - came with 433 DPI, that should have been nearly enough, right?

Now, we are hearing about 538 DPI flat-panel screens, 2560x1440 resolution, that are coming out in smartphone or small tablet form factor. Where are all these people that can tell apart 443 DPI but can't tell apart 538 DPI? Other than the "my display has more pixels than your display" claim, is there really a tangible benefit for phones to have higher and higher DPI?

If you make goggles that use smartphone displays, you love these higher display resolutions. Goggles magnify displays so when a 1080p display gets magnified to - for the sake of examples - 90 degrees wide, the pixel density is 1920/90=21.3 pixels/degree, so still far from eye-limiting. a 2560x1440 display magnified the same way will produce 28.4 pixels/degree, providing a tangible improvement. To get to eye-limiting resolution at 90 degrees, you'd need a screen that is physically small enough to be worn on the head yet has 5400 pixels across. Good luck finding the GPU that can drive interactive content for that display!

Several years ago, my company developed the xSight HMD which used a unique optical tiling system to combine multiple 800x600 OLED displays into one large virtual display that had about 1920x1080 pixels. Today, similar performance can be achieved without tiling.

Thus, as long as goggle makers can ride the smartphone wave, they can get better and better resolutions, but since even 2560x1440 sounds like somewhat of an overkill, what's next? Will the next-generation goggles continue to use flat panel displays or will they gravitate towards other technologies?

It should be a fascinating 2014. Happy New Year!

What I'm looking for at CES

If you like to experience sensory overload, go to CES. 3200 exhibitors, hundreds of large-screen TVs hanging off the ceiling, celebrity appearances, flashy cars, and of course Las Vegas outside the convention center.

Here's what I'm looking to see at CES:

• The latest and greatest in portable (man worn) display technologies. Would love to see which are ready for prime time within a Sensics product.
• All kinds of sensors: motion sensors, hand and finger sensors, biometric sensors, eye trackers, full body sensing, proximity sensors, embedded cameras for augmented reality. In short, anything that can be reasonably combined with a VR goggle to create the next-generation experience.
• Companies that have VR/AR ideas and concepts and unique specifications but need help in refining these specifications and then building high-performance and affordable products around them. In the last year, my company has done several such projects and have the designs, IP and decade of experience to help. High-end gaming goggle? Unique display or signal processing requirements? We can help.
• Last, catch up with friends and professional acquaintances. It's not often that so many of my network are in the same city at the same time.

Whether you are 'buying' or 'selling' in these categories, drop me a note and perhaps we can meet at the show.

Happy Holidays to all. Rest well and get ready for CES 2014!

The VR goggle as a Sensory and Computing platform

While there is still a lot of work to do on display technologies and optics to get the best possible image in front of the eyes, the real promise of virtual reality goggles is to serve as a portable sensory and computing platform.

Goggles are becoming a platform for several reasons:

• They are a physical platform. Once you securely position goggles on the head, you now have a physical base to attach additional sensors and peripherals: cameras, trackers, depth sensors and more.
• Portable computing is becoming evermore powerful, thereby creating an incentive to process and analyze sensory data on the goggles as opposed to transmitting large amount of information to some computing base. Furthermore, a key part of the value of goggles is their portability, so the ability to process 'locally' - on the goggle - contributes to the realization of this value.
• As goggles become increasingly immersive, the value of sensors increase as a way to tie the experience into physical objects in the immediate surroundings, as well as connect the actions and context of the user to what is happening 'inside' the display.

One could look at the following diagram - courtesy of Sensics - as a good illustration to what these sensors might be:

One could imaging several types of sensors as feeding into the goggle:

• Orientation and position sensors - whether for the head, limbs or objects such as a gaming sword
• Cameras that might provide visible, IR or depth map information
• Positional sensors such as GPS or indoor location sensors
• Eye tracking sensors to understand gaze direction and potentially provide user interface
• Biometric sensors such as heart rate, perspiration, blood pressure. An eye tracker can also provide pupil size which is another biometric input.
• and more

One would then need to turn this sensor data into useful information. For instance, turn position and orientation of various body parts into understanding of gestures; turn a depth map into detection of nearby obstacles; detect faces and markers from a video image.

As we have discussed before, a virtual reality abstraction layer is going to speed up this process as it will allow those that turn data into information to relieve themselves from the need to worry about the particular formats and features of individual sensors and focus on a category of sensors.

There are several places where this transformation could happen: on the goggle with the help of an embedded processor (as shown in the above diagram); Near the goggle on a device such as a tablet or powerful smartphone; At a stationary device such as a desktop PC or gaming console. Performing this transformation in or near the goggle allows running the application software in or near the goggle, leading to a truly portable solution.

What is the best place to do so? As Nelson Mandela - and others - have said "Where you stand depends on where you sit". If you are a PC vendor that is married to everything happening on the PC, you shudder at the notion of the goggle being an independent computing platform. If you are a goggle vendor, you might find that this architecture opens up additional opportunities that do not exist when you are physically and logically tied to a stationary platform. If you are a vendor making phones or tablets, this might allow you to position the phone as a next-generation portable gaming platform.

So, beyond innovations in displays and optics, I think we will see lots of interesting sensors and sensor fusion applications in 2014.

What do you think? Let me know.

Not just micro displays: adding flat panel products to the mix

After about a decade of making virtual reality goggles and other near-eye devices based on micro displays, my company started demonstrating and shipping goggles that are based on flat-panel displays such as those found in smartphones. Many that have visited the I/ITSEC training show in Orlando had a chance to experience some of our new offering, and we were very happy with the feedback received.

It was not a difficult decision. Given the increasing resolution, diverse supplier base and lower cost of flat panel displays (as opposed to OLED micro displays), it made sense to start applying our innovation and our expertise in goggles to this new display technology.

To date, we have exclusively used OLEDs from eMagin. There are many good folks at eMagin, and they have nice products, but given their well-documented delivery challenges and new designs made possible by flat-panel displays, they won't be our exclusive display supplier anymore.

Where and when is it best to use OLED micro displays?

• Where physical space is limited, such as when building a simulated rifle scope
• When the contrast and response time of OLEDs are a must (that is, until OLED flat panels become widely available for goggle use)
• Where harsh environmental conditions - especially temperature - are required, such as in our ruggedized HMD for training
• Where high pixel density is required
• Where being able to purchase replacement parts for many years is important
• When low power consumption is critical

Where and when is it best to use flat panel displays?

• Where wide field of view is particularly important
• When it is important to have supplier diversity
• When cost is a major factor

Not married to one display technology or another, it is now possible to choose the best one for each new product.

Do you make your own [...]?

Quite often, I get asked whether my company makes specific components - motion trackers, electronics, optics, etc. - that we use inside our virtual reality goggles.

As one would expect, we look at these 'make vs. buy' decisions individually, and ask several questions:

• Can we add value to our customers if we 'make'? Can we generate a product that is significantly better or lower-cost or offers some other unique benefits relative to the 'buy' alternative?
• How many of these do we expect to make? We'd be much more likely to buy when only small quantities are required and more inclined to make when there are more units.
• Can we afford it?
• Can we build it on time?
• Can we create value for our shareholders by generating valuable patent filings or know-how?
• Is this a discipline that we need to understand very well for our future business?
• Does a 'buy' option exist?

Historically, these have been our answers:

• Orientation trackers: we typically buy, but we then try to improve what we buy. We have worked with many of the leading orientation tracking vendors - Intersense, Intertial labs, Hillcrest, YEI - and have decided against developing our own. However, we have often worked with these manufacturers to introduce new features in their products or to optimize them for HMD use. We have added some of our own features such as predictive tracking when those did not exist. Last, we prefer to encapsulate the vendor-specific API with a standard Sensics interface because it allows our customers the benefit of maintaining their software investments when we change the motion tracker vendor inside our products.
• Optics. To date, we have always designed and made optics ourselves. We have made optics for small and large displays, using glass, plastic and fiber, using a variety of manufacturing technologies. We believe that the our portfolio of optical designs is an advantage, and that optics are a critical part of the goggle experience.
• Electronics. We often design our own electronics. Sometimes, we need special high-speed processing, and in other instances, we feel that we need something beyond simple driving of a display. This can be unique video processing, distortion correction or packaging that supports a particularly compact design.
• Displays. We buy. We don't have the know-how nor the capital to make our own displays and in the world of changing display technologies, we're glad not to be locked into a specific one. Having said that, we have worked with eMagin in prior years to modify the size of one of their OLED driver boards to make a system more compact and achieve better optical design. It was a financial investment, but we felt we added value to our customers.
• Mechanical design. We rarely design accessories such as helmet-mounts, but we do love to design goggle enclosures whether to give it our unique 'look', or to include innovative features such as hand tracking sensors.
• Software. We write our own (or pay to have it written). Our software is so deeply tied to the unique functionality of our designs that it is not available for off-the-shelf purchase.

If you are a manufacturer and would like to see how we can use some of our technologies to help you get new, innovative products to market on short order, drop me a line.

An Interview with Sebastien Kuntz, CEO of "I'm in VR"

Following my blog post "Where are the VR abstraction layers" I had an opportunity to speak with Sebastien Kuntz, CEO of "I'm in VR" a Paris-based company that is attempting to create such layers. I've known Sebastien for several years, since he was part of Virtools (now Dassaut Systemes) and it was good to catch up and get his up-to-date perspective.

Sebastien Kuntz, CEO of "I'm in VR"

Sebastien, thank you for speaking with me. For those that do not know you, please tell us who you are and what you do?

My name is Sebastien Kuntz and I am CEO of "i’m in VR". Our goal is to accelerate the democratization of virtual reality and towards that goal we created the MiddleVR middleware software product. I have about 12 years of experience virtual reality, starting at the French Railroad company working on immersive training, and continuing as the lead VR engineer in Virtools, which made 3D software. After Virtools was acquired by Dassault Systemes, I decided to start my own company and that is how "i’m in VR" was born.

What is the problem that you are trying to solve?

Creating VR applications is complex because you have to take care of a lot of things – tracking devices, stereoscopy, multiple computers synchronization (if you are working with a Cave), interactions with the virtual environment (VE).

This is even more complex when you want to deploy the same application on multiple VR systems - different HMDs, VR-Walls, Caves ... We want developers to focus on making great applications instead of working on low-level issues that are already solved.
MiddleVR helps you in two ways:

• It simplifies the creation of your VR applications with our integration in Unity. It manages the devices and cameras for you, and offers high-level interactions such as navigations, selection and manipulation of objects. Soon we will add easy-to-use immersive menus, more interactions and haptics feedback.
• It simplifies the deployement of your VR applications on multiple VR systems: the MiddleVR configuration tool helps you easily create a description of any VR system, from low-cost to high-end. You can then run your application and it will be dynamically reconfigured to work with your VR system without modification.

MiddleVR is an easy-to-use, modern and commercial equivalent of Cavelib and VR-Juggler.

How do you provide this portability of a VR application to different systems ?

MiddleVR provides several layers of abstraction.

• Device drivers: the developers don't have a direct access to native drivers, they have access to what we call "virtual devices", or proxy devices. The native drivers write tracker data like position and orientation directly in such a "virtual device". This means that we can change the native drivers at runtime while the application is still referencing the same "virtual device".  
• Display: all the cameras and viewports are created at runtime depending on the current configuration. This means your application is not dependent on a particular VR display.
• 3D nodes: Most of the time the developer does not care about the information from a tracker, he is more interested in the position of the user's head or hand for example. MiddleVR provides a configurable representation of the user, whose parts can be manipulated by tracking devices. For example the Oculus Rift orientation tracker can rotate the 3D node representing the user's head, while a Razer Hydra can move the user's hands. Then in your application you can simply ask "Where is the user's head ? Is his hand close to this object ?", which does not rely on any particular device. This also has the big advantage of putting the user back in the center of the application development!
• Interactions: At an even higher level, the choice of an interaction technique is highly dependent on the characteristics of the hardware. If you have a treadmill you will not navigate in a VE in the same way as if you only have a mouse, or a joystick, or if you want to use gestures... The choice of a navigation technique should be made at runtime based on the available hardware. In the same way, selecting and manipulating an object in a VE can be made very efficient if you use the right interaction techniques for your particular hardware. This is work in progress, but we would like to provide this kind of interactions abstraction. We are also working on offering immersive menus and GUIs based on HTML5.

Will this interaction layer also allow you to define gestures?

Yes, the interaction layer will certainly allow you to define and analyze gestures. Though this particular functionality is not yet implemented in the product, you will be able to integrate your own gestural interactions.

Do you extend Unity's capabilities specifically for VR systems ?

Yes we provide active stereoscopy, which Unity cannot do.
We also provide application synchronization on multiple computers, which is required for VR systems such as Caves. We synchronize the state of all input devices, Unity physics nodes, the display of new images (swap-buffer locking) and left/right eye images display in active stereo(genlocking). As mentioned, we will also offer our own way of creating immersive menus and GUIs because Unity’s GUI system has a hard time dealing with stereoscopy and multiple viewports. 

Do you support other engines other than Unity?

MiddleVR has been created to be generic, so technically it was designed to be integrated into multiple 3D engines, but have not done so yet. It’s the usual balance of time and resources. We made several promising prototypes though.

How far are we from true ‘plug and play’ with regards to displays, trackers and other peripherals?

We are not there yet. You need a lot more information to completely describe a VR system than most people think.

First, there is no standard way to detect exactly which goggle, tv, projector or tracker is plugged in. [Editor's note: EDID does provide some of that information]

Then in a display (HMD, 3D monitor, CAVE), it is not enough to describe resolution and field of view. You also need to understand the field of regard. With an HMD you can look in all directions. With a 3D monitor or most CAVEs, you are not going to be able to see an image if you look towards the back. The VR middleware needs to be aware of this and allow interaction methods that adapt to the field of regard. Moreover you have to know the physical size of the screens to compute the correct perspective. 

I believe we should describe a VR system not based on its technical characteristics such as display resolution or number of cameras for optical tracking, but rather in terms of what those characteristics means for the user in terms of immersion and interaction! For example:

• What is the end-to-end latency of the VR system for each application? This will directly influence the overall perception of the VE.
• What is the tracking volume and resolution in terms of position and orientation? This will directly influence how the user interacts with the VE: we will not interact the same with a Leap Motion which has a small tracking volume, or with the Oculus Rift tracker which can only report orientations or with 20 Vicon cameras able to track a whole room with both positions and orientations.
• What is the angular resolution of the display? If you can't read a text from a given distance, you will have to be able to get the text closer to you. If you can read the text because your VR system has a better angular resolution, you don't necessarily need this particular interaction.
• What is the field of regard ? As discussed above this also influences your possible actions.

The user's experience is based on its perceptions and actions, so we should only be talking about what is influencing those parameters. This requires a bit more work because they are highly dependent on the final integration of the VR system.

We are not aware of standards work done to create these ‘descriptors’ but we would certainly support such effort as it would benefit the industry and our customers.

Are most of your customers today what we would call ‘professional applications’ or are you seeing game companies interested in this as well?

Gaming in VR is certainly gaining momentum and we are very interested in working with game developers on creating this multi-device capability. We are already working with some early adopters.

We are working hard to follow this path. For instance, we are about to release a free edition of MiddleVR based on low-end VR devices and would like to provide a new commercial licence for this kind of developments. This is in our DNA, this is why we were born! We want to help the current VR democratisation. 

When you think about porting a game to VR, there are two steps (as you have mentioned in your blog): the first one is to adapt the application to 3D, to motion tracking ,etc. This is something you need to do regardless of the device you want to use.

The second is to adapt it to a particular device or set of devices. We can help with both, and especially with the 2nd step. There will be many more goggles coming to market in the next few months. Why just write for one particular VR system when you can write a game that will support all of them ?

What’s a good way to learn more about what MiddleVR can do for application developers? Any white paper or video that you recommend?

Our website has a lot of information. You can find a 5 minutes tutorial, and here another video demonstrating the capabilities of the Free edition.

Sebastien, thank you very much for speaking with me. I look forward to seeing more of I'm in VR in action.

Thank you, Yuval