zSight HMD unveiling archeology secrets as part of Operation Lune

From the Nautical Archeological Society

La Lune, the Sun King’s flagship, sank off the Toulon coastline in 1664. Almost 350 years later, a one-of-a-kind underwater archaeological expedition will unveil the secrets that this wreck conceals. A team of international experts embarked on this breathtaking venture armed with up-to-the-minute technology. Join the action, and a Sensics zSight HMD, and watch history meet the virtual realm.

Watch how the zSight HMD is used in the 3D exploration of this historical site.

Here come the sensors

Google project Tango is the latest example of increasingly-sophisticated sensors making their way to portable computing platforms such as phones, tablets and virtual reality goggles.

This mobile device includes several sensors:

• An orientation sensor, likely providing yaw/pitch/roll, angular velocity and linear rotation. These have been fairly standard in modern mobile devices.
• Cameras for both visible and IR spectrum. These are color cameras that aside from the usual RGB image also have pixels that sense near-IR wavelengths.
• A depth sensor, providing real-time mapping of the distance (e.g. depth) of various points in space - such as walls, people, hands - from the sensor. As an aside, there are several ways to sense depth: structured light such as the Kinect which projects a seemingly random pattern of dots into space and analyzes their reflections, time of flight such as SoftKinetic which measures the round-trip time it takes for light to return to the sensor and single- or multi-camera solutions that use image processing to estimate depth.

Technically, what is unique about the new platform is that it has dedicated power-efficient vision processors that allow it to continuously analyze, fuse and decode the information from the various sensors. This is news because previous processors consumed the mobile battery too fast for continuous use. But, the real reason to be excited about project Tango is that it provide both extra motivation as well as a hardware platform to many developers so that they can develop new 3D applications and increase awareness for the power of sensors.

I've written many times about the importance of sensors in goggles as a way to turn "dumb goggles" into "smart goggles", so I am a believer. It will be fun to see some the new applications that come out of project Tango, as well as how these types of sensors could be used in goggles in the future.

"Your hotel room is 5.6 meters diagonal" and other VR marketing nonsense

If you are seriously interested in learning about virtual reality technologies, you might want to skip this particular post. This post is about the little marketing inventions that VR vendors use, inventions that amuse and annoy me at the same time. As such, it should not be taken too seriously.

• "A diagonal field of view of 60 degrees". Since when did diagonal field of view become an important measure? I know that the diagonal field of view is larger than the horizontal or vertical field of view, which is probably why it is chosen, but I think humans can visualize much better horizontal or vertical fields view. For instance, if you are looking for a hotel room in the city and the hotel room says that it has a diagonal size of 5.6 meters, why is that useful? Would it be more useful to know that siad hotel room is 4 x 3 x 2.5 meters LxWxH? The diagonal field is a carry over from the television world, where you buy a 60" television, though the aspect ratio for televisions (width:height) is much more consistent than the aspect ratio for goggles. A 60" television with a 16:9 aspect ratio has a 52" horizontal and 30" vertical size (I looked it up here. If you need the largest number how about circumference? Your 60" TV just became 164" by that measure.

• "It is like watching a 70" television from 6 feet". This may be my favorite. In an effort to illustrate how wide the image, there is the (diagonal) TV analogy? 70" from 6 feet sounds a lot but it is actually a mere 52 degrees (I looked it up here). I would have just loved to be in that marketing meeting. 70" from 6 feet does not sound that impressive? Maybe we should write 140" from 12 feet (hint: the field of view is the same). I think the cake goes here to the vendor that likened their experience to a 750" screen - though I guess the intention was to visualize a movie theater experience. Google it to find out who.

• "We have dynamic resolution which is like the human eyes". Translation: our optics are so-so and the image outside the center is not really in focus and has quite a bit of distortion, but this is OK because humans see better in the center of the visual field relative to how they see in the peripheral vision. What happens when you turn your eye and the side of the image is now viewed by your central vision?

• "We have a 9-axis motion tracker". So, let's see: X, Y, Z are three, Yaw, Pitch and Roll are the next three, so what's 7, 8, 9? Time travel? I'd like to think this is often more of an honest mistake than an attempt at deception. Usually, this refers to a 9-sensor motion tracker that has a gyroscope, accelerometer and magnetometer and thus reports linear and angular acceleration as well as angular position. 

• "Our micro-display has 4 million pixels". Usually, this is triple-counting. A third of these pixels are red, a third are blue and a third are green. This statement usually means "we have a 1280x1024 pixel display but each pixel is full color and is made of 3 sub pixels"

• "Our goggle has 1080p resolution". This sounds a lot better than SXGA (1280x1024) resolution, but often 1080p resolution in a goggle could mean 1080p across both eyes, so 960x1080 per eye and thus fewer pixels than 1280x1024 per eye.

I was thinking for a while whether I should include "retina display" (as in "my phone as a retina display but yours does not") in this list, and decided against it. Retina display is a lovely way to trademark the benefit - high resolution that is similar to the eye's resolution - but other than scholarly discussions whether 'retina display' is indeed 'eye limiting', I don't see it as a misleading claim. It talks about the benefit much like Hertz Rental Car's GPS systems are called NeverLost to showcase what it does for you as opposed to how it does it.

One last note: my company is also guilty in some of the above sins - after all, we sell in the same market and cater to the same customers that have been trained to look for "diagonal field of view" or other not-so-important measures. 

What have I forgotten? Let me know.

Volkswagen uses virtual reality in interactive exhibit at shopping malls

My company's customer, Maniak Experiencial, delivered an interactive exhibit for Volkswagen. In this exhibit, a futuristic car was placed in shopping malls, and treated visitors to an exciting driving simulator using the zSight HMD

Check out the many images and additional details here

"Resolutions" for the New Year

Happy New Year!

This post is not going to be about the "less coffee; more exercise" type of resolutions for the new year. Instead, let's discuss display resolutions and how they are shaping up for 2014.

A few years ago, Apple introduced the "Retina Display" as a marketing term. The thought was that the resolution of the display is so high that when held in the typical viewing distance, the pixel density is practically as high as can be distinguished by the naked eye, and thus increasing the resolution for that viewing distance won't generate any benefits.

Let's run through the numbers. 25cm (10 inches) is considered to be closest comfortable viewing distance for most people. The visual acuity of the eye - assuming 20/20 vision - is considered to be 1 arcmin/pixel or approximately 60 pixels per degree. At a distance of 25 cm, 1 degree takes up 25*tan(1 degree) or about 0.44 cm. Thus, if 60 pixels take up 0.44 cm, 1 inch would require 60*2.54/0.44 = 346 pixels/inch, sometimes referred to as 346 DPI (dots per inch) or PPI (pixels per inch) at this distance of 25 cm . Indeed, Apple has 326 DPI in the iPhone 4, so pretty close.

Some have argued that Apple's claim is misleading and that a display needs to have at 477 DPI to truly have eye-limiting resolution. So, when new 5" 1080p flat panel displays - such as those from Sharp - came with 433 DPI, that should have been nearly enough, right?

Now, we are hearing about 538 DPI flat-panel screens, 2560x1440 resolution, that are coming out in smartphone or small tablet form factor. Where are all these people that can tell apart 443 DPI but can't tell apart 538 DPI? Other than the "my display has more pixels than your display" claim, is there really a tangible benefit for phones to have higher and higher DPI?

If you make goggles that use smartphone displays, you love these higher display resolutions. Goggles magnify displays so when a 1080p display gets magnified to - for the sake of examples - 90 degrees wide, the pixel density is 1920/90=21.3 pixels/degree, so still far from eye-limiting. a 2560x1440 display magnified the same way will produce 28.4 pixels/degree, providing a tangible improvement. To get to eye-limiting resolution at 90 degrees, you'd need a screen that is physically small enough to be worn on the head yet has 5400 pixels across. Good luck finding the GPU that can drive interactive content for that display!

Several years ago, my company developed the xSight HMD which used a unique optical tiling system to combine multiple 800x600 OLED displays into one large virtual display that had about 1920x1080 pixels. Today, similar performance can be achieved without tiling.

Thus, as long as goggle makers can ride the smartphone wave, they can get better and better resolutions, but since even 2560x1440 sounds like somewhat of an overkill, what's next? Will the next-generation goggles continue to use flat panel displays or will they gravitate towards other technologies?

It should be a fascinating 2014. Happy New Year!

What I'm looking for at CES

If you like to experience sensory overload, go to CES. 3200 exhibitors, hundreds of large-screen TVs hanging off the ceiling, celebrity appearances, flashy cars, and of course Las Vegas outside the convention center.

Here's what I'm looking to see at CES:

• The latest and greatest in portable (man worn) display technologies. Would love to see which are ready for prime time within a Sensics product.
• All kinds of sensors: motion sensors, hand and finger sensors, biometric sensors, eye trackers, full body sensing, proximity sensors, embedded cameras for augmented reality. In short, anything that can be reasonably combined with a VR goggle to create the next-generation experience.
• Companies that have VR/AR ideas and concepts and unique specifications but need help in refining these specifications and then building high-performance and affordable products around them. In the last year, my company has done several such projects and have the designs, IP and decade of experience to help. High-end gaming goggle? Unique display or signal processing requirements? We can help.
• Last, catch up with friends and professional acquaintances. It's not often that so many of my network are in the same city at the same time.

Whether you are 'buying' or 'selling' in these categories, drop me a note and perhaps we can meet at the show.

Happy Holidays to all. Rest well and get ready for CES 2014!

The VR goggle as a Sensory and Computing platform

While there is still a lot of work to do on display technologies and optics to get the best possible image in front of the eyes, the real promise of virtual reality goggles is to serve as a portable sensory and computing platform.

Goggles are becoming a platform for several reasons:

• They are a physical platform. Once you securely position goggles on the head, you now have a physical base to attach additional sensors and peripherals: cameras, trackers, depth sensors and more.
• Portable computing is becoming evermore powerful, thereby creating an incentive to process and analyze sensory data on the goggles as opposed to transmitting large amount of information to some computing base. Furthermore, a key part of the value of goggles is their portability, so the ability to process 'locally' - on the goggle - contributes to the realization of this value.
• As goggles become increasingly immersive, the value of sensors increase as a way to tie the experience into physical objects in the immediate surroundings, as well as connect the actions and context of the user to what is happening 'inside' the display.

One could look at the following diagram - courtesy of Sensics - as a good illustration to what these sensors might be:

One could imaging several types of sensors as feeding into the goggle:

• Orientation and position sensors - whether for the head, limbs or objects such as a gaming sword
• Cameras that might provide visible, IR or depth map information
• Positional sensors such as GPS or indoor location sensors
• Eye tracking sensors to understand gaze direction and potentially provide user interface
• Biometric sensors such as heart rate, perspiration, blood pressure. An eye tracker can also provide pupil size which is another biometric input.
• and more

One would then need to turn this sensor data into useful information. For instance, turn position and orientation of various body parts into understanding of gestures; turn a depth map into detection of nearby obstacles; detect faces and markers from a video image.

As we have discussed before, a virtual reality abstraction layer is going to speed up this process as it will allow those that turn data into information to relieve themselves from the need to worry about the particular formats and features of individual sensors and focus on a category of sensors.

There are several places where this transformation could happen: on the goggle with the help of an embedded processor (as shown in the above diagram); Near the goggle on a device such as a tablet or powerful smartphone; At a stationary device such as a desktop PC or gaming console. Performing this transformation in or near the goggle allows running the application software in or near the goggle, leading to a truly portable solution.

What is the best place to do so? As Nelson Mandela - and others - have said "Where you stand depends on where you sit". If you are a PC vendor that is married to everything happening on the PC, you shudder at the notion of the goggle being an independent computing platform. If you are a goggle vendor, you might find that this architecture opens up additional opportunities that do not exist when you are physically and logically tied to a stationary platform. If you are a vendor making phones or tablets, this might allow you to position the phone as a next-generation portable gaming platform.

So, beyond innovations in displays and optics, I think we will see lots of interesting sensors and sensor fusion applications in 2014.

What do you think? Let me know.