5 – Optical See-Through vs. Video See-Through

Video See-Through

If you’re interested in augmented reality, you’ve probably thought about optical see-through and video see-through.

Video-see through systems present video feeds from cameras inside head-mounted devices. This is the standard method that phones, for example, use AR with.  This can be useful when you need to experience something remotely: a robot which you send to fix a leak inside a chemical plant; a vacation destination that you’re thinking about. This is also useful when using an image enhancement system: thermal imagery, night-vision devices, etc.

Optical See-Through

Optical see-through systems combine computer-generated imagery with “through the glasses” image of the real world, usually through a slanted semi-transparent mirror. If you are in a mission-critical application and you’re concerned what happens should your power fail, an optical see-through solution will allow you to see something in that extreme situation. If you are concerned about the ultimate image quality, portable cameras and fully-immersive head-mounted display can’t match the “direct view” experience.

Another issue is the optics blocking the light itself. For example, if an LCOS optics blocks out 40% of outside light then optical see-through would seem as if you are viewing the real world through sunglasses.  Once again this comes down to user preference- would you prefer the camera view or view the real world as is but with a slight camera tint?   Every optics in smart glasses blocks out a different amount of light & some more than others, so it ultimately comes down to preference.

Optical See-Through vs. Video See-Through

One big advantage for video see-through is that any AR objects mapped to real-world objects are accurate because the processing happens on the same pipeline.  For optical see-through, a manual translation is needed so that the AR objects are mapped to the real world.

How to make your app optical see-through:

Step 1: Do not render the camera view.  If you are testing on the phone, the screen background will look black (deep black image below) since the actual camera view is not being rendered.  If testing on the X1, you will just see a slightly dark shade tint for the light blockage.

Step 2: Render your AR content.

Step 3: (Only if your AR content is mapped to a real-world object) Adjust the relevant AR data

For see-through video apps, no manual adjustment is needed as the standard camera feed is used.

One aspect of see-through video systems is that it’s much easier to match the video latency with the computer graphics latency. Latency (delay) is inherent to immersive imaging systems: motion trackers are not instantaneous; computer graphic generation is not immediate, and even when refreshing images at 60, 70, even 120 Hz, there is a lag from sensing to imaging. When computer graphics need to be overlaid on the image from the actual world, there is a difference between video see-through and optical see-through. Optical see-through offers no latency, which sounds great, except that there is an inherent mismatch or lack of synchronization between what you see through the glasses and the graphics. If you’re showing a virtual sofa inside a real living room, this mismatch can be distracting. In contrast, using video see-through allows you to synchronize the delay so that your video and graphics are always in sync.

1