How to do occlusion with Google Tango in Unity?

Question

I am tring to do occlusion with Google Tango in Unity.

What I want is pretty simple to understand: when there is a real object in front of a virtual object, the virtual object is hidden (or rendered differently)

The perfect result would be like it is in this impressive video I found: https://www.youtube.com/watch?v=EpDhaM7ZhZs .

I already tried the "Enable occlusion" option of the Tango Camera and I am not so happy with the results (it is not accurate and not real time as it is based on mesh reconstruction from the point cloud).

If you have hints, tips or ideas about how to achieve this (like in the video), that would be awesome!

Answer 1

Occlusion is still a very experimental feature on Tango. The problem is that it's very hard to do occlusion with high fidelity and high performance, here's couple of ideas on how to achieve it using different method:

1. Use 3D reconstruction.

Tango does provide functionalities to construct 3D meshes from point cloud, you can find sample code from Tango sample code repository (C, Java, Unity). If you have a world that is pre-scanned, you can essentially use that mesh data to occluded virtual object.

2. Run time up-sampling depth image.

You can also project all point clouds on to an image plane, up-sample it, and use the image as a depth buffer for rendering. This is what ARScreen occlusion is using in TangoUnitySDK. Due to the limitation of Tango depth sensing hardware, the result quality is not very ideal, and it will not work if all physical objects are far away(beyond 4 meters) from the device.