How to mimic HoloLens 2 hand tracking wIth Windows

Question

The HoloLens 2 will feature hand tracking and the ability to reach out and poke UI elements. With Unity and the Mixed Reality Toolkit V2, the input for the hand-tracked near interactions (ie poking) comes the PokePointer class and to generate events for GameObjects having BaseNearInteractionTouchable components.

My question is how can we get the same PokePointer events from virtual reality controllers such as the Windows Mixed Reality controllers? This would make it possible to prototype on the desktop using a VR headset and even directly use the same near interactions of the Mixed Reality Toolkit within VR applications.

Can the PokePointer component be attached to a hand GameObject that is a controller model? Or is there a better way to do this through the MRTK profiles system?

Answer 1

Actually, it's possible to add a poke pointer and a grab pointer to a VR device. In fact, adding basic functionality without visualization can be done without even writing any code!

Getting the existing grab and poke pointers to work with VR

1. Open your current pointer configuration profile by selecting the MixedRealityToolkit object in the scene view, going to the inspector window, then navigating to Input -> Pointers.

2. Under pointer options, set the controller type for the PokePointer and the Grab Pointer to include your VR Controller type (in my case, it was Windows Mixed Reality, though you may wish to use OpenVR)

3. The poke pointer is configured to follow the Index Finger Pose, which does not exist for VR. So you will need to open the PokePointer.prefab file and in the inspector, Under Poke Poker -> Pose Action, set the value to "Pointer Pose"

4. Hit play. The grab pointer will be slightly below and do the right of the motion controller gizmo, and the poke pointer will appear be right at the origin.

Bonus: Improving the grab, poke pointers by using custom pointer

You can greatly improve the pointers you have by using custom pointers instead of the default pointers. For example, you can:

• have the poke pointer be offset from the gizmo origin by setting the PokePointer's raycastorigin field to a custom transform
• Add visuals to actually show where the pointers are

I've created an example that demonstrates a custom grab and poke pointer which visualizes the grab and poke locations, and also offsets the poke position to be more convenient. You can download a unitypackage of the sample here, or just clone the mrtktips repository and look at the VRGrabPokePointers scene.

Note: to get the visuals to actually show up, use the following script (pointers currently disable all renderers on startup to avoid flickering).

using UnityEngine;

public class EnableRenderers : MonoBehaviour
{
    void Start()
    {
        foreach (var renderer in GetComponentsInChildren<Renderer>())
        {
            renderer.enabled = true;
        }
    }
}

You can see an example of a custom MRTK and pointer profile in the example here, and also in the VRGrabPokePointersUnity scene