Hololens font camera

Question

I'm working with hololens, and I'm trying to get the image of the front camera. The only thing I need is to take the image of each frame of that camera and transform it into a byte array.

Answer 1

In player settings, enable access to the camera, and try again.

Answer 2

1. Follow Unity Example in the documentation. It has a well written example: https://docs.unity3d.com/Manual/windowsholographic-photocapture.html

Copied from the unity documentation from the link above:

using UnityEngine;
using System.Collections;
using System.Linq;
using UnityEngine.XR.WSA.WebCam;

public class PhotoCaptureExample : MonoBehaviour {
PhotoCapture photoCaptureObject = null;
Texture2D targetTexture = null;

// Use this for initialization
void Start() {
    Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
    targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);

    // Create a PhotoCapture object
    PhotoCapture.CreateAsync(false, delegate (PhotoCapture captureObject) {
        photoCaptureObject = captureObject;
        CameraParameters cameraParameters = new CameraParameters();
        cameraParameters.hologramOpacity = 0.0f;
        cameraParameters.cameraResolutionWidth = cameraResolution.width;
        cameraParameters.cameraResolutionHeight = cameraResolution.height;
        cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;

        // Activate the camera
        photoCaptureObject.StartPhotoModeAsync(cameraParameters, delegate (PhotoCapture.PhotoCaptureResult result) {
            // Take a picture
            photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
        });
    });
}

void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame) {
    // Copy the raw image data into the target texture
    photoCaptureFrame.UploadImageDataToTexture(targetTexture);

    // Create a GameObject to which the texture can be applied
    GameObject quad = GameObject.CreatePrimitive(PrimitiveType.Quad);
    Renderer quadRenderer = quad.GetComponent<Renderer>() as Renderer;
    quadRenderer.material = new Material(Shader.Find("Custom/Unlit/UnlitTexture"));

    quad.transform.parent = this.transform;
    quad.transform.localPosition = new Vector3(0.0f, 0.0f, 3.0f);

    quadRenderer.material.SetTexture("_MainTex", targetTexture);

    // Deactivate the camera
    photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}

void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result) {
    // Shutdown the photo capture resource
    photoCaptureObject.Dispose();
    photoCaptureObject = null;
}
}

To get bytes: Replace the following method:

    void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame) {
    // Copy the raw image data into the target texture
    photoCaptureFrame.UploadImageDataToTexture(targetTexture);

    // Create a GameObject to which the texture can be applied
    GameObject quad = GameObject.CreatePrimitive(PrimitiveType.Quad);
    Renderer quadRenderer = quad.GetComponent<Renderer>() as Renderer;
    quadRenderer.material = new Material(Shader.Find("Custom/Unlit/UnlitTexture"));

    quad.transform.parent = this.transform;
    quad.transform.localPosition = new Vector3(0.0f, 0.0f, 3.0f);

    quadRenderer.material.SetTexture("_MainTex", targetTexture);

    // Deactivate the camera
    photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}

with the method below:

void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
    List<byte> imageBufferList = new List<byte>();
    imageBufferList.Clear();
    photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);
    var bytesArray = imageBufferList.ToArray();
}

2. If you are using Unity 2018.1 or 2018.2 (I think also 2017.4) now, then it will NOT work. Unity has public tracker to resolve it: https://issuetracker.unity3d.com/issues/windowsmr-failure-to-take-photo-capture-in-hololens
3. I created a workaround until Unity fixes the bug: https://github.com/MSAlshair/HoloLensMediaCapture

Basic sample without using PhotoCapture from unity as a workaround: More details in the link above

You must add #if WINDOWS_UWP to be able to use MediaCapture: Ideally, you want to use PhotoCapture from unity to avoid this, but until Unity resolve the issue, you can use something like this.

#if WINDOWS_UWP
        public async System.Threading.Tasks.Task<byte[]> GetPhotoAsync()
        {
            //Get available devices info
            var devices = await Windows.Devices.Enumeration.DeviceInformation.FindAllAsync(
                Windows.Devices.Enumeration.DeviceClass.VideoCapture);
            var numberOfDevices = devices.Count;

            byte[] photoBytes = null;

            //Check if the device has camera
            if (devices.Count > 0)
            {
                Windows.Media.Capture.MediaCapture mediaCapture = new Windows.Media.Capture.MediaCapture();
                await mediaCapture.InitializeAsync();

                //Get Highest available resolution
                var highestResolution = mediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(
                    Windows.Media.Capture.MediaStreamType.Photo).
                    Select(item => item as Windows.Media.MediaProperties.ImageEncodingProperties).
                    Where(item => item != null).
                    OrderByDescending(Resolution => Resolution.Height * Resolution.Width).
                    ToList().First();

                using (var photoRandomAccessStream = new Windows.Storage.Streams.InMemoryRandomAccessStream())
                {
                    await mediaCapture.CapturePhotoToStreamAsync(highestResolution, photoRandomAccessStream);

                    //Covnert stream to byte array
                    photoBytes = await ConvertFromInMemoryRandomAccessStreamToByteArrayAsync(photoRandomAccessStream);
                }
            }
            else
            {
                System.Diagnostics.Debug.WriteLine("No camera device detected!");
            }

            return photoBytes;
        }

        public static async System.Threading.Tasks.Task<byte[]> ConvertFromInMemoryRandomAccessStreamToByteArrayAsync(
            Windows.Storage.Streams.InMemoryRandomAccessStream inMemoryRandomAccessStream)
        {
            using (var dataReader = new Windows.Storage.Streams.DataReader(inMemoryRandomAccessStream.GetInputStreamAt(0)))
            {
                var bytes = new byte[inMemoryRandomAccessStream.Size];
                await dataReader.LoadAsync((uint)inMemoryRandomAccessStream.Size);
                dataReader.ReadBytes(bytes);

                return bytes;
            }
        }
#endif

Do NOT forget to add capabilities to the manifest:

1. WebCam
2. Microphone: I am not sure if it needed since we are only taking photos or not, but I added it anyway.
3. Pictures Library if you want to save to it