ARCore Save Camera Image (Unity C#) on Button clic

Question

I've a similiar question like the follow three SO questions:

• Save AcquireCameraImageBytes() from Unity ARCore to storage as an image
• Save Camera image from unity ARCore
• ARCore for Unity save camera image

My aim is to save the camera image (without the augmented objects) and the pose of the camera on a button click. Today I tried the whole day on saving the camera image with ARCore. I tried the different approaches from the three questions linked above, but it didn't worke. My C# Script which is attached to a button:

using UnityEngine;
using UnityEngine.UI;
using GoogleARCore;
using System.IO;
public class takeimg : MonoBehaviour
{
    private Texture2D m_TextureRender;
    public Button yourButton;
    private byte[] m_EdgeDetectionResultImage = null;

    void Start()
    {
        Button btn = yourButton.GetComponent<Button>();
        btn.onClick.AddListener(TaskOnClick);
    }

    void TaskOnClick()
    {
        var image = Frame.CameraImage.AcquireCameraImageBytes();

        m_TextureRender = new Texture2D(image.Width, image.Height, TextureFormat.RGBA32, false, false);
        m_EdgeDetectionResultImage = new byte[image.Width * image.Height * 4];


        System.Runtime.InteropServices.Marshal.Copy(image.Y, m_EdgeDetectionResultImage, 0, image.Width * image.Height * 4);

        m_TextureRender.LoadRawTextureData(m_EdgeDetectionResultImage);
        m_TextureRender.Apply();

        var encodedJpg = m_TextureRender.EncodeToJPG();
        var path = Application.persistentDataPath;

        File.WriteAllBytes(path + "/test.jpg", encodedJpg);
    }
}

Currently the image i get looks: Saved Image It looks similiar to the third SO question I linked above. So something is still wrong/missing. Can someone help me what's wrong? Something with the buffers?

Update: In the meantime I managed to get back a black/withe picture: bw picture Here my new TaskOnClick function:

void TaskOnClick()
{
    var image = Frame.CameraImage.AcquireCameraImageBytes();

    byte[] bufferY = new byte[image.Width * image.Height];
    byte[] bufferU = new byte[image.Width * image.Height / 2];
    byte[] bufferV = new byte[image.Width * image.Height / 2];
    System.Runtime.InteropServices.Marshal.Copy(image.Y, bufferY, 0, image.Width * image.Height);
    System.Runtime.InteropServices.Marshal.Copy(image.U, bufferU, 0, image.Width * image.Height / 2);
    System.Runtime.InteropServices.Marshal.Copy(image.V, bufferV, 0, image.Width * image.Height / 2);


    m_TextureRender = new Texture2D(image.Width, image.Height, TextureFormat.RGBA32, false, false);


    Color c = new Color();
    for (int y = 0; y < image.Height; y++) {
        for (int x =0; x<image.Width;x++) {
            float Y = bufferY[y * image.Width + x];
            float U = bufferU[(y/2) * image.Width + x];
            float V = bufferV[(y/2) * image.Width + x];
            c.r = Y;
            c.g = Y;
            c.b = Y;

            c.r /= 255.0f;
            c.g /= 255.0f;
            c.b /= 255.0f;

            if (c.r < 0.0f) c.r = 0.0f;
            if (c.g < 0.0f) c.g = 0.0f;
            if (c.b < 0.0f) c.b = 0.0f;

            if (c.r > 1.0f) c.r = 1.0f;
            if (c.g > 1.0f) c.g = 1.0f;
            if (c.b > 1.0f) c.b = 1.0f;

            c.a = 1.0f;
            m_TextureRender.SetPixel(image.Width-1-x, y, c);      
        }
    }

    var encodedJpg = m_TextureRender.EncodeToJPG();
    var path = Application.persistentDataPath;
    File.WriteAllBytes(path + "/test.jpg", encodedJpg);
}

Can someone tell, which is the acutal YUV to RGB conversation Google ARCore is useing? I tried some, but the colors in the pictures looked always wrong... Is there an easier way to save the camera image from the current frame than my solution?

Answer 1

You could do this pretty easily with the NatCam API: https://assetstore.unity.com/packages/tools/integration/natcam-webcam-api-52154

There is also NatCorder if you want video