Xamarin tf.lite input objects

问题:

Im trying to reproduce tensorflow object detection on xamarin.

    private MappedByteBuffer LoadModelFile()
    {
        AssetFileDescriptor fileDescriptor = Assets.OpenFd("detect.tflite");           
        FileInputStream inputStream = new FileInputStream(fileDescriptor.FileDescriptor);
        FileChannel fileChannel = inputStream.Channel;
        long startOffset = fileDescriptor.StartOffset;
        long declaredLength = fileDescriptor.DeclaredLength;
        return fileChannel.Map(FileChannel.MapMode.ReadOnly, startOffset, declaredLength);
    }

        View view = (View) sender;

        MappedByteBuffer buffer = LoadModelFile();

        Interpreter interpreter = new Interpreter(buffer);

        var sr = new StreamReader(Assets.Open("labels.txt"));

        var labels = sr.ReadToEnd()
            .Split('\n')
            .Select(s => s.Trim())
            .Where(s => !string.IsNullOrEmpty(s))
            .ToList();

        var bitmap =  BitmapFactory.DecodeResource(Resources, 2130837608);
        var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, 1000, 750, false)
            .Copy(Bitmap.Config.Argb8888, false);

        float[][][][] imgData = null;

        imgData  = new float[1][][][];
        imgData[0] = new float[1000][][];

        for (int i = 0; i < imgData[0].Length; i++)
        {
            imgData[0][i] = new float[750][];
            for (int j = 0; j < imgData[0][i].Length; j++)
            {
                imgData[0][i][j] = new float[3];
            }
        }
        var intValuess = new int[1000 * 750];
        resizedBitmap.GetPixels(intValuess, 0, 1000, 0, 0, 1000, 750);

        int pixels = 0;
        for (int i = 0; i < imgData[0].Length; i++)
        {
            for (int j = 0; j < imgData[0][i].Length; j++)
            {
                var val = intValuess[pixels++];
                imgData[0][i][j][0] = (float)((val >> 16) & 0xFF);
                imgData[0][i][j][1] = (float)((val >> 8) & 0xFF);
                imgData[0][i][j][2] = (float)(val & 0xFF);
            }
        }

        var outputs = new float[labels.Count];

        interpreter.Run(imgData, outputs);

but i have error "cannot convert float[][][][] to Java.Lang.Object in line interpreter.Run(imgData, outputs); How i can convert float[][][][] to Java.Lang.Object or where i can find tensorflow lite with xamarin examples.

回答1:

I know it has been a while since you asked this question but maybe my response can be useful to someone. I am also trying to use Xamarin with tflite, to run a simple CNN. Here is my code:

private MappedByteBuffer LoadModelFile()
{
    var assets = Application.Context.Assets;
    AssetFileDescriptor fileDescriptor = assets.OpenFd("seed_model_no_qt.tflite");
    FileInputStream inputStream = new FileInputStream(fileDescriptor.FileDescriptor);
    FileChannel fileChannel = inputStream.Channel;
    long startOffset = fileDescriptor.StartOffset;
    long declaredLength = fileDescriptor.DeclaredLength;
    return fileChannel.Map(FileChannel.MapMode.ReadOnly, startOffset, declaredLength);
}

private string Classify(MediaFile mediaFile)
{
    var assets = Application.Context.Assets;

    Bitmap bp = BitmapFactory.DecodeStream(mediaFile.GetStream());
    var resizedBitmap = Bitmap.CreateScaledBitmap(bp, 1280, 1280, false).Copy(Bitmap.Config.Argb8888, false);

    var bufint = new int[1280 * 1280];
    resizedBitmap.GetPixels(bufint, 0, 1280, 0, 0, 1280, 1280);
    int pixels = 0;
    var input_buffer = new byte[4 * 1280 * 1280 * 3];            
    for(int i = 0; i < 1280; i++)
    {
        for(int k = 0; k < 1280; k++)
        {
            int val = bufint[pixels++];
            Array.Copy(BitConverter.GetBytes(((val >> 16) & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12, 4);
            Array.Copy(BitConverter.GetBytes(((val >> 8) & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12 + 4, 4);
            Array.Copy(BitConverter.GetBytes((val & 0xFF) * (1f / 255f)), 0, input_buffer, (i * 1280 + k) * 12 + 8, 4);
        }
    }
    var bytebuffer = Java.Nio.ByteBuffer.Wrap(input_buffer);
    var output = Java.Nio.ByteBuffer.AllocateDirect(4*160*160);
    interpreter.Run(bytebuffer, output);

    var buffer = new byte[4 * 160 * 160];

    Marshal.Copy(output.GetDirectBufferAddress(), buffer, 0, 4 * 160 * 160);

    float sum = 0.0f;
    for(int i = 0; i < 160*160; i++)
    {
        sum += BitConverter.ToSingle(buffer, i * 4);
    }

    return "Count : " + ((int)(sum/255)).ToString();
}

I reused your LoadModelFile() function as it is. The code takes an image from a mediaFile (coming from the phone camera), then resizes it to 1280x1280 rgb image before feeding it to a CNN as an array of float32 values. Your float[][][][] to Java.Lang.Object issue came from the interpreter.Run() method expecting a Java Object. Some people online solve it by giving a Java.Nio.ByteBuffer as a parameter, instead of an array. It implies some bitwise manipulations but the Run method does accept the ByteBuffer object. When filling the ByteBuffer, I advise you not to use its methods such as PutFloat(), but to fill a byte[] buffer and then use the Java.Nio.ByteBuffer.Wrap() method as I did. Using ByteBuffer's methods seemed to imply large performance issues in my case. Same thing happens when manipulating the output of my CNN (a 160x160 heatmap of float32 values). Using ByteBuffer.Get() method to access the values was very slow. Instead, use Marshal.Copy to store the values into a byte array, then get back the float values with BitConverter.ToSingle.