Correct way to marshall uchar[] from native dll to

Question

I'm trying to marshall some data that my native dll allocated via CoTaskMemAlloc into my c# application and wondering if the way I'm doing it is just plain wrong or I'm missing some sublte decorating of the method c# side.

Currently I have c++ side.

extern "C" __declspec(dllexport) bool __stdcall CompressData(  unsigned char* pInputData, unsigned int inSize, unsigned char*& pOutputBuffer, unsigned int& uOutputSize)
{ ...
    pOutputBuffer = static_cast<unsigned char*>(CoTaskMemAlloc(60000));
    uOutputSize = 60000;

And on the C# side.

    private const string dllName = "TestDll.dll";

    [System.Security.SuppressUnmanagedCodeSecurity]
    [DllImport(dllName)]
    public static extern bool CompressData(byte[] inputData, uint inputSize, out byte[] outputData, out uint outputSize );
    ...
    byte[] outputData;
    uint outputSize;
    bool ret = CompressData(packEntry.uncompressedData, (uint)packEntry.uncompressedData.Length, out outputData, out outputSize);

here outputSize is 60000 as expected, but outputData has a size of 1, and when I memset the buffer c++ side, it seems to only copy across 1 byte, so is this just wrong and I need to marshall the data outside the call using an IntPtr + outputSize, or is there something sublte I'm missing to get working what I have already?

Thanks.

Answer 1

There are two things.

First, the P/Invoke layer does not handle reference parameters in C++, it can only work with pointers. The last two parameters (pOutputBuffer and uOutputSize) in particular are not guaranteed to marshal correctly.

I suggest you change your C++ method declaration to (or create a wrapper of the form):

extern "C" __declspec(dllexport) bool __stdcall CompressData(  
    unsigned char* pInputData, unsigned int inSize, 
    unsigned char** pOutputBuffer, unsigned int* uOutputSize)

That said, the second problem comes from the fact that the P/Invoke layer also doesn't know how to marshal back "raw" arrays (as opposed to say, a SAFEARRAY in COM that knows about it's size) that are allocated in unmanaged code.

This means that on the .NET side, you have to marshal the pointer that is created back, and then marshal the elements in the array manually (as well as dispose of it, if that's your responsibility, which it looks like it is).

Your .NET declaration would look like this:

[System.Security.SuppressUnmanagedCodeSecurity]
[DllImport(dllName)]
public static extern bool CompressData(byte[] inputData, uint inputSize, 
    ref IntPtr outputData, ref uint outputSize);

Once you have the outputData as an IntPtr (this will point to the unmanaged memory), you can convert into a byte array by calling the Copy method on the Marshal class like so:

var bytes = new byte[(int) outputSize];

// Copy.
Marshal.Copy(outputData, bytes, 0, (int) outputSize);

Note that if the responsibility is yours to free the memory, you can call the FreeCoTaskMem method, like so:

Marshal.FreeCoTaskMem(outputData);

Of course, you can wrap this up into something nicer, like so:

static byte[] CompressData(byte[] input, int size)
{
    // The output buffer.
    IntPtr output = IntPtr.Zero;

    // Wrap in a try/finally, to make sure unmanaged array
    // is cleaned up.
    try
    {
        // Length.
        uint length = 0;

        // Make the call.
        CompressData(input, size, ref output, ref length);

        // Allocate the bytes.
        var bytes = new byte[(int) length)];

        // Copy.
        Marshal.Copy(output, bytes, 0, bytes.Length);

        // Return the byte array.
        return bytes;
    }
    finally
    {
        // If the pointer is not zero, free.
        if (output != IntPtr.Zero) Marshal.FreeCoTaskMem(output);
    }
}

Answer 2

The pinvoke marshaller cannot guess how large the returned byte[] might be. Raw pointers to memory in C++ do not have a discoverable size of the pointed-to memory block. Which is why you added the uOutputSize argument. Good for the client program but not quite good enough for the pinvoke marshaller. You have to help and apply the [MarshalAs] attribute to pOutputBuffer, specifying the SizeParamIndex property.

Do note that the array is getting copied by the marshaller. That's not so desirable, you can avoid it by allowing the client code to pass an array. The marshaller will pin it and pass the pointer to the managed array. The only trouble is that the client code will have no decent way to guess how large to make the array. The typical solution is to allow the client to call it twice, first with uOutputSize = 0, the function returns the required array size. Which would make the C++ function look like this:

extern "C" __declspec(dllexport) 
int __stdcall CompressData(
     const unsigned char* pInputData, unsigned int inSize, 
     [Out]unsigned char* pOutputBuffer, unsigned int uOutputSize)