How can I quickly read bytes from a memory mapped

Question

In some situations the MemoryMappedViewAccessor class just doesn't cut it for reading bytes efficiently; the best we get is the generic ReadArray<byte> which it the route for all structs and involves several unnecessary steps when you just need bytes.

It's possible to use a MemoryMappedViewStream, but because it's based on a Stream you need to seek to the correct position first, and then the read operation itself has many more unnecessary steps.

Is there a quick, high-performance way to read an array of bytes from a memory-mapped file in .NET, given that it should just be a particular area of the address space to read from?

Answer 1

See this bug report: No way to determine internal offset used by MemoryMappedViewAccessor - Makes SafeMemoryMappedViewHandle property unusable.

From the report:

MemoryMappedViewAccessor has a SafeMemoryMappedViewHandle property, which returns the ViewHandle being used internally by the MemoryMappedView, but does not have any property to return the offset being used by the MemoryMappedView.

As the MemoryMappedView is page aligning the offset requested in MemoryMappedFile.CreateViewAccessor(offset,size) it is impossible to use the SafeMemoryMappedViewHandle for anything useful without knowing the offset.

Note that what we actually want to do is use the AcquirePointer(ref byte* pointer) method to allow some fast pointer based (possibly unmanaged) code to run. We're OK with the pointer being page aligned, but it must be possible to find out what the offset from the originally requested address is.

Answer 2

A safe version of this solution is:

var file = MemoryMappedFile.CreateFromFile(...);
var accessor = file.CreateViewAccessor();
var bytes = new byte[yourLength];

// assuming the string is at the start of the file
// aka position: 0
// https://msdn.microsoft.com/en-us/library/dd267761(v=vs.110).aspx
accessor.ReadArray<byte>(
    position: 0,      // The number of bytes in the accessor at which to begin reading
    array: bytes,     // The array to contain the structures read from the accessor
    offset: 0,        // The index in `array` in which to place the first copied structure
    count: yourLength // The number of structures of type T to read from the accessor.
);

var myString = Encoding.UTF8.GetString(bytes);

I have tested this, it does work. I cannot comment on it's performance or if it's the BEST overall solution just that it works.

Answer 3

This solution requires unsafe code (compile with /unsafe switch), but grabs a pointer to the memory directly; then Marshal.Copy can be used. This is much, much faster than the methods provided by the .NET framework.

    // assumes part of a class where _view is a MemoryMappedViewAccessor object

    public unsafe byte[] ReadBytes(int offset, int num)
    {
        byte[] arr = new byte[num];
        byte *ptr = (byte*)0;
        this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.Copy(IntPtr.Add(new IntPtr(ptr), offset), arr, 0, num);
        this._view.SafeMemoryMappedViewHandle.ReleasePointer();
        return arr;
    }

    public unsafe void WriteBytes(int offset, byte[] data)
    {
        byte* ptr = (byte*)0;
        this._view.SafeMemoryMappedViewHandle.AcquirePointer(ref ptr);
        Marshal.Copy(data, 0, IntPtr.Add(new IntPtr(ptr), offset), data.Length);
        this._view.SafeMemoryMappedViewHandle.ReleasePointer();
    }

Answer 4

I know this is an older question which has been answered but I wanted to add my two cents.

I ran a test with both the accepted answer (using the unsafe code) and with the MemoryMappedViewStream approach for reading a 200MB byte array.

MemoryMappedViewStream

        const int MMF_MAX_SIZE = 209_715_200;
        var buffer = new byte[ MMF_VIEW_SIZE ];

        using( var mmf = MemoryMappedFile.OpenExisting( "mmf1" ) )
        using( var view = mmf.CreateViewStream( 0, buffer.Length, MemoryMappedFileAccess.ReadWrite ) )  
        {
            if( view.CanRead )
            {
                Console.WriteLine( "Begin read" );
                sw.Start( );
                view.Read( buffer, 0, MMF_MAX_SIZE );
                sw.Stop( );
                Console.WriteLine( $"Read done - {sw.ElapsedMilliseconds}ms" );
            }
        }

I ran the test 3 times with each approach and received the following times.

MemoryMappedViewStream:

1. 483ms
2. 501ms
3. 490ms

Unsafe method

1. 531ms
2. 517ms
3. 523ms

From the small amount of testing it looks like the MemoryMappedViewStream has a very slight advantage. With that in mind for anyone reading this post down the road I would go with the MemoryMappedViewStream.