Efficient small byte-arrays in C#

I have a huge collection of very small objects. To ensure the data is stored very compactly I rewrote the class to store all information within a byte-array with variable-byte encoding. Most instances of these millions of objects need only 3 to 7 bytes to store all the data.

After memory-profiling I found out that these byte-arrays always take at least 32 bytes.

Is there a way to store the information more compactly than bit-fiddled into a byte[]? Would it be better to point to an unmanaged array?

class MyClass
{
    byte[] compressed;

    public MyClass(IEnumerable<int> data)
    {
        compressed = compress(data);
    }

    private byte[] compress(IEnumerable<int> data)
    {
        // ...
    }

    private IEnumerable<int> decompress(byte[] compressedData)
    {
        // ...
    }

    public IEnumerable<int> Data { get { return decompress(compressed); } }
}

There are a couple problems you're facing that eat up memory. One is object overhead, and the other is objects aligning to 32 or 64 bit boundaries (depending on your build). Your current approach suffers from both issues. The following sources describe this in more detail:

Of Memory and Strings
How much memory does a C# string take up

I played around with this when I was fiddling with benchmarking sizes.

A solution that is simple would be to simply create a struct that has a single member that is a long value. Its methods would handle packing and unpacking bytes into and out of that long, using shift and mask bit fiddling.

Another idea would be a class that served up objects by ID, and stored the actual bytes in a single backing List<byte>. But this would get complicated and messy. I think the struct idea is much more straightforward.