Concatenating a C# List of byte[]

Question

I am creating several byte arrays that need to be joined together to create one large byte array - i'd prefer not to use byte[]'s at all but have no choice here...

I am adding each one to a List as I create them, so I only have to do the concatenation once I have all the byte[]'s, but my question is, what is the best way of actually doing this?

When I have a list with an unknown number of byte[]'s and I want to concat them all together.

Thanks.

Answer 1

hmm how about list.addrange?

Answer 2

Use Linq:

    List<byte[]> list = new List<byte[]>();
    list.Add(new byte[] { 1, 2, 3, 4 });
    list.Add(new byte[] { 1, 2, 3, 4 });
    list.Add(new byte[] { 1, 2, 3, 4 });

    IEnumerable<byte> result = Enumerable.Empty<byte>();

    foreach (byte[] bytes in list)
    {
        result = result.Concat(bytes);
    }

    byte[] newArray = result.ToArray();

Maybe faster solution would be (not declaring array upfront):

IEnumerable<byte> bytesEnumerable = GetBytesFromList(list);

byte[] newArray = bytesEnumerable.ToArray();

private static IEnumerable<T> GetBytesFromList<T>(IEnumerable<IEnumerable<T>> list)
{
    foreach (IEnumerable<T> elements in list)
    {
        foreach (T element in elements)
        {
            yield return element;
        }
    }
}

It seems like above would iterate each array only once.

Answer 3

write them all to a MemoryStream instead of a list. then call MemoryStream.ToArray(). Or when you have the list, first summarize all the byte array lengths, create a new byte array with the total length, and copy each array after the last in the big array.

Answer 4

Instead of storing each byte array into a List<byte[]>, you could instead add them directly to a List<byte>, using the AddRange method for each one.

Answer 5

Here's a solution based on Andrew Bezzub's and fejesjoco's answers, pre-allocating all the memory needed up front. This yields Θ(N) memory usage and Θ(N) time (N being the total number of bytes).

byte[] result = new byte[list.Sum(a => a.Length)];
using(var stream = new MemoryStream(result))
{
    foreach (byte[] bytes in list)
    {
        stream.Write(bytes, 0, bytes.Length);
    }
}
return result;

Answer 6

listOfByteArrs.SelectMany(byteArr=>byteArr).ToArray()

The above code will concatenate a sequence of sequences of bytes into one sequence - and store the result in an array.

Though readable, this is not maximally efficient - it's not making use of the fact that you already know the length of the resultant byte array and thus can avoid the dynamically extended .ToArray() implementation that necessarily involves multiple allocations and array-copies. Furthermore, SelectMany is implemented in terms of iterators; this means lots+lots of interface calls which is quite slow. However, for small-ish data-set sizes this is unlikely to matter.

If you need a faster implementation you can do the following:

var output = new byte[listOfByteArrs.Sum(arr=>arr.Length)];
int writeIdx=0;
foreach(var byteArr in listOfByteArrs) {
    byteArr.CopyTo(output, writeIdx);
    writeIdx += byteArr.Length;
}

or as Martinho suggests:

var output = new byte[listOfByteArrs.Sum(arr => arr.Length)];
using(var stream = new MemoryStream(output))
    foreach (var bytes in listOfByteArrs)
        stream.Write(bytes, 0, bytes.Length);

Some timings:

var listOfByteArrs = Enumerable.Range(1,1000)
    .Select(i=>Enumerable.Range(0,i).Select(x=>(byte)x).ToArray()).ToList();

Using the short method to concatenate these 500500 bytes takes 15ms, using the fast method takes 0.5ms on my machine - YMMV, and note that for many applications both are more than fast enough ;-).

Finally, you could replace Array.CopyTo with the static Array.Copy, the low-level Buffer.BlockCopy, or a MemoryStream with a preallocated back buffer - these all perform pretty much identically on my tests (x64 .NET 4.0).