interesting OutOfMemoryException with StringBuilde

Question

I have the need to continuously build large strings in a loop and save them to database wich currently occasioanlly yields an OutOfMemoryException.

What is basically going on here is I create a string using XmlWriter with StringBuilder based on some data. Then I call a method from an external library that converts this xml string to some other string. After that the converted string is saved to the database. This whole thing is done repeatedly in a loop about a 100 times for different data.

The strings by itself are not too big (below 500kByte each) and the process memory is not increasing during this loop. But still, occasionally I get a OutOfMemeoryExcpetion within StringBuilder.Append. Interestingly this exception does not result in a crash. I can catch that exception and continue the loop.

What is going on here? Why would I get an OutOfMemoryException allthough there is still enough free memory available in the system? Is this some GC heap problem?

Given that I can't circumvent converting all these strings, what could I do to make this work reliably? Should I force a GC collection? Should put a Thread.Sleep into the loop? Should I stop using StringBuilder? Should simply retry when confronted with a OutOfMemoryException?

Answer 1

With the sizes you mention you are probably running into Large Object Heap (LOH) fragmentation.

Reusing StringBuilder objects is not a direct solution, you need to get a grip on the underlying buffers.
If possible, calculate or estimate the size beforehand and pre-allocate.

And it could help if you round up allocations, let's say to multiples of 20k or so. That could improve reuse.

Answer 2

There is memory but no contiguous segment that can handle the size of your string builder. You have to know that each time the buffer of the string builder is too short, its size is doubled. If you can define (in the ctor) the size of your builder, it's better. You MAY call GC.Collect() when you are done with a large collection of objects.

Actually, when you have an OutOfMemory, it generaly shows a bad design, you may use the hard drive (temp files) instead of memory, you shouldn't allocate memory again and again (try to reuse objects/buffers/...).

I STRONGLY advice you to read this post “Out Of Memory” Does Not Refer to Physical Memory from Eric Lippert.

Answer 3

Try to reuse StringBuilder object when you do data generation.

After or before use just reset the size of the StringBuilder to 0 and start appending. This will decrease number of allocations and possibly make OutOfMemory situation very rare.

To illustrate my point:

void MainProgram()
{
    StringBuilder builder = new StringBuilder(2 * 1024); //2 Kb

    PerformOperation(builder);
    PerformOperation(builder);
    PerformOperation(builder);
    PerformOperation(builder);
}

void PerformOperation(StringBuilder builder)
{
    builder.Length = 0;

    //
    // do the work here builder.Append(...);
    //
}