Is there a memory leak in the ConcurrentBag imp

Question

Possible Duplicate:
Possible memoryleak in ConcurrentBag?

Edit1:

The actual question is. Can you confirm this or is my sample wrong and I am missing somthing obvious?

I have thought that ConcurrentBag is simpy a replacement for an unorderd list. But I was wrong. ConcurrentBag does add itself to as ThreadLocal to the creating thread which does basically cause a memory leak.

   class Program
    {
        static void Main(string[] args)
        {
            var start = GC.GetTotalMemory(true);
            new Program().Start(args);
            Console.WriteLine("Diff: {0:N0} bytes", GC.GetTotalMemory(true) - start);
            GC.Collect();
            GC.WaitForPendingFinalizers();
            GC.Collect();
            Thread.Sleep(5000);
        }

        private void Start(string[] args)
        {
            for (int i = 0; i < 1000; i++)
            { 
                var bag = new ConcurrentBag<byte>();
                bag.Add(1);
                byte by;
                while (bag.TryTake(out by)) ;
            }
        }

I can make Diff 250 KB or 100 GB depending on how much data I add to the bags. The data nor the bags go away.

When I break into this with Windbg and I do a !DumpHeap -type Concurrent

....

000007ff00046858        1           24 System.Threading.ThreadLocal`1+GenericHolder`3[[System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]], System],[System.Threading.ThreadLocal`1+C0[[System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]], System]], mscorlib],[System.Threading.ThreadLocal`1+C0[[System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]], System]], mscorlib],[System.Threading.ThreadLocal`1+C0[[System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]], System]], mscorlib]]
000007feed812648        2           64 System.Collections.Concurrent.ConcurrentStack`1[[System.Int32, mscorlib]]
000007feece41528        1          112 System.Collections.Concurrent.CDSCollectionETWBCLProvider
000007ff000469e0     1000        32000 System.Threading.ThreadLocal`1+Boxed[[System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]], System]]
000007feed815900     1000        32000 System.Collections.Concurrent.ConcurrentStack`1+Node[[System.Int32, mscorlib]]
000007ff00045530     1000        72000 System.Collections.Concurrent.ConcurrentBag`1+ThreadLocalList[[System.Byte, mscorlib]]

When I do create an empty ConcurrentBag to let some worker threads add data to it ConcurrentBag and its data will be there as long as the creating thread is still alive.

This way I got a several GB memory leak. I did "fix" this by using a List and locks. ConcurrentBag may be fast but it is useless as simple replacement for a List with the same object lifetime.

If I ever create a ConcurrentBag on the main thread I will keep it as long as the thread is alive. This is not something I would expect and it can cause major pain.

Answer 1

From the documentation

ConcurrentBag is a thread-safe bag implementation, optimized for scenarios where the same thread will be both producing and consuming data stored in the bag.

and from When to use a thread-safe collection

In mixed producer-consumer scenarios, ConcurrentBag is generally much faster and more scalable than any other concurrent collection type for both large and small workloads.

I'd say that your assumptions about ConcurrentBag are incorrect. First, it doesn't add itsels to ThreadLocal, it uses thread local storage to provide separate internal lists for each thread that access it. It is more than just a thread-safe unordered list.

What you consider a memory leak is actually the expected behavior once you realize that the bag uses TLS - there is no need to clear the data as long as the thread is in use.

Having said all that, I hadn't realized the extra functionality of ConcurrentBag myself until just now.

I've found a very good description of how ConcurrentBag uses separate lists and the cost of its methods in different scenarions in "What is ConcurrentBag". I do wish this description appeared in the MSDN documentation.

Personally, I'll start using ConcurrentBag a lot more now that I know of its special behavior.

UPDATE:

Just checked this post by Ayende saying that "ThreadLocal, which ConcurrentBag uses, didn’t expect to have a lot of instances. That has been fixed, and now can run fairly fast"

Answer 2

You are right that ConcurrentBag creates a ThreadLocal copy, in fact they are optimized for scenarios where the same thread is reading and writing the data to the bag: "... ConcurrentBag is a thread-safe bag implementation, optimized for scenarios where the same thread will be both producing and consuming data stored in the bag."

On the otherhand, I do not see a strange behaviour here; the thread lives and the concurrent bag lives. When thread finishes GC will do it's job.

Answer 3

Why not move the Console.WriteLine after the second GC.Collect()? Otherwise you may be looking at some more objects than you intended.

You can also try putting everything in your Main into a loop to get a few stats. Even if you were to not move your write you probably would see smaller deltas afterwards.

Cheers!