glibc application holding onto unused memory until

Question

I have a C++ application (gcc 4.9.1, glibc 2.17) running on linux (Centos 7). It uses various third-party libraries, notably Boost 1.61. As the application runs, I can watch its memory usage increasing steadily via htop's VIRT and RES columns, or the ps command, etc. If I let it go long enough, it will use enormous amounts of that memory and swamp the box.

Sounds like a leak, but it passes valgrind with only a few bytes leaked, all in places I'd expect. Debug print messages indicate program flow is as expected.

Digging further via the debugger, I find that most of this memory is still in use when __run_exit_handlers is invoked at the end of main. I can step through various calls to free as it works through the global destructor chain. After it finishes those, I observe only a minimal downward change in the apparent memory usage. Then, finally it calls _exit(), and only then is the memory restored to the operating system, all at once.

Can anyone offer me additional tips on how to proceed debugging this? Why won't my program give that memory back?

Answer 1

Everything here is based on GNU libc implementation of malloc running on Linux.

Test program below does not seem to give away any memory to the system after freeing memory (strace does not show sbrk calls that return memory back to the kernel):

int main()
{
    static const int N = 5000000;
    static void *arr[N];

    for (int i = 0; i < N; i++)
        arr[i] = std::malloc(1024);

    // reverse to simplify allocators job
    for (int i = N - 1; i >= 0; i--)
        std::free(arr[i]);
}

Looks like glibc does not give away memory back at all. According to mallopt(3) man page, parameter M_TRIM_THRESHOLD is responsible for giving away memory. By default it is 128kb, while test program allocates and frees 5 GB of memory. Looks like some other details of malloc implementation do not let it free memory.

At the moment I can recommend following solutions:

• If you can, try calling malloc_trim once in a while or after freeing a lot of memory. This should force trimming and should give memory back to OS using MADV_DONTNEED.
• Avoid allocating large number of small objects using malloc or operator new, instead allocate them from a memory pool of a size greater than M_MMAP_THRESHOLD. Try destroing that pool afterwards if program logic allows this. Memory chunks of size greater than M_MMAP_THRESHOLD are immediately released back to OS.
• Same as previous one, but should be faster: allocate memory pools for small objects using mmap and release memory back to OS using madvise and MADV_DONTNEED/MADV_FREE.
• Try using another allocator that might take advantage of MADV_FREE to return memory back to the system (jemalloc?).

---

I have found this old (2006) ticket on glibc's bugzilla. It says there that free never returns memory back to the kernel, unless malloc_trim is called.

Newer versions of free seem to have code that executes internal systrim function that should trim top of the arena, but I wasn't able to make it work.

Answer 2

You can profile your memory allocation using valgrind --tool=massif ./executable

Check out the documentation at http://valgrind.org/docs/manual/ms-manual.html

Then once you have profiling data you can apply memory pools and other techniques. Since you already use Boost you can find several such tools in Boost.