Memory leak in C++ Threads, Used VALGRIND to debug

Question

I have a thread system that keeps allways opened a certain amount of threads. The problem with this program si that it is having some memory leak problems. I am starting the program at around 8MB (RAM memory) and in 1 day i see it at 800MB. I used valgrind to track down this issue and i found the following while running this command time valgrind --leak-check=yes ./test2; :

I get this when i use 1000 treads and it is not even finishing.

Thread 157: status = VgTs_WaitSys
==16172==    at 0x4160F8B: ??? (in /lib/libc-2.5.so)
==16172==    by 0x40FF58F: _IO_file_underflow@@GLIBC_2.1 (in /lib/libc-2.5.so)
==16172==    by 0x40FFC8A: _IO_default_uflow (in /lib/libc-2.5.so)
==16172==    by 0x410100C: __uflow (in /lib/libc-2.5.so)
==16172==    by 0x40F48B5: _IO_getline_info (in /lib/libc-2.5.so)
==16172==    by 0x40F4800: _IO_getline (in /lib/libc-2.5.so)
==16172==    by 0x40F3779: fgets (in /lib/libc-2.5.so)
==16172==    by 0x80496CF: exec(char const*) (in /home/pthread_test)
==16172==    by 0x80498C2: thread_test(void*) (in /home/pthread_test)
==16172==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==16172==    by 0x417047D: clone (in /lib/libc-2.5.so)

Thread 158: status = VgTs_WaitSys
==16172==    at 0x4160F8B: ??? (in /lib/libc-2.5.so)
==16172==    by 0x40FF58F: _IO_file_underflow@@GLIBC_2.1 (in /lib/libc-2.5.so)
==16172==    by 0x40FFC8A: _IO_default_uflow (in /lib/libc-2.5.so)
==16172==    by 0x410100C: __uflow (in /lib/libc-2.5.so)
==16172==    by 0x40F48B5: _IO_getline_info (in /lib/libc-2.5.so)
==16172==    by 0x40F4800: _IO_getline (in /lib/libc-2.5.so)
==16172==    by 0x40F3779: fgets (in /lib/libc-2.5.so)
==16172==    by 0x80496CF: exec(char const*) (in /home/pthread_test)
==16172==    by 0x80498C2: thread_test(void*) (in /home/pthread_test)
==16172==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==16172==    by 0x417047D: clone (in /lib/libc-2.5.so)

When using just ./test2 (running without valgrind), absolutely all threads finishes and i can see in the debug.log that every thread finishes

tail debug.log
END:1972:191
END:1984:191
END:1992:191
END:1974:191
END:1986:191
END:1979:191
END:1993:191
END:1983:191
END:1999:191 <-- even thread 1999 finished
END:1991:191

When i am doing with 250 threads i get this at the end of the program

THREAD [1993] ending
THREAD [1983] ending
THREAD [1999] ending
THREAD [1991] ending
==23152== Thread 25:
==23152== Invalid free() / delete / delete[]
==23152==    at 0x400579D: free (vg_replace_malloc.c:325)
==23152==    by 0x3B59DD: ??? (in /lib/libc-2.5.so)
==23152==    by 0x3B5556: ??? (in /lib/libc-2.5.so)
==23152==    by 0x40013B2: _vgnU_freeres (vg_preloaded.c:62)
==23152==    by 0x333A73: _Exit (in /lib/libc-2.5.so)
==23152==    by 0x40AAF8: start_thread (in /lib/libpthread-2.5.so)
==23152==    by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==  Address 0x409b2c8 is not stack'd, malloc'd or (recently) free'd
==23152==
==23152==
==23152== HEAP SUMMARY:
==23152==     in use at exit: 160,850 bytes in 12,002 blocks
==23152==   total heap usage: 20,151 allocs, 8,150 frees, 2,640,139 bytes allocated
==23152==
==23152== Thread 1:
==23152== 152 bytes in 1 blocks are possibly lost in loss record 2 of 8
==23152==    at 0x4004EC2: calloc (vg_replace_malloc.c:418)
==23152==    by 0x291FF9: _dl_allocate_tls (in /lib/ld-2.5.so)
==23152==    by 0x40B026: pthread_create@@GLIBC_2.1 (in /lib/libpthread-2.5.so)
==23152==    by 0x80493DF: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 3 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x804981D: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152==    by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 4 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x804944D: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== 20,000 bytes in 2,000 blocks are definitely lost in loss record 5 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C56: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x8049950: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152==    by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 26,890 bytes in 2,000 blocks are definitely lost in loss record 6 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x8049950: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152==    by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 30,890 bytes in 2,000 blocks are definitely lost in loss record 7 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x804981D: thread_test(void*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x40A911: start_thread (in /lib/libpthread-2.5.so)
==23152==    by 0x37547D: clone (in /lib/libc-2.5.so)
==23152==
==23152== 42,890 bytes in 2,000 blocks are definitely lost in loss record 8 of 8
==23152==    at 0x4005B83: malloc (vg_replace_malloc.c:195)
==23152==    by 0x3105DF: strdup (in /lib/libc-2.5.so)
==23152==    by 0x8049C64: file_put_contents(char*, char*, char*) (in /home/freelancer1/pthreads_test/test2)
==23152==    by 0x804944D: main (in /home/freelancer1/pthreads_test/test2)
==23152==
==23152== LEAK SUMMARY:
==23152==    definitely lost: 160,670 bytes in 12,000 blocks
==23152==    indirectly lost: 0 bytes in 0 blocks
==23152==      possibly lost: 152 bytes in 1 blocks
==23152==    still reachable: 28 bytes in 1 blocks
==23152==         suppressed: 0 bytes in 0 blocks
==23152== Reachable blocks (those to which a pointer was found) are not shown.
==23152== To see them, rerun with: --leak-check=full --show-reachable=yes
==23152==
==23152== For counts of detected and suppressed errors, rerun with: -v
==23152== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 36 from 8)

Here is the source code :

#include <iostream>
#include <cstdlib>
#include <pthread.h>

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <cstring>
#include <stdexcept>

using namespace std;

void file_put_contents(char* filename, char* content, char* mode)
{
    char* filename2;
    filename2 = strdup(filename);

    char* content2;
    content2 = strdup(content);

    char* writemode = "a";
    if(strstr(mode,"FILE_APPEND"))
    {
        writemode = "a";
    }
    else if(strstr(mode,"FILE_OVERWRITE"))
    {
        writemode = "w";
    }
    else
    {
        writemode = "w";
    }

    FILE * file;
    file = fopen(filename2,writemode);
    fprintf(file,content2);
    fclose(file);
}

int NUM_THREADS = 2000;
int MAX_THREADS = 500;
int THREADSTACK = 65536;

struct thread_struct{
    int arg1;
    int arg2;
};

pthread_mutex_t mutex_;
pthread_mutex_t mutex2_;
static unsigned int thread_count = 0;

string exec(const char* cmd)
{
    int DEBUG=0;

    char buffer[5000];
    string result = "";
    FILE* pipe = popen(cmd, "r");
    if (!pipe && DEBUG) throw runtime_error("popen() failed!");
    try 
    {
        while (!feof(pipe)) 
        {
            if (fgets(buffer, 128, pipe) != NULL)
            {
                result += buffer;
            }

        }
    }
    catch(...)
    {
        pclose(pipe);
        throw;
    }

    pclose(pipe);
    return result;
}


void *thread_test(void *arguments)
{
    pthread_mutex_lock(&mutex_);
    thread_count++;
    pthread_mutex_unlock(&mutex_);

    struct thread_struct *args = (thread_struct*)arguments;

    int thread_id = (int) args->arg1;

    char debug_var[100];

    // UGLY DEBUG
    memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"BEGIN:%d:%d\n",thread_id,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");

    int random_sleep;
    random_sleep = rand() % 10 + 3;
    char command[100];
    memset(command,0,sizeof(command));
    sprintf(command,"sleep %d",random_sleep);
    exec(command);

    pthread_mutex_lock(&mutex_);
    thread_count--;
    pthread_mutex_unlock(&mutex_);

    // UGLY DEBUG
    memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"END:%d:%d\n",thread_id,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");

    printf("THREAD [%d] ending\n",thread_id);

    pthread_exit(NULL);
}



int main()
{
    // pthread_t threads[NUM_THREADS];
    int rc;

    printf("Beginning ...");
    usleep(1000000);

    srand ((unsigned)time(NULL));
    unsigned int thread_count_now = 0;

    pthread_attr_t  attrs;
    pthread_attr_init(&attrs);
    pthread_attr_setstacksize(&attrs, THREADSTACK);
    pthread_attr_setdetachstate(&attrs, PTHREAD_CREATE_DETACHED);

    pthread_mutex_init(&mutex2_, NULL);

    for(int i=0; i < NUM_THREADS; i++ )
    {
        create_thread:
        pthread_mutex_lock(&mutex_);
        thread_count_now = thread_count;
        pthread_mutex_unlock(&mutex_);

        struct thread_struct struct1;
        struct1.arg1 = i;
        struct1.arg2 = 999;

        pthread_t temp_thread;

        printf("CREATE thread [%d]\n",i);
        rc = pthread_create(&temp_thread, &attrs, &thread_test, (void *)&struct1);

        char debug_var[100];
        memset(debug_var,0,sizeof(debug_var)); sprintf(debug_var,"THREAD:%d:RC=%d:%d\n",i,rc,__LINE__); file_put_contents("debug.log",debug_var,"FILE_APPEND");

        if (rc)
        {
            printf("Unable to create thread %d\n",rc);

            sleep(1);
            pthread_detach(temp_thread);
            goto create_thread;
        }   

        usleep(10);

        while(true)
        {
            if(thread_count >= MAX_THREADS)
            {
                // printf("Thread POOL full %d of %d\n",thread_count,MAX_THREADS);
            }
            else
            {
                break;
            }

            usleep(10);
        }
    }

    pthread_attr_destroy(&attrs);

    pthread_mutex_destroy(&mutex2_);

    printf("Proccess completed!\n");

    pthread_exit(NULL);

    return 1;
}

Did i forgot somewhere to free memory?

I don't quite understand those logs perfectly, can you give me any clue? What is wrong? What do i have to change?

Thank you.