Malloc Memory Questions

Question

First of all I noticed when I malloc memory vs. calloc the memory footprint is different. I am working with datasets of several GB. It is ok for this data to be random.

I expected that I could just malloc a large amount of memory and read whatever random data was in it cast to a float. However, looking at the memory footprint in the process viewer the memory is obviously not being claimed (vs. calloc where I see a large foot print). I ran a loop to write data into the memory and then I saw the memory footprint climb. Am I correct in saying that the memory isn't actually claimed until I initialize it?

Finally after I passed 1024*1024*128 bytes (1024 MB in the process viewer) I started getting segfaults. Calloc however seems to initialize the full amount up to 1 GB. Why do I get segfaults when initializing memory in a for loop with malloc at this number 128MB and why does the memory footprint show 1024MB?

If malloc a large amount from memory and then read from it what am I getting (since the process viewer shows almost no footprint until I initialize it)?

Finally is there any way for me to alloc more than 4GB? I am testing memory hierarchy performance.

Code for #2:

    long long int i;
    long long int *test=(long long int*)malloc(1024*1024*1024);
    for (i=0;i<1024*1024*128;i++)
            test[i]=i;

    sleep(15);

Answer 1

1- If you are working on a 32-bit machine you can't have a variable with more than 2GBs allocated to it.

2- If you are working on a 64-bit machine you can allocate as much as RAM+Swap memory in total, however, allocating all for one variable requires a big consequent chunk of memory which might not be available. Try it with a linked list, where each element has only 1 MB assigned and you can achieve a higher memory allocated in total.

3- As noted by you and Sharth, unless you use your memory, linux won't allocate it.

Answer 2

Your #2 is failing with a segfault either because sizeof(long long int) > 8 or because your malloc returned NULL. That is very possible if you are requesting 1 GB of RAM.

More info on #2. From your 128 MB comment I get the idea that you may not realize what's happening. Because you declare the array pointer as long long int the size of each array element is 8 bytes. 1024/8 == 128 so that is why your loop works. It did when I tried it, anyway.

Answer 3

Your for loop in your example code is actually touching 1GB of memory, since it is indexing 128*1024*1024 long longs, and each long long is 8 bytes.

Answer 4

Some notes:

1. As the comments note, Linux doesn't actually allocate your memory until you use it.
2. When you use calloc instead of malloc, it zeroes out all the memory you requested. This is equivalent to using it.