Moving stack and reserving half of addressable spa

For the implementation of a memory monitoring library, I would like to move the stack of the monitored program, and reserve the upper half of the virtual memory for use in the monitoring library, following the "half'n'half"-model describe used by tools such as TaintTrace, LIFT and Hobbes. However, it does not seem clear to me how to actually do this. Just mmaping the memory required fails (cannot allocate memory). Do I need to modify crt? The linker?

I'm not too sure how you would allocate tainted and untainted memory and optimize from within a Unix like operating system.

Memory allocation in a Unix system is done using brk() and sbrk(). There is not much more to it than that. For other memory spaces you need to use shared memory allocations (you can make them read/write only by your process so it is "shared" just between you.)

The problem you'll run into is the growing memory allocations. I'm not too sure, but from what I remember from using shared memory, you could not easily extend the allocated buffer. But newer interfaces may work well for you.

--- Update:

Reading the malloc() documentation, I found the following:

Normally, malloc() allocates memory from the heap, and adjusts the size of the heap as required, using sbrk(2). When allocating blocks of memory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation allocates the memory as a private anonymous mapping using mmap(2). MMAP_THRESHOLD is 128 kB by default, but is adjustable using mallopt(3). Allocations performed using mmap(2) are unaffected by the RLIMIT_DATA resource limit (see getrlimit(2)).

Which means the OS already makes use of mmap() to allocate buffers larger than 128Kb by default. Also the standard malloc() is limited by the getrlimit() of RLIMIT_DATA documented here:

The maximum size of the process's data segment (initialized data, uninitialized data, and heap). This limit affects calls to brk(2) and sbrk(2), which fail with the error ENOMEM upon encountering the soft limit of this resource.

However, as the malloc() says, if you allocate large buffers (128Kb+) then you are not affected by that limit.

That being said, a lot of the memory is allocated for things such as the kernel itself, your program, the I/O mapped to addresses (DMA), interrupt table, memory tables (MMU management), and shared memory buffers (i.e. mmap() using SHARED will allocate memory addresses that all processes must have access to even if you're not using those yet, they get reserved, in case you were to access them.)

Once of those things that makes use of shared memory are shared libraries. (the binaries are loaded in shared memory so once for all the processes running against said shared libraries.)