I believe it's a general issue but if it helps I'm running LucidDB database (0.9.4) on Mac OS X 10.8 with 16GB RAM. The database software is written mainly in Java with some parts in C++. When I increased buffer pool size to 1GB I got the following error.
java(669,0xfc621000) malloc: * mmap(size=16777216) failed (error code=12) error: can't allocate region ** set a breakpoint in malloc_error_break to debug
I believe that buffer pool uses shared memory so I increased max available shared memory (kern.sysv.shmmax, kern.sysv.shmall) to 2GB. I also increased java heap size (Xms, Xmx) to 1536MB. I'm stuck, any hints?