Excessive synchronizing of memory mapped file in A

Question

I am currently working on an Apache module that uses a large mmap file to share data between processes. This is created on start-up and removed when the server shuts down (May choose to keep it at a later stage). I have implemented this using the Apache APR libraries, and it works well, at least for smaller files.

When the size of the memory mapped file however increases (there is still enough RAM to cache it when the server is running) the system at times virtually grinds to a halt as it seems the operating system (Linux in my case) consumes a lot of resources synchronizing the file with the disk. Is there any way to control/reduce this synchronization?

Since I at the moment do not need the contents synchronized to disk I should probably have used a shared memory segment instead, and will try that instead. I am however still interested in ways to control synchronization of memory mapped files.

Answer 1

Memory-mapped files are explicitly synchronized with msync(), though the OS can flush the memory's contents in the event of a page fault. So a couple of things to look for:

1. Does the Apache APR library's source code have a lot of msync()'s?
2. Is your application jumping around in memory? Ie, are you accessing non-sequintial memory? This often happens with improper data structures or when nested loop indices don't match multidimensional array indices.

You can try profiling your code to see if there are inefficiencies that would lead to an unexpected number of page faults.

Answer 2

Writing to your mapped file is creating dirty pages - memory pages that need, at some point, to be written out to their backing store (in this case, your on-disk file).

The writeout of dirty pages is tuneable by some knobs in /proc/sys/vm/. In particular, if the amount of dirty data in your mapped file is typically large compared to your total system memory, you may want to increase dirty_ratio significantly (say, to 60), and dirty_background_ratio a little (say, to 30).

You may also want to increase dirty_expire_centisecs, but the default is already quite long (5 minutes) (this is the age that dirty data must reach before it becomes eligible for writeout).

It is worth pointing out that switching to shared memory is simply a matter of creating your mapped file in /dev/shm/, assuming your distribution mounts a tmpfs there.