RAM drive for compiling - is there such a thing?

Question

An answer (see below) to one of the questions right here on Stack Overflow gave me an idea for a great little piece of software that could be invaluable to coders everywhere.

I'm imagining RAM drive software, but with one crucial difference - it would mirror a real folder on my hard drive. More specifically - the folder which contains the project I'm currently working on. This way any builds would be nearly instantaneous (or at least a couple orders of magnitude faster). The RAM drive would synchronize its contents with the hard disk drive in background using only idle resources.

A quick Google search revealed nothing, but perhaps I just don't know how to Google. Perhaps someone knows of such a software? Preferably free, but reasonable fees might be OK too.

Added: Some solutions have been suggested which I discarded in the very beginning. They would be (in no particular order):

• Buy a faster hard disk drive (SSD maybe or 10K RPM). I don't want a hardware solution. Not only software has the potential to be cheaper (freeware, anyone?), but it can also be used in environments where hardware modifications would be unwelcome if not impossible - say, at the office.
• Let OS/HDD do the caching - it knows better how to use your free RAM. The OS/HDD have generic cache algorithms that cache everything and try to predict which data will be most needed in the future. They have no idea that for me the priority is my project folder. And as we all know quite well - they don't really cache it much anyway. ;)
• There are plenty of RAM drives around; use one of those. Sorry, that would be reckless. I need my data to be synchronized back to the HDD whenever there is a bit of free time. In the case of a power failure I could bear losing the last five minutes of work, but not everything since my last checkin.

Added 2: An idea that came up - use a normal RAM drive plus a background folder synchronizer (but I do mean background). Is there any such thing?

Added 3: Interesting. I just tried out a simple RAM drive at work. The rebuild time drops from ~14 secs to ~7 secs (not bad), but incremental build is still at ~5 secs - just like on the HDD. Any ideas why? It uses aspnet_compiler and aspnet_merge. Perhaps they do something with other temp files elsewhere?

Added 4: Oh, nice new set of answers! :) OK, I've got a bit more info for all you naysayers. :)

One of the main reasons for this idea is not the above-mentioned software (14 secs build time), but another one that I didn't have access at the time. This other application has a 100 MB code base, and its full build takes about 5 minutes. Ah yes, it's in Delphi 5, so the compiler isn't too advanced. :) Putting the source on a RAM drive resulted in a BIG difference. I got it below a minute, I think. I haven't measured. So for all those who say that the OS can cache stuff better - I'd beg to differ.

Related Question:

RAM disk for speed up IDE

Note on first link: The question to which it links has been deleted because it was a duplicate. It asked:

What do you do while your code’s compiling?

And the answer by Dmitri Nesteruk to which I linked was:

I compile almost instantly. Partly due to my projects being small, partly due to the use of RAM disks.

Answer 1

The disk slowdown you incur is mainly write, and also possibly due to virus scanners. It can vary greatly between OSes too.

With the idea that writes are slowest, I would be tempted to setup a build where intermediate (for example, .o files) and binaries get output to a different location such as a RAM drive.

You could then link this bin/intermediate folder to faster media (using a symbolic link or NTFS junction point).

Answer 2

My final solution to the problem is vmtouch: https://hoytech.com/vmtouch/ This tool locks the current folder into (ram) cache and vmtouch daemonizes into background.

sudo vmtouch -d -L ./

Put this in shell rc for fast access:

alias cacheThis = 'sudo vmtouch -d -L ./'

I searched for a ready made script for quite a while, because I didn't want to waste a lot of time on writing my own ramdisk-rsync-script. I'm sure I would have missed some edge cases, which would be quite unpleasant if important code was involved. And I never liked the polling approach.

Vmtouch seems like the perfect solution. In addition it doesn't waste memory like a fixed size ramdisk does. I didn't do a benchmark, because 90% of my 1Gig source+build folder were already cached, but at least it feels faster ;)

Answer 3

In Linux (you never mentioned which OS you're on, so this could be relevant) you can create block devices from RAM and mount them like any other block device (that is, a HDD).

You can then create scripts that copy to and from that drive on start-up / shutdown, as well as periodically.

For example, you could set it up so you had ~/code and ~/code-real. Your RAM block gets mounted at ~/code on startup, and then everything from ~/code-real (which is on your standard hard drive) gets copied over. On shutdown everything would be copied (rsync'd would be faster) back from ~/code to ~/code-real. You would also probably want that script to run periodically, so you didn't lose much work in the event of a power failure, etc.

I don't do this anymore (I used it for Opera when the 9.5 beta was slow, no need anymore).

Here is how to create a RAM disk in Linux.

Answer 4

Use https://wiki.archlinux.org/index.php/Ramdisk to make the RAM disk.

Then I wrote these scripts to move directories to and from the RAM disk. Backup is made in a tar file before moving into the RAM disk. The benefit of doing it this way is that the path stays the same, so all your configuration files don't need to change. When you are done, use uramdir to bring back to disk.

Edit: Added C code that will run any command it is given on an interval in background. I am sending it tar with --update to update the archive if any changes.

I believe this general-purpose solution beats making a unique solution to something very simple. KISS

Make sure you change path to rdbackupd

ramdir

#!/bin/bash

# May need some error checking for bad input.

# Convert relative path to absolute
# /bin/pwd gets real path without symbolic link on my system and pwd
# keeps symbolic link. You may need to change it to suit your needs.
somedir=`cd $1; /bin/pwd`;
somedirparent=`dirname $somedir`

# Backup directory
/bin/tar cf $somedir.tar $somedir

# Copy, tried move like https://wiki.archlinux.org/index.php/Ramdisk
# suggests, but I got an error.
mkdir -p /mnt/ramdisk$somedir
/bin/cp -r  $somedir /mnt/ramdisk$somedirparent

# Remove  directory
/bin/rm -r $somedir

# Create symbolic link. It needs to be in parent of given folder.
/bin/ln -s /mnt/ramdisk$somedir $somedirparent

#Run updater
~/bin/rdbackupd "/bin/tar -uf $somedir.tar $somedir" &

uramdir

#!/bin/bash

#Convert relative path to absolute
#somepath would probably make more sense
# pwd and not /bin/pwd so we get a symbolic path.
somedir=`cd $1; pwd`;

# Remove symbolic link
rm $somedir

# Copy dir back
/bin/cp -r /mnt/ramdisk$somedir $somedir

# Remove from ramdisk
/bin/rm -r /mnt/ramdisk$somedir

# Stop
killall rdbackupd

rdbackupd.cpp

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <signal.h>
#include <sys/time.h>

struct itimerval it;
char* command;

void update_archive(int sig)
{
    system(command);
}

int main(int argc, char**argv)
{
    it.it_value.tv_sec     = 1;   // Start right now
    it.it_value.tv_usec    = 0;
    it.it_interval.tv_sec  = 60;  // Run every 60 seconds
    it.it_interval.tv_usec = 0;

    if (argc < 2)
    {
        printf("rdbackupd: Need command to run\n");
        return 1;
    }
    command = argv[1];

    signal(SIGALRM, update_archive);
    setitimer(ITIMER_REAL, &it, NULL); // Start

    while(true);

    return 0;
}

Answer 5

This sounds like disk caching which your operating system and / or your hard drive will handle for you automatically (to varying degrees of performance, admittedly).

My advice is, if you don't like the speed of your drive, buy a high speed drive purely for compiling purposes. Less labor on your part and you might have the solution to your compiling woes.

Since this question was originally asked, spinning hard disks have become miserable tortoises when compared to SSDs. They are very close to the originally requested RAM disk in a SKU that you can purchase from Newegg or Amazon.

Answer 6

Just as James Curran says, the fact that most programs follow the law of locality of references, the frequent code and data page count will be narrowed over time to a manageable size by the OS disk cache.

RAM disks were useful when operating systems were built with limitations such as stupid caches (Win 3.x, Win 95, DOS). The RAM disk advantage is near zero and if you assign a lot of RAM it will suck memory available to the system cache manager, hurting overall system performance. The rule of thumb is: let your kernel to do that. This is the same as the "memory defragmentation" or "optimizers" programs: they actually force pages out of cache (so you get more RAM eventually), but causing the system to do a lot of page-faulting over time when your loaded programs begin to ask for code/data that was paged out.

So for more performance, get a fast disk I/O hardware subsystem, maybe RAID, faster CPU, better chipset (no VIA!), more physical RAM, etc.