RAM drive for compiling - is there such a thing?

Question

An answer (see below) to one of the questions right here on Stack Overflow gave me an idea for a great little piece of software that could be invaluable to coders everywhere.

I'm imagining RAM drive software, but with one crucial difference - it would mirror a real folder on my hard drive. More specifically - the folder which contains the project I'm currently working on. This way any builds would be nearly instantaneous (or at least a couple orders of magnitude faster). The RAM drive would synchronize its contents with the hard disk drive in background using only idle resources.

A quick Google search revealed nothing, but perhaps I just don't know how to Google. Perhaps someone knows of such a software? Preferably free, but reasonable fees might be OK too.

Added: Some solutions have been suggested which I discarded in the very beginning. They would be (in no particular order):

• Buy a faster hard disk drive (SSD maybe or 10K RPM). I don't want a hardware solution. Not only software has the potential to be cheaper (freeware, anyone?), but it can also be used in environments where hardware modifications would be unwelcome if not impossible - say, at the office.
• Let OS/HDD do the caching - it knows better how to use your free RAM. The OS/HDD have generic cache algorithms that cache everything and try to predict which data will be most needed in the future. They have no idea that for me the priority is my project folder. And as we all know quite well - they don't really cache it much anyway. ;)
• There are plenty of RAM drives around; use one of those. Sorry, that would be reckless. I need my data to be synchronized back to the HDD whenever there is a bit of free time. In the case of a power failure I could bear losing the last five minutes of work, but not everything since my last checkin.

Added 2: An idea that came up - use a normal RAM drive plus a background folder synchronizer (but I do mean background). Is there any such thing?

Added 3: Interesting. I just tried out a simple RAM drive at work. The rebuild time drops from ~14 secs to ~7 secs (not bad), but incremental build is still at ~5 secs - just like on the HDD. Any ideas why? It uses aspnet_compiler and aspnet_merge. Perhaps they do something with other temp files elsewhere?

Added 4: Oh, nice new set of answers! :) OK, I've got a bit more info for all you naysayers. :)

One of the main reasons for this idea is not the above-mentioned software (14 secs build time), but another one that I didn't have access at the time. This other application has a 100 MB code base, and its full build takes about 5 minutes. Ah yes, it's in Delphi 5, so the compiler isn't too advanced. :) Putting the source on a RAM drive resulted in a BIG difference. I got it below a minute, I think. I haven't measured. So for all those who say that the OS can cache stuff better - I'd beg to differ.

Related Question:

RAM disk for speed up IDE

Note on first link: The question to which it links has been deleted because it was a duplicate. It asked:

What do you do while your code’s compiling?

And the answer by Dmitri Nesteruk to which I linked was:

I compile almost instantly. Partly due to my projects being small, partly due to the use of RAM disks.

Answer 1

I'm surprised at how many people suggest that the OS can do a better job at figuring out your caching needs than you can in this specialized case. While I didn't do this for compiling, I did do it for similar processes and I ended up using a RAM disk with scripts that automated the synchronization.

In this case, I think I'd go with a modern source control system. At every compile it would check in the source code (along an experimental branch if needed) automatically so that every compile would result in the data being saved off.

To start development, start the RAM disk and pull the current base line. Do the editing, compile, edit, compile, etc. - all the while the edits are being saved for you.

Do the final check in when happy, and you don't even have to involve your regular hard disk drive.

But there are background synchronizers that will automate things - the issue is that they won't be optimized for programming either and may need to do full directory and file scans occasionally to catch changes. A source code control system is designed for exactly this purpose, so it would likely be lower overhead even though it exists in your build setup.

Keep in mind that a background sync task, in the case of a power outage, is undefined. You would end up having to figure out what was saved and what wasn't saved if things went wrong. With a defined save point (at each compile, or forced by hand) you'd have a pretty good idea that it was at least in a state where you thought you could compile it. Use a VCS and you can easily compare it to the previous code and see what changes you've applied already.

Answer 2

I wonder if you could build something like a software RAID 1 where you have a physical disk/partition as a member, and a chunk of RAM as a member.

I bet with a bit of tweaking and some really weird configuration one could get Linux to do this. I am not convinced that it would be worth the effort though.

Answer 3

I don't have exactly what you're looking for, but I'm now using a combination of Ramdisk and DRAM ramdisk. Since this is Windows, I have a hard 3 GB limit for core memory, meaning I cannot use too much memory for a RAM disk. 4 GB extra on the 9010 really rocks it. I let my IDE store all its temporary stuff on the solid state RAM disk and also the Maven repository. The DRAM RAM disk has a battery backup to the flash card. This sounds like an advertisement, but it really is an excellent setup.

The DRAM disk has double SATA-300 ports and comes out with 0.0 ms average seek on most tests ;) Something for the Christmas stocking?

Answer 4

We used to do this years ago for a 4GL macro-compiler; if you put the macro library and support libraries and your code on a RAM disk, compiling an application (on an 80286) would go from 20 minutes to 30 seconds.

Answer 5

Your OS will cache things in memory as it works. A RAM disk might seem faster, but that's because you aren't factoring in the "copy to RAMDisk" and "copy from RAMDisk" times. Dedicating RAM to a fixed size ramdisk just reduces the memory available for caching. The OS knows better what needs to be in RAM.

Answer 6

See Speeding up emerge with tmpfs (Gentoo Linux wiki).

Speeding up compiles using RAM drives under Gentoo was the subject of a how-to written many eons ago. It provides a concrete example of what has been done. The gist is that all source and build intermediate file are redirected to a RAM disk for compile, while final binaries are directed to the hard drive for install.

Also, I recommend exploring maintaining your source on hard drive, but git push your latest source changes to a clone respository that resides on the RAM disk. Compile the clone. Use your favorite script to copy the binaries created.

I hope that helps.