Reducing repository size in Mercurial

Question

As my team works on a given project with the source in a Mercurial repository, the repository is obviously growing in size. As such, cloning a repository over the network becomes slower and slower.

Are there any techniques that are used for pruning out older commits or reducing the size of the repo to make the cloning operation faster over a slow network?

(We are using TortoiseHg as the Mercurial client but that (I'm guessing) shouldn't make a difference to the solution to this problem.)

Answer 1

You can use a dedicated clone of the remote repo on your computer as a cache for clone operations. So you don't need to transfer the whole repo over the net every time, but only the parts which are not already there.

Answer 2

If you only need the files in a given revision, but never need to examine history or make new commits, then downloading a snapshot can be faster.

The normal hgweb CGI script can provide a zip or tar file for any revision. The archives are generated on the fly. You only need to add

[web]
allow_archive = gz, zip, bz2

to your configuration file. You can then find archives under URLs like

http://server.com/repo/archive/rev.zip

Replace the revision number with the branch name it changeset hash you want Download the file with wget, curl, or a similar tool.

This strategy only pays off when the history is very large compared to the size of a single changeset.

This can be the case if the repository contains large files that change often. The largefiles extension can be an alternative here: it allows you to only download the files needed for the revision you checkout. That way you avoid downloading the history for big files and save significant amounts of bandwidth.

Answer 3

One option is to use the convert extension to decompose your repository to a group of smaller repositories.

Say you have a repository that has evolved to contain many projects (folders). And you have decided you’d be better off if each project (folder) were a separate repository. You can use the convert extension to do just that and retain your changeset history.

Answer 4

Sometimes problems like this can be caused if you have large binary files in your repository. Any updates to them tend to cause large diffs and make the size to go up more drastically than normal.

If this applies to you, it may be worth looking at the Large-files extension distributed with Mercurial 2.0. I've not used it personally, and it sounds like it still has some rouge edges, but if includes a command lfconvert which will convert the repo for you. You could then try it to see if it clones any quicker.