Partial clone with Git and Mercurial

Question

Is it possible to clone only one branch (or from a given commit) in Git and Mercurial? I mean, I want to clone a central repo but since it's huge I'd like to only get part of it and still be able to contribute back my changes. Is it possible? Like, I only want from Tag 130 onwards or something like that?

If so, how?

Answer 1

Regarding Git it might be of a historical significance that Linus Torvalds answered this question from the conceptual perspective back in 2007 in a talk that was recorded and is available online.

The question is whether it is possible to check out only some files out of a Git repository.

Tech Talk: Linus Torvalds on git t=43:10

To summarize, he said that one of the design decisions of Git that sets it apart from other source management systems (he cites BitKeeper and SVN) is that Git manages content, not files. The implications being that e.g. a diff of a subset of files in two revisions is computed by first taking the whole diff and then pruning it only to the files that were requested. Another is that you have to check out the whole history; in an all or nothing fashion. For this reason, he suggests splitting loosely related components among multiple repositories and mentions a then ongoing effort to implement an user interface for managing a repository that is structured as a super-project holding smaller repositories.

As far as I know this fundamental design decision still apples today. The super-project thing probably became what now are submodules.

Answer 2

This method creates an unversioned archive without subrepositories:

hg clone -U ssh://machine//directory/path/to/repo/project projecttemp

cd projecttemp

hg archive -r tip ../project-no-subrepos

The unversioned source code without the subrepositoies is in the project-no-subrepos directory

Answer 3

In mercurial, you should be able to so some of this using:

hg convert --banchmap FILE SOURCEDEST REVMAP

You may also want:

--config convert.hg.startrev=REV

The source can be git, mercurial, or a variety of other systems.

I haven't tried it, but convert is quite rich.

Answer 4

In Git land you are talking about three different types of partial clones:

• shallow clones: I want history from revision point X onward.

  Use git clone --depth <n> <url> for that, but please remember that shallow clones are somewhat limited in interacting with other repositories. You would be able to generate patches and send them via email.

• partial clone by filepath: I want all revision history history in some directory /path.

  Not possible in Git. With modern Git though you can have sparse checkout, i.e. you have whole history but you check out (have in working area) only subset of all files.

• cloning only selected branch: I want to clone only one branch (or selected subset of branches).

  Possible, and

  before git 1.7.10 not simple: you would need to do what clone does manually, i.e. git init [<directory>], then git remote add origin <url>, edit .git/config replacing * in remote.origin.fetch by requested branch (probably 'master'), then git fetch .

  as of git 1.7.10 git clone offers the --single-branch option which seems like it was added just for this purpose, and seems pretty easy.

  Note however that because branches usually share most of their history, the gain from cloning only a subset of branches might be smaller than you think.

You can also do a shallow clone of only selected subset of branches.

If you know how people will want to break things down by filepath (multiple projects in the same repository) you can use submodules (sort of like svn:externals) to pre-split the repo into separately cloneable portions.

Answer 5

In mercurial land you're talking about three different types of partial clones:

• shallow clones: I want the history from revision point X onward use the remotefilelog extension
• partial clones by filepath: I want all revision history in directory /path with experimental narrowhg extension or I want only files in directory /path to be in my working directory with experimental sparse extension (shipped since version 4.3, see hg help sparse).
• partial clones by branch: I want all revision history on branch Y: use clone -r

If you know how people will want to break things down by filepath (multiple projects in the same repo (shame on you)) you can use subrepositories (sort of like svn externals) to pre-split the repo into separately cloneable portions

Also, as to the "so huge I'd like to only get a part of it": You really only have to do that one time ever. Just clone it while you have lunch, and then you have it forever more. Subsequently you can pull and get deltas efficiently going forward. And if you want another clone of it, just clone your first clone. Where you got a clone doesn't matter (and local clones take up no additional diskspace since they're hard links under the covers).

Answer 6

The selected answer provides a good overview, but lacks a complete example.

Minimize your download and checkout footprint (a), (b):

git clone --no-checkout --depth 1 --single-branch --branch (name) (repo) (folder)
cd (folder)
git config core.sparseCheckout true
echo "target/path/1" >>.git/info/sparse-checkout
echo "target/path/2" >>.git/info/sparse-checkout
git checkout

Periodically optimize your local repository footprint (c) (optional, use with care):

git clean --dry-run # consider and tweak results then switch to --force
git gc
git repack -Ad
git prune

See also: How to handle big repositories with git