Combining multiple git repositories

Question

Let's say I've got a setup that look something like

phd/code/
phd/figures/
phd/thesis/

For historical reasons, these all have their own git repositories. But I'd like to combine them into a single one to simplify things a little. For example, right now I might make two sets of changes and have to do something like

cd phd/code
git commit 
cd ../figures
git commit

It'd be (now) nice to just to perform

cd phd
git commit

There seems to be a couple of ways of doing this using submodules or pulling from my sub-repositories, but that's a little more complex than I'm looking for. At the very least, I'd be happy with

cd phd
git init
git add [[everything that's already in my other repositories]]

but that doesn't seem like a one-liner. Is there anything in git that can help me out?

Answer 1

You could try the subtree merge strategy. It will let you merge repo B into repo A. The advantage over git-filter-branch is it doesn't require you to rewrite your history (breaking SHA1 sums).

Answer 2

Here's a solution I gave here:

1. First do a complete backup of your phd directory: I don't want to be held responsible for your losing years of hard work! ;-)

   $ cp -r phd phd-backup
   

2. Move the content of phd/code to phd/code/code, and fix the history so that it looks like it has always been there (this uses git's filter-branch command):

   $ cd phd/code
   $ git filter-branch --index-filter \
       'git ls-files -s | sed "s#\t#&code/#" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
        git update-index --index-info &&
        mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
   

3. Same for the content of phd/figures and phd/thesis (just replace code with figures and thesis).

   Now your directory structure should look like this:

   phd
     |_code
     |    |_.git
     |    |_code
     |         |_(your code...)
     |_figures
     |    |_.git
     |    |_figures
     |         |_(your figures...)
     |_thesis
          |_.git
          |_thesis
               |_(your thesis...)
   

4. Then create a git repository in the root directory, pull everything into it and remove the old repositories:

   $ cd phd
   $ git init
   
   $ git pull code
   $ rm -rf code/code
   $ rm -rf code/.git
   
   $ git pull figures --allow-unrelated-histories
   $ rm -rf figures/figures
   $ rm -rf figures/.git
   
   $ git pull thesis --allow-unrelated-histories
   $ rm -rf thesis/thesis
   $ rm -rf thesis/.git
   

   Finally, you should now have what you wanted:

   phd
     |_.git
     |_code
     |    |_(your code...)
     |_figures
     |    |_(your figures...)
     |_thesis
          |_(your thesis...)
   

One nice side to this procedure is that it will leave non-versioned files and directories in place.

Hope this helps.

---

Just one word of warning though: if your code directory already has a code subdirectory or file, things might go very wrong (same for figures and thesis of course). If that's the case, just rename that directory or file before going through this whole procedure:

$ cd phd/code
$ git mv code code-repository-migration
$ git commit -m "preparing the code directory for migration"

And when the procedure is finished, add this final step:

$ cd phd
$ git mv code/code-repository-migration code/code
$ git commit -m "final step for code directory migration"

Of course, if the code subdirectory or file is not versioned, just use mv instead of git mv, and forget about the git commits.

Answer 3

@MiniQuark solution helped me a lot, but unfortunately it doesn't take into account tags which are in source repositories (At least in my case). Below is my improvement to @MiniQuark answer.

1. First create directory which will contain composed repo and merged repos, create directory for each merged one.

   $ mkdir new_phd
   $ mkdir new_phd/code
   $ mkdir new_phd/figures
   $ mkdir new_phd/thesis
   

2. Do a pull of each repository and fetch all tags. (Presenting instructions only for code sub-directory)

   $ cd new_phd/code
   $ git init
   $ git pull ../../original_phd/code master
   $ git fetch ../../original_phd/code refs/tags/*:refs/tags/*
   

3. (This is improvement to point 2 in MiniQuark answer) Move the content of new_phd/code to new_phd/code/code and add code_ prefeix before each tag

   $ git filter-branch --index-filter 'git ls-files -s | sed "s-\t\"*-&code/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' --tag-name-filter 'sed "s-.*-code_&-"' HEAD

4. After doing so there will be twice as many tags as it was before doing filter-branch. Old tags remain in repo and new tags with code_ prefix are added.

   $ git tag
   mytag1
   code_mytag1
   

   Remove old tags manually:

   $ ls .git/refs/tags/* | grep -v "/code_" | xargs rm

   Repeat point 2,3,4 for other subdirectories

5. Now we have structure of directories as in @MiniQuark anwser point 3.

6. Do as in point 4 of MiniQuark anwser, but after doing a pull and before removing .git dir, fetch tags:

   $ git fetch catalog refs/tags/*:refs/tags/*

   Continue..

This is just another solution. Hope it helps someone, it helped me :)

Answer 4

To merge a secondProject within a mainProject:

A) In the secondProject

git fast-export --all --date-order > /tmp/secondProjectExport

B) In the mainProject:

git checkout -b secondProject
git fast-import --force < /tmp/secondProjectExport

In this branch do all heavy transformation you need to do and commit them.

C) Then back to the master and a classical merge between the two branches:

git checkout master
git merge secondProject

Answer 5

Actually, git-stitch-repo now supports branches and tags, including annotated tags (I found there was a bug which I reported, and it got fixed). What i found useful is with tags. Since tags are attached to commits, and some of the solutions (like Eric Lee's approach) fails to deal with tags. You try to create a branch off an imported tag, and it will undo any git merges/moves and sends you back like the consolidated repository being near identical to the repository that the tag came from. Also, there are issues if you use the same tag across multiple repositories that you 'merged/consolidated'. For example, if you have repo's A ad B, both having tag rel_1.0. You merge repo A and repo B into repo AB. Since rel_1.0 tags are on two different commits (one for A and one for B), which tag will be visible in AB? Either the tag from the imported repo A or from imported repo B, but not both.

git-stitch-repo helps to address that problem by creating rel_1.0-A and rel_1.0-B tags. You may not be able to checkout rel_1.0 tag and expect both, but at least you can see both, and theoretically, you can merge them into a common local branch then create a rel_1.0 tag on that merged branch (assuming you just merge and not change source code). It's better to work with branches, as you can merge like branches from each repo into local branches. (dev-a and dev-b can be merged into a local dev branch which can then be pushed to origin).

Answer 6

Perhaps, simply (similarly to the previous answer, but using simpler commands) making in each of the separate old repositories a commit that moves the content into a suitably named subdir, e.g.:

$ cd phd/code
$ mkdir code
# This won't work literally, because * would also match the new code/ subdir, but you understand what I mean:
$ git mv * code/
$ git commit -m "preparing the code directory for migration"

and then merging the three separate repos into one new, by doing smth like:

$ cd ../..
$ mkdir phd.all
$ cd phd.all
$ git init
$ git pull ../phd/code
...

Then you'll save your histories, but will go on with a single repo.