How do you merge two Git repositories?

Question

Consider the following scenario:

I have developed a small experimental project A in its own Git repo. It has now matured, and I'd like A to be part of larger project B, which has its own big repository. I'd now like to add A as a subdirectory of B.

How do I merge A into B, without losing history on any side?

Answer 1

To merge a A within B:

1) In the project A

git fast-export --all --date-order > /tmp/ProjectAExport

2) In the project B

git checkout -b projectA
git fast-import --force < /tmp/ProjectAExport

In this branch do all operations you need to do and commit them.

C) Then back to the master and a classical merge between the two branches:

git checkout master
git merge projectA

Answer 2

A single branch of another repository can be easily placed under a subdirectory retaining its history. For example:

git subtree add --prefix=rails git://github.com/rails/rails.git master

This will appear as a single commit where all files of Rails master branch are added into "rails" directory. However the commit's title contains a reference to the old history tree:

Add 'rails/' from commit <rev>

Where <rev> is a SHA-1 commit hash. You can still see the history, blame some changes.

git log <rev>
git blame <rev> -- README.md

Note that you can't see the directory prefix from here since this is an actual old branch left intact. You should treat this like a usual file move commit: you will need an extra jump when reaching it.

# finishes with all files added at once commit
git log rails/README.md

# then continue from original tree
git log <rev> -- README.md

There are more complex solutions like doing this manually or rewriting the history as described in other answers.

The git-subtree command is a part of official git-contrib, some packet managers install it by default (OS X Homebrew). But you might have to install it by yourself in addition to git.

Answer 3

I have gathered a lot of information here on Stack OverFlow, etc., and have manage to put a script together which solves the problem for me.

The caveat is that it only takes into account the 'develop' branch of each repository and merges it into a separate directory in a completely new repository.

Tags and other branches are ignored - this might not be what you want.

The script even handles feature branches and tags - renaming them in the new project so you know where they came from.

#!/bin/bash
#
################################################################################
## Script to merge multiple git repositories into a new repository
## - The new repository will contain a folder for every merged repository
## - The script adds remotes for every project and then merges in every branch
##   and tag. These are renamed to have the origin project name as a prefix
##
## Usage: mergeGitRepositories.sh <new_project> <my_repo_urls.lst>
## - where <new_project> is the name of the new project to create
## - and <my_repo_urls.lst> is a file containing the URLs to the repositories
##   which are to be merged on separate lines.
##
## Author: Robert von Burg
##            [e-mail]
##
## Version: 0.2.0
## Created: 2015-06-17
##
################################################################################
#

# Disallow using undefined variables
shopt -s -o nounset

# Script variables
declare SCRIPT_NAME="${0##*/}"
declare SCRIPT_DIR="$(cd ${0%/*} ; pwd)"
declare ROOT_DIR="$PWD"


# Detect proper usage
if [ "$#" -ne "2" ] ; then
  echo -e "ERROR: Usage: $0 <new_project> <my_repo_urls.lst>"
  exit 1
fi


# Script functions
function failed() {
  echo -e "ERROR: Merging of projects failed:"
  echo -e "$1"
  exit 1
}

function commit_merge() {
  current_branch="$(git symbolic-ref HEAD 2>/dev/null)"
  CHANGES=$(git status | grep "working directory clean")
  MERGING=$(git status | grep "merging")
  if [[ "$CHANGES" != "" ]] && [[ "$MERGING" == "" ]] ; then
    echo -e "INFO:   No commit required."
  else
    echo -e "INFO:   Committing ${sub_project}..."
    if ! git commit --quiet -m "[Project] Merged branch '$1' of ${sub_project}" ; then
      failed "Failed to commit merge of branch '$1' of ${sub_project} into ${current_branch}"
    fi
  fi
}


## Script variables
PROJECT_NAME="${1}"
PROJECT_PATH="${ROOT_DIR}/${PROJECT_NAME}"
REPO_FILE="${2}"
REPO_URL_FILE="${ROOT_DIR}/${REPO_FILE}"


# Make sure the REPO_URL_FILE exists
if [ ! -e "${REPO_URL_FILE}" ] ; then
  echo -e "ERROR: Repo file ${REPO_URL_FILE} does not exist!"
  exit 1
fi


# Make sure the required directories don't exist
if [ -e "${PROJECT_PATH}" ] ; then
  echo -e "ERROR: Project ${PROJECT_NAME} already exists!"
  exit 1
fi


# Create the new project
echo -e "INFO: Creating new git repository ${PROJECT_NAME}..."
echo -e "===================================================="
cd ${ROOT_DIR}
mkdir ${PROJECT_NAME}
cd ${PROJECT_NAME}
git init
echo "Initial Commit" > initial_commit

# Since this is a new repository we need to have at least one commit
# thus were we create temporary file, but we delete it again.
# Deleting it guarantees we don't have conflicts later when merging
git add initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
git rm --quiet initial_commit
git commit --quiet -m "[Project] Initial Master Repo Commit"
echo


# Merge all projects into th branches of this project
echo -e "INFO: Merging projects into new repository..."
echo -e "===================================================="
for url in $(cat ${REPO_URL_FILE}) ; do

  # Extract the name of this project
  export sub_project=${url##*/}
  sub_project=${sub_project%*.git}

  echo -e "INFO: Project ${sub_project}"
  echo -e "----------------------------------------------------"

  # Fetch the project
  echo -e "INFO:   Fetching ${sub_project}..."
  git remote add "${sub_project}" "${url}"
  if ! git fetch --no-tags --quiet ${sub_project} 2>/dev/null ; then
    failed "Failed to fetch project ${sub_project}"
  fi

  # Add remote branches
  echo -e "INFO:   Creating local branches for ${sub_project}..."
  while read branch ; do 
    branch_ref=$(echo $branch | tr " " "\t" | cut -f 1)
    branch_name=$(echo $branch | tr " " "\t" | cut -f 2 | cut -d / -f 3-)

    echo -e "INFO:   Creating branch ${branch_name}..."

    # Create and checkout new merge branch off of master
    git checkout --quiet -b "${sub_project}/${branch_name}" master
    git reset --hard --quiet
    git clean -d --force --quiet

    # Merge the project
    echo -e "INFO:   Merging ${sub_project}..."
    if ! git merge --quiet --no-commit "remotes/${sub_project}/${branch_name}" 2>/dev/null ; then
      failed "Failed to merge branch 'remotes/${sub_project}/${branch_name}' from ${sub_project}"
    fi

    # And now see if we need to commit (maybe there was a merge)
    commit_merge "${sub_project}/${branch_name}"

    # Relocate projects files into own directory
    if [ "$(ls)" == "${sub_project}" ] ; then
      echo -e "WARN:   Not moving files in branch ${branch_name} of ${sub_project} as already only one root level."
    else
      echo -e "INFO:   Moving files in branch ${branch_name} of ${sub_project} so we have a single directory..."
      mkdir ${sub_project}
      for f in $(ls -a) ; do
        if  [[ "$f" == "${sub_project}" ]] || 
            [[ "$f" == "." ]] || 
            [[ "$f" == ".." ]] ; then 
          continue
        fi
        git mv -k "$f" "${sub_project}/"
      done

      # Commit the moving
      if ! git commit --quiet -m  "[Project] Move ${sub_project} files into sub directory" ; then
        failed "Failed to commit moving of ${sub_project} files into sub directory"
      fi
    fi
    echo
  done < <(git ls-remote --heads ${sub_project})


  # Checkout master of sub probject
  if ! git checkout "${sub_project}/master" 2>/dev/null ; then
    failed "sub_project ${sub_project} is missing master branch!"
  fi

  # Copy remote tags
  echo -e "INFO:   Copying tags for ${sub_project}..."
  while read tag ; do 
    tag_ref=$(echo $tag | tr " " "\t" | cut -f 1)
    tag_name=$(echo $tag | tr " " "\t" | cut -f 2 | cut -d / -f 3)

    # hack for broken tag names where they are like 1.2.0^{} instead of just 1.2.0
    tag_name="${tag_name%%^*}"

    tag_new_name="${sub_project}/${tag_name}"
    echo -e "INFO:     Copying tag ${tag_name} to ${tag_new_name} for ref ${tag_ref}..."
    if ! git tag "${tag_new_name}" "${tag_ref}" 2>/dev/null ; then
      echo -e "WARN:     Could not copy tag ${tag_name} to ${tag_new_name} for ref ${tag_ref}"
    fi
  done < <(git ls-remote --tags ${sub_project})

  # Remove the remote to the old project
  echo -e "INFO:   Removing remote ${sub_project}..."
  git remote rm ${sub_project}

  echo
done


# Now merge all project master branches into new master
git checkout --quiet master
echo -e "INFO: Merging projects master branches into new repository..."
echo -e "===================================================="
for url in $(cat ${REPO_URL_FILE}) ; do

  # extract the name of this project
  export sub_project=${url##*/}
  sub_project=${sub_project%*.git}

  echo -e "INFO:   Merging ${sub_project}..."
  if ! git merge --quiet --no-commit "${sub_project}/master" 2>/dev/null ; then
    failed "Failed to merge branch ${sub_project}/master into master"
  fi

  # And now see if we need to commit (maybe there was a merge)
  commit_merge "${sub_project}/master"

  echo
done


# Done
cd ${ROOT_DIR}
echo -e "INFO: Done."
echo

exit 0

You can also get it from http://paste.ubuntu.com/11732805

First create a file with the URL to each repository, e.g.:

[e-mail]:eitchnet/ch.eitchnet.parent.git
[e-mail]:eitchnet/ch.eitchnet.utils.git
[e-mail]:eitchnet/ch.eitchnet.privilege.git

Then call the script giving a name of the project and the path to the script:

./mergeGitRepositories.sh eitchnet_test eitchnet.lst

The script itself has a lot of comments which should explain what it does.

Answer 4

If you want to put the files from a branch in repo B in a subtree of repo A and also preserve the history, keep reading. (In the example below, I am assuming that we want repo B's master branch merged into repo A's master branch.)

In repo A, first do the following to make repo B available:

git remote add B ../B # Add repo B as a new remote.
git fetch B

Now we create a brand new branch (with only one commit) in repo A that we call new_b_root. The resulting commit will have the files that were committed in the first commit of repo B's master branch but put in a subdirectory called path/to/b-files/.

git checkout --orphan new_b_root master
git rm -rf . # Remove all files.
git cherry-pick -n `git rev-list --max-parents=0 B/master`
mkdir -p path/to/b-files
git mv README path/to/b-files/
git commit --date="$(git log --format='%ai' $(git rev-list --max-parents=0 B/master))"

Explanation: The --orphan option to the checkout command checks out the files from A's master branch but doesn't create any commit. We could have selected any commit because next we clear out all the files anyway. Then, without committing yet (-n), we cherry-pick the first commit from B's master branch. (The cherry-pick preserves the original commit message which a straight checkout doesn't seem to do.) Then we create the subtree where we want to put all files from repo B. We then have to move all files that were introduced in the cherry-pick to the subtree. In the example above, there's only a README file to move. Then we commit our B-repo root commit, and, at the same time, we also preserve the timestamp of the original commit.

Now, we'll create a new B/master branch on top of the newly created new_b_root. We call the new branch b:

git checkout -b b B/master
git rebase -s recursive -Xsubtree=path/to/b-files/ new_b_root

Now, we merge our b branch into A/master:

git checkout master
git merge --allow-unrelated-histories --no-commit b
git commit -m 'Merge repo B into repo A.'

Finally, you can remove the B remote and temporary branches:

git remote remove B
git branch -D new_b_root b

The final graph will have a structure like this:

Answer 5

I wanted to move a small project to a subdirectory of a larger one. Since my small project did not have many commits, I used git format-patch --output-directory /path/to/patch-dir. Then on the larger project, I used git am --directory=dir/in/project /path/to/patch-dir/*.

This feels way less scary and way more cleaner than a filter-branch. Granted, it may not be applicable to all cases.

Answer 6

The submodule approach is good if you want to maintain the project separately. However, if you really want to merge both projects into the same repository, then you have a bit more work to do.

The first thing would be to use git filter-branch to rewrite the names of everything in the second repository to be in the subdirectory where you would like them to end up. So instead of foo.c, bar.html, you would have projb/foo.c and projb/bar.html.

Then, you should be able to do something like the following:

git remote add projb [wherever]
git pull projb

The git pull will do a git fetch followed by a git merge. There should be no conflicts, if the repository you're pulling to does not yet have a projb/ directory.

Further searching indicates that something similar was done to merge gitk into git. Junio C Hamano writes about it here: http://www.mail-archive.com/[e-mail]/msg03395.html