How to clone all projects of a group at once in Gi

Question

In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?

Answer 1

One liner with curl, jq, tr

for repo in $(curl "https://<your-host>/api/v4/groups/<group_id>?private_token=<your_private_token>" | jq .projects[].ssh_url_to_repo | tr -d '"'); do git clone $repo; done;

Answer 2

If you are okay with some shell sorecery this will clone all the repos grouped by their group-id (you need jq and parallel)

seq 3                                                                           \
| parallel curl -s "'https://[gitlabUrl]/api/v4/projects?page={}&per_page=100&private_token=[privateToken]'
                     | jq '.[] | .ssh_url_to_repo, .name, .namespace.path'" \
| tr -d '"'                                                                 \
| awk '{ printf "%s ", $0; if (NR % 3 == 0) print " " }'                    \
| parallel --colsep ' ' 'mkdir -p {2} && git clone {1} {3}/{2}'

Answer 3

There is a tool called myrepos, which manages multiple version controls repositories. Updating all repositories simply requires one command:

mr update

In order to register all gitlab projects to mr, here is a small python script. It requires the package python-gitlab installed:

import os
from subprocess import call
from gitlab import Gitlab

# Register a connection to a gitlab instance, using its URL and a user private token
gl = Gitlab('http://192.168.123.107', 'JVNSESs8EwWRx5yDxM5q')
groupsToSkip = ['aGroupYouDontWantToBeAdded']

gl.auth() # Connect to get the current user

gitBasePathRelative = "git/"
gitBasePathRelativeAbsolut = os.path.expanduser("~/" + gitBasePathRelative)
os.makedirs(gitBasePathRelativeAbsolut,exist_ok=True)

for p in gl.Project():
    if not any(p.namespace.path in s for s in groupsToSkip):
        pathToFolder = gitBasePathRelative + p.namespace.name + "/" + p.name
        commandArray = ["mr", "config", pathToFolder, "checkout=git clone '" + p.ssh_url_to_repo + "' '" + p.name + "'"]
        call(commandArray)

os.chdir(gitBasePathRelativeAbsolut)

call(["mr", "update"])

Answer 4

Here is another example of a bash script to clone all the repos in a group. The only dependency you need to install is jq (https://stedolan.github.io/jq/). Simply place the script into the directory you want to clone your projects into. Then run it as follows:

./myscript <group name> <private token> <gitlab url>

i.e.

./myscript group1 abc123tyn234 http://yourserver.git.com

Script:

#!/bin/bash
if command -v jq >/dev/null 2>&1; then
  echo "jq parser found";
else
  echo "this script requires the 'jq' json parser (https://stedolan.github.io/jq/).";
  exit 1;
fi

if [ -z "$1" ]
  then
    echo "a group name arg is required"
    exit 1;
fi

if [ -z "$2" ]
  then
    echo "an auth token arg is required. See $3/profile/account"
    exit 1;
fi

if [ -z "$3" ]
  then
    echo "a gitlab URL is required."
    exit 1;
fi

TOKEN="$2";
URL="$3/api/v3"
PREFIX="ssh_url_to_repo";

echo "Cloning all git projects in group $1";

GROUP_ID=$(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups?search=$1 | jq '.[].id')
echo "group id was $GROUP_ID";
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$GROUP_ID/projects?per_page=100 | jq --arg p "$PREFIX" '.[] | .[$p]' | xargs -L1 git clone

Answer 5

An alternative based on Dmitriy's answer -- in the case you were to clone repositories in a whole group tree recursively.

#!/usr/bin/python3
import os
import sys
import gitlab
import subprocess

glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
root = sys.argv[2]

def visit(group):
    name = group.name
    real_group = glab.groups.get(group.id)

    os.mkdir(name)
    os.chdir(name) 

    clone(real_group.projects.list(all=True))

    for child in real_group.subgroups.list():
        visit(child)

    os.chdir("../")

def clone(projects):
    for repo in projects:
        command = f'git clone {repo.ssh_url_to_repo}'
        process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
        output, _ = process.communicate()
        process.wait()

glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
root = sys.argv[2]

for group in groups:
    if group.name == root:
        visit(group)

Answer 6

Another way to do it with Windows "Git Bash" that has limited packages installed :

#!/bin/bash
curl -o projects.json https://<GitLabUrl>/api/v4/projects?private_token=<YourToken>
i=0
while : ; do
    echo "/$i/namespace/full_path" > jsonpointer
    path=$(jsonpointer -f jsonpointer projects.json 2>/dev/null | tr -d '"')
    [ -z "$path" ] && break
    echo $path
    if [ "${path%%/*}" == "<YourProject>" ]; then
        [ ! -d "${path#*/}" ] && mkdir -p "${path#*/}"
        echo "/$i/ssh_url_to_repo" > jsonpointer
        url=$(jsonpointer -f jsonpointer projects.json 2>/dev/null | tr -d '"')
        ( cd "${path#*/}" ; git clone --mirror "$url" )
    fi
    let i+=1
done 
rm -f projects.json jsonpointer