How to clone all projects of a group at once in Gi

Question

In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?

Answer 1

An updated Python 3 script that accomplishes this really effectively using Gitlab's latest api and proper pagination:

import requests
import subprocess, shlex
import os

print('Starting getrepos process..')

key = '12345678901234567890' # your gitlab key
base_url = 'https://your.gitlab.url/api/v4/projects?simple=true&per_page=10&private_token='
url = base_url + key

base_dir = os.getcwd()

while True:
    print('\n\nRetrieving from ' + url)
    response = requests.get(url, verify = False)
    projects = response.json()

    for project in projects:
        project_name = project['name']
        project_path = project['namespace']['full_path']
        project_url = project['ssh_url_to_repo']

        os.chdir(base_dir)
        print('\nProcessing %s...' % project_name)

        try:
            print('Moving into directory: %s' % project_path)
            os.makedirs(project_path, exist_ok = True)
            os.chdir(project_path)
            cmd = shlex.split('git clone --mirror %s' % project_url)
            subprocess.run(cmd)
        except Exception as e:
            print('Error: ' + e.strerror)

    if 'next' not in response.links:
        break

    url = response.links['next']['url'].replace('127.0.0.1:9999', 'your.gitlab.url')


print('\nDone')

Requires the requests library (for navigating to the page links).

Answer 2

Not really, unless:

• you have a 21st project which references the other 20 as submodules.
  (in which case a clone followed by a git submodule update --init would be enough to get all 20 projects cloned and checked out)

• or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)

---

Since 2015, Jay Gabez mentions in the comments (August 2019) the tool gabrie30/ghorg

ghorg allows you to quickly clone all of an orgs, or users repos into a single directory.

Usage:

$ ghorg clone someorg
$ ghorg clone someuser --clone-type=user --protocol=ssh --branch=develop
$ ghorg clone gitlab-org --scm=gitlab --namespace=gitlab-org/security-products
$ ghorg clone --help

---

Also (2020): https://github.com/ezbz/gitlabber

usage: gitlabber [-h] [-t token] [-u url] [--debug] [-p]
                [--print-format {json,yaml,tree}] [-i csv] [-x csv]
                [--version]
                [dest]

Gitlabber - clones or pulls entire groups/projects tree from gitlab

Answer 3

You can refer to this ruby script here: https://gist.github.com/thegauraw/da2a3429f19f603cf1c9b3b09553728b

But you need to make sure that you have the link to the organization gitlab url (which looks like: https://gitlab.example.com/api/v3/ for example organization) and private token (which looks like: QALWKQFAGZDWQYDGHADS and you can get in: https://gitlab.example.com/profile/account once you are logged in). Also do make sure that you have httparty gem installed or gem install httparty

Answer 4

In response to @Kosrat D. Ahmad as I had the same issue (with nested subgroups - mine actually went as much as 5 deep!)

#!/bin/bash
URL="https://mygitlaburl/api/v4"
TOKEN="mytoken"

function check_subgroup {
echo "checking $gid"
if [[ $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r) != "" ]]; then
  for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r)
  do
    check_subgroup
  done
else
  echo $gid >> top_level
fi
}

> top_level #empty file
> repos #empty file
for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/ | jq .[].id -r)
do
  check_subgroup
done
# This is necessary because there will be duplicates if each group has multiple nested groups. I'm sure there's a more elegant way to do this though!
for gid in $(sort top_level | uniq)
do
  curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid | jq .projects[].http_url_to_repo -r >> repos
done

while read repo; do
  git clone $repo
done <repos

rm top_level
rm repos

Note: I use jq .projects[].http_url_to_repo this can be replaced with .ssh_url_to_repo if you'd prefer.

Alternatively strip out the rm's and look at the files individually to check the output etc.

Admittedly this will clone everything, but you can tweak it however you want.

Resources: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-subgroups

Answer 5

Here's an example in Python 3:

from urllib.request import urlopen
import json
import subprocess, shlex

allProjects     = urlopen("http://[yourServer:port]/api/v4/projects?private_token=[yourPrivateTokenFromUserProfile]&per_page=100000")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict: 
    try:
        thisProjectURL  = thisProject['ssh_url_to_repo']
        command     = shlex.split('git clone %s' % thisProjectURL)
        resultCode  = subprocess.Popen(command)

    except Exception as e:
        print("Error on %s: %s" % (thisProjectURL, e.strerror))

Answer 6

I built a script (curl, git, jq required) just for that. We use it and it works just fine: https://gist.github.com/JonasGroeger/1b5155e461036b557d0fb4b3307e1e75

To find out your namespace, its best to check the API quick:

curl "https://domain.com/api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN"

There, use "namespace.name" as NAMESPACE for your group.

The script essentially does:

1. Get all Projects that match your PROJECT_SEARCH_PARAM

2. Get their path and ssh_url_to_repo

   2.1. If the directory path exists, cd into it and call git pull

   2.2. If the directory path does not exist, call git clone