In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?
可以将文章内容翻译成中文,广告屏蔽插件可能会导致该功能失效(如失效，请关闭广告屏蔽插件后再试):
问题:
回答1:
Not really, unless:
you have a 21st project which references the other 20 as submodules.
(in which case a clone followed by agit submodule update --initwould be enough to get all 20 projects cloned and checked out)or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)
Since 2015, Jay Gabez mentions in the comments (August 2019) the tool gabrie30/ghorg
ghorgallows you to quickly clone all of an orgs, or users repos into a single directory.
Usage:
$ ghorg clone someorg
$ ghorg clone someuser --clone-type=user --protocol=ssh --branch=develop
$ ghorg clone gitlab-org --scm=gitlab --namespace=gitlab-org/security-products
$ ghorg clone --help
Also (2020): https://github.com/ezbz/gitlabber
usage: gitlabber [-h] [-t token] [-u url] [--debug] [-p]
[--print-format {json,yaml,tree}] [-i csv] [-x csv]
[--version]
[dest]
Gitlabber - clones or pulls entire groups/projects tree from gitlab
回答2:
Here's an example in Python 3:
from urllib.request import urlopen
import json
import subprocess, shlex
allProjects = urlopen("http://[yourServer:port]/api/v4/projects?private_token=[yourPrivateTokenFromUserProfile]&per_page=100000")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict:
try:
thisProjectURL = thisProject['ssh_url_to_repo']
command = shlex.split('git clone %s' % thisProjectURL)
resultCode = subprocess.Popen(command)
except Exception as e:
print("Error on %s: %s" % (thisProjectURL, e.strerror))
回答3:
One liner with curl, jq, tr
for repo in $(curl "https://<your-host>/api/v4/groups/<group_id>?private_token=<your_private_token>" | jq .projects[].ssh_url_to_repo | tr -d '"'); do git clone $repo; done;
回答4:
There is a tool called myrepos, which manages multiple version controls repositories. Updating all repositories simply requires one command:
mr update
In order to register all gitlab projects to mr, here is a small python script. It requires the package python-gitlab installed:
import os
from subprocess import call
from gitlab import Gitlab
# Register a connection to a gitlab instance, using its URL and a user private token
gl = Gitlab('http://192.168.123.107', 'JVNSESs8EwWRx5yDxM5q')
groupsToSkip = ['aGroupYouDontWantToBeAdded']
gl.auth() # Connect to get the current user
gitBasePathRelative = "git/"
gitBasePathRelativeAbsolut = os.path.expanduser("~/" + gitBasePathRelative)
os.makedirs(gitBasePathRelativeAbsolut,exist_ok=True)
for p in gl.Project():
if not any(p.namespace.path in s for s in groupsToSkip):
pathToFolder = gitBasePathRelative + p.namespace.name + "/" + p.name
commandArray = ["mr", "config", pathToFolder, "checkout=git clone '" + p.ssh_url_to_repo + "' '" + p.name + "'"]
call(commandArray)
os.chdir(gitBasePathRelativeAbsolut)
call(["mr", "update"])
回答5:
I built a script (curl, git, jq required) just for that. We use it and it works just fine: https://gist.github.com/JonasGroeger/1b5155e461036b557d0fb4b3307e1e75
To find out your namespace, its best to check the API quick:
curl "https://domain.com/api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN"
There, use "namespace.name" as NAMESPACE for your group.
The script essentially does:
- Get all Projects that match your
PROJECT_SEARCH_PARAM Get their
pathandssh_url_to_repo2.1. If the directory
pathexists, cd into it and callgit pull2.2. If the directory
pathdoes not exist, callgit clone
回答6:
Here is another example of a bash script to clone all the repos in a group. The only dependency you need to install is jq (https://stedolan.github.io/jq/). Simply place the script into the directory you want to clone your projects into. Then run it as follows:
./myscript <group name> <private token> <gitlab url>
i.e.
./myscript group1 abc123tyn234 http://yourserver.git.com
Script:
#!/bin/bash
if command -v jq >/dev/null 2>&1; then
echo "jq parser found";
else
echo "this script requires the 'jq' json parser (https://stedolan.github.io/jq/).";
exit 1;
fi
if [ -z "$1" ]
then
echo "a group name arg is required"
exit 1;
fi
if [ -z "$2" ]
then
echo "an auth token arg is required. See $3/profile/account"
exit 1;
fi
if [ -z "$3" ]
then
echo "a gitlab URL is required."
exit 1;
fi
TOKEN="$2";
URL="$3/api/v3"
PREFIX="ssh_url_to_repo";
echo "Cloning all git projects in group $1";
GROUP_ID=$(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups?search=$1 | jq '.[].id')
echo "group id was $GROUP_ID";
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$GROUP_ID/projects?per_page=100 | jq --arg p "$PREFIX" '.[] | .[$p]' | xargs -L1 git clone
回答7:
If you are okay with some shell sorecery this will clone all the repos grouped by their group-id (you need jq and parallel)
seq 3 \
| parallel curl -s "'https://[gitlabUrl]/api/v4/projects?page={}&per_page=100&private_token=[privateToken]'
| jq '.[] | .ssh_url_to_repo, .name, .namespace.path'" \
| tr -d '"' \
| awk '{ printf "%s ", $0; if (NR % 3 == 0) print " " }' \
| parallel --colsep ' ' 'mkdir -p {2} && git clone {1} {3}/{2}'
回答8:
An updated Python 3 script that accomplishes this really effectively using Gitlab's latest api and proper pagination:
import requests
import subprocess, shlex
import os
print('Starting getrepos process..')
key = '12345678901234567890' # your gitlab key
base_url = 'https://your.gitlab.url/api/v4/projects?simple=true&per_page=10&private_token='
url = base_url + key
base_dir = os.getcwd()
while True:
print('\n\nRetrieving from ' + url)
response = requests.get(url, verify = False)
projects = response.json()
for project in projects:
project_name = project['name']
project_path = project['namespace']['full_path']
project_url = project['ssh_url_to_repo']
os.chdir(base_dir)
print('\nProcessing %s...' % project_name)
try:
print('Moving into directory: %s' % project_path)
os.makedirs(project_path, exist_ok = True)
os.chdir(project_path)
cmd = shlex.split('git clone --mirror %s' % project_url)
subprocess.run(cmd)
except Exception as e:
print('Error: ' + e.strerror)
if 'next' not in response.links:
break
url = response.links['next']['url'].replace('127.0.0.1:9999', 'your.gitlab.url')
print('\nDone')
Requires the requests library (for navigating to the page links).
回答9:
Yep it's possible, here is the code.
prerequisites:
pip install python-gitlab
#!/usr/bin/python3
import os
import sys
import gitlab
import subprocess
glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
groupname = sys.argv[2]
for group in groups:
if group.name == groupname:
projects = group.projects.list(all=True)
for repo in projects:
command = f'git clone {repo.ssh_url_to_repo}'
process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
output, _ = process.communicate()
process.wait()
Example:
- create .py file (ex. gitlab-downloader.py)
- copy-paste code from above
- on Linux OS (or OSX) do chmod +x on the script file (ex. chmod +x gitlab-downloader.py)
- run it with 3 params: Gitlab hostname, groupname, your Personal Access Token(see https://gitlab.exmaple.com/profile/personal_access_tokens)
回答10:
You can refer to this ruby script here: https://gist.github.com/thegauraw/da2a3429f19f603cf1c9b3b09553728b
But you need to make sure that you have the link to the organization gitlab url (which looks like: https://gitlab.example.com/api/v3/ for example organization) and private token (which looks like: QALWKQFAGZDWQYDGHADS and you can get in: https://gitlab.example.com/profile/account once you are logged in). Also do make sure that you have httparty gem installed or gem install httparty
回答11:
I have written the script to pull the complete code base from gitlab for particular group.
for pag in {1..3} // number of pages projects has span {per page 20 projects so if you have 50 projects loop should be 1..3}
do
curl -s http://gitlink/api/v4/groups/{groupName}/projects?page=$pag > url.txt
grep -o '"ssh_url_to_repo": *"[^"]*"' url.txt | grep -o '"[^"]*"$' | while read -r line ; do
l1=${line%?}
l2=${l1:1}
echo "$l2"
git clone $l2
done
done
回答12:
Another way to do it with Windows "Git Bash" that has limited packages installed :
#!/bin/bash
curl -o projects.json https://<GitLabUrl>/api/v4/projects?private_token=<YourToken>
i=0
while : ; do
echo "/$i/namespace/full_path" > jsonpointer
path=$(jsonpointer -f jsonpointer projects.json 2>/dev/null | tr -d '"')
[ -z "$path" ] && break
echo $path
if [ "${path%%/*}" == "<YourProject>" ]; then
[ ! -d "${path#*/}" ] && mkdir -p "${path#*/}"
echo "/$i/ssh_url_to_repo" > jsonpointer
url=$(jsonpointer -f jsonpointer projects.json 2>/dev/null | tr -d '"')
( cd "${path#*/}" ; git clone --mirror "$url" )
fi
let i+=1
done
rm -f projects.json jsonpointer
回答13:
An alternative based on Dmitriy's answer -- in the case you were to clone repositories in a whole group tree recursively.
#!/usr/bin/python3
import os
import sys
import gitlab
import subprocess
glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
root = sys.argv[2]
def visit(group):
name = group.name
real_group = glab.groups.get(group.id)
os.mkdir(name)
os.chdir(name)
clone(real_group.projects.list(all=True))
for child in real_group.subgroups.list():
visit(child)
os.chdir("../")
def clone(projects):
for repo in projects:
command = f'git clone {repo.ssh_url_to_repo}'
process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
output, _ = process.communicate()
process.wait()
glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
root = sys.argv[2]
for group in groups:
if group.name == root:
visit(group)
回答14:
In response to @Kosrat D. Ahmad as I had the same issue (with nested subgroups - mine actually went as much as 5 deep!)
#!/bin/bash
URL="https://mygitlaburl/api/v4"
TOKEN="mytoken"
function check_subgroup {
echo "checking $gid"
if [[ $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r) != "" ]]; then
for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid/subgroups/ | jq .[].id -r)
do
check_subgroup
done
else
echo $gid >> top_level
fi
}
> top_level #empty file
> repos #empty file
for gid in $(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/ | jq .[].id -r)
do
check_subgroup
done
# This is necessary because there will be duplicates if each group has multiple nested groups. I'm sure there's a more elegant way to do this though!
for gid in $(sort top_level | uniq)
do
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$gid | jq .projects[].http_url_to_repo -r >> repos
done
while read repo; do
git clone $repo
done <repos
rm top_level
rm repos
Note: I use jq .projects[].http_url_to_repo this can be replaced with .ssh_url_to_repo if you'd prefer.
Alternatively strip out the rm's and look at the files individually to check the output etc.
Admittedly this will clone everything, but you can tweak it however you want.
Resources: https://docs.gitlab.com/ee/api/groups.html#list-a-groups-subgroups
