How to clone all repos at once from GitHub?

Question

I have a company GitHub account and I want to back up all of the repositories within, accounting for anything new that might get created for purposes of automation. I was hoping something like this:

git clone [e-mail]:company/*.git 

or similar would work, but it doesn't seem to like the wildcard there.

Is there a way in Git to clone and then pull everything assuming one has the appropriate permissions?

Answer 1

I created a sample batch script. You can download all private/public repositories from github.com. After a repository is downloaded, it is automatically converted to a zip file.

@echo off
setlocal EnableDelayedExpansion
SET "username=olyanren"
SET "password=G....."
set "mypath=%cd%\"
SET "url=https://%username%:[e-mail]/%username%/"
FOR /F "tokens=* delims=" %%i in (files.txt) do (
SET repo=%%i
rmdir /s /q !repo!
git clone "!url!!repo!.git"
cd !repo!
echo !mypath!
git archive --format=zip -o "!mypath!!repo!.zip" HEAD
cd ..
)

Note: files.txt file should contain only repository names like:

repository1
repository2

Answer 2

The prevailing answers here don't take into account that the Github API will only return a maximum of 100 repositories despite what you may specify in per_page. If you are cloning a Github org with more than 100 repositories, you will have to follow the paging links in the API response.

I wrote a CLI tool to do just that:

clone-github-org -o myorg

This will clone all repositories in the myorg organization to the current working directory.

Answer 3

On Windows and all UNIX/LINUX systems, using Git Bash or any other Terminal, replace YOURUSERNAME by your username and use:

CNTX={users|orgs}; NAME={username|orgname}; PAGE=1
curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" |
  grep -e 'git_url*' |
  cut -d \" -f 4 |
  xargs -L1 git clone

Set CNTX=users and NAME=yourusername, to download all your repositories. Set CNTX=orgs and NAME=yourorgname, to download all repositories of your organization.

The maximum page-size is 100, so you have to call this several times with the right page number to get all your repositories (set PAGE to the desired page number you want to download).

Here is a shell script that does the above: https://gist.github.com/erdincay/4f1d2e092c50e78ae1ffa39d13fa404e

Answer 4

Organisation repositories

To clone all repos from your organisation, try the following shell one-liner:

GHORG=company; curl "https://api.github.com/orgs/$GHORG/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

---

User repositories

Cloning all using Git repository URLs:

GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -o 'git@[^"]*' | xargs -L1 git clone

Cloning all using Clone URL:

GHUSER=CHANGEME; curl "https://api.github.com/users/$GHUSER/repos?per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Here is the useful shell function which can be added to user's startup files (using curl + jq):

# Usage: gh-clone-user (user)
gh-clone-user() {
  curl -sL "https://api.github.com/users/$1/repos?per_page=1000" | jq -r '.[]|.clone_url' | xargs -L1 git clone
}

---

Private repositories

If you need to clone the private repos, you can add Authorization token either in your header like:

-H 'Authorization: token <token>'

or pass it in the param (?access_token=TOKEN), for example:

curl -s "https://api.github.com/users/$GHUSER/repos?access_token=$GITHUB_API_TOKEN&per_page=1000" | grep -w clone_url | grep -o '[^"]\+://.\+.git' | xargs -L1 git clone

Notes:

• To fetch only private repositories, add type=private into your query string.
• Another way is to use hub after configuring your API key.

See also:

• GitHub REST API v3 - List your repositories
• How to download GitHub Release from private repo using command line.
• How to retrieve the list of all github repositories of a person?.

---

<sup>Hints:
- To increase speed, set number of parallel processes by specifying -P parameter for xargs (-P4 = 4 processes).
- If you need to raise the GitHub limits, try authenticating by specifying your API key.
- Add --recursive to recurse into the registered submodules, and update any nested submodules within.</sup>

Answer 5

I don't think it's possible to do it that way. Your best bet is to find and loop through a list of an Organization's repositories using the API.

Try this:

• Create an API token by going to Account Settings -> Applications
• Make a call to: http://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}
• The response will be a JSON array of objects. Each object will include information about one of the repositories under that Organization. I think in your case, you'll be looking specifically for the ssh_url property.
• Then git clone each of those ssh_urls.

It's a little bit of extra work, but it's necessary for GitHub to have proper authentication.

Answer 6

To clone only private repos, given an access key, and given python 3 and requests module installed:

ORG=company; ACCESS_KEY=0000000000000000000000000000000000000000; for i in $(python -c "import requests; print(' '.join([x['ssh_url'] for x in list(filter(lambda x: x['private'] ,requests.get('https://api.github.com/orgs/$ORG/repos?per_page=1000&access_token=$ACCESS_KEY').json()))]))"); do git clone $i; done;