How to clone all repos at once from GitHub?

Question

I have a company GitHub account and I want to back up all of the repositories within, accounting for anything new that might get created for purposes of automation. I was hoping something like this:

git clone [e-mail]:company/*.git 

or similar would work, but it doesn't seem to like the wildcard there.

Is there a way in Git to clone and then pull everything assuming one has the appropriate permissions?

Answer 1

For orgs you have access to with private repos:

curl -u <YOUR_GITHUB_USERNAME> -s https://api.github.com/orgs/<ORG_NAME>/repos?per_page=200 | ruby -rubygems -e ’require “json”; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo[“html_url”]} ]}'

It uses the html_url, so you don't need an access_token just enter your github password when prompted.

Answer 2

Go to Account Settings -> Application and create an API key
Then insert the API key, github instance url, and organization name in the script below

#!/bin/bash

# Substitute variables here
ORG_NAME="<ORG NAME>"
ACCESS_TOKEN="<API KEY>"
GITHUB_INSTANCE="<GITHUB INSTANCE>

URL="https://${GITHUB_INSTANCE}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}"

curl ${URL} | ruby -rjson -e 'JSON.load(STDIN.read).each {|repo| %x[git clone #{repo["ssh_url"]} ]}'

Save that in a file, chmod u+x the file, then run it.

_{Thanks to Arnaud for the ruby code.}

Answer 3

This gist accomplishes the task in one line on the command line:

curl -s https://api.github.com/orgs/[your_org]/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Replace [your_org] with your organization's name. And set your per_page if necessary.

UPDATE:

As ATutorMe mentioned, the maximum page size is 100, according to the GitHub docs.

If you have more than 100 repos, you'll have to add a page parameter to your url and you can run the command for each page.

curl -s "https://api.github.com/orgs/[your_org]/repos?page=2&per_page=100" | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'

Note: The default per_page parameter is 30.

Answer 4

So, in practice, if you want to clone all repos from the organization FOO which match BAR, you could use the one-liner below, which requires jq and common cli utilities

curl 'https://api.github.com/orgs/FOO/repos?access_token=SECRET' |
  jq '.[] |
  .ssh_url' |
  awk '/BAR/ {print "git clone " $0 " & "}' |
  sh

Answer 5

Clone all your repos that are not forks:

curl -u "username" https://api.github.com/user/repos\?page\=1\&per_page\=100 |
  jq -r 'map(select(.fork == false)) | .[] | .ssh_url' |
  xargs -L1 git clone

Clone your gists:

curl https://api.github.com/users/username/gists\?page\=1\&per_page\=100 |
   jq -r ".[] | .git_pull_url +\" '\" + (.files|keys|join(\"__\") + \"'\")" |
   xargs -L1 git clone

This jq command is complex because gists' repo's name are hashes, so that command concatenates all filenames to be the repo's name

---

You can filter the JSON arbitrarily using jq

install: sudo apt-get install jq

In the example above, I filtered out forks using this: curl ... | jq -r 'map(select(.fork == false))' ... -- useful for not cloning repos where you've made casual pull requests

jq supports some very advanced features. man jq is your friend

---

You can authenticate with curl -u "username" ... to access private repos

---

Guthub's API urls

• Your repos (needs authentication): https://api.github.com/user/repos\?page\=1\&per_page\=100
• Any user: https://api.github.com/users/other_username/repos\?page\=1\&per_page\=100
• Orgs: https://api.github.com/orgs/orgname/repos\?page\=1\&per_page\=100

Github API Docs for repos

Answer 6

You can get a list of the repositories by using curl and then iterate over said list with a bash loop:

GIT_REPOS=`curl -s curl https://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN} | grep ssh_url | awk -F': ' '{print $2}' | sed -e 's/",//g' | sed -e 's/"//g'`
for REPO in $GIT_REPOS; do
  git clone $REPO
done