Docker: go get from a private GitHub repo

Question

I'm trying to run a container that will expose a golang service from a package that I have on a private GitHub repo.

Since I am working with GCE, my starter image is google/debian:wheezy.

After installing all the required dependancies and tools, I am running

RUN go get github.com/<my_org>/<my_package>

where the package is a private repo.

I have added my GitHub SSH keys to allow the cloning from the private repo to the docker file:

ADD priv/id_rsa /root/.ssh/id_rsa
ADD priv/id_rsa.pub /root/.ssh/id_rsa.pub

Still, I am getting an error during the go get process when go tried to clone the repo:

# cd .; git clone https://github.com/<my_org>/<my_package> /gopath/src/github.com/<my_org>/<my_package>
Cloning into '/gopath/src/github.com/<my_org>/<my_package>'...
fatal: could not read Username for 'https://github.com': No such device or address
package github.com/<my_org>/<my_package>: exit status 128

To debug the problem, from the Dockerfile, I am running:

RUN ssh-keyscan -t rsa github.com 2>&1 >> /root/.ssh/known_hosts

And this tells me there are some problems. It looks like validating the private key is OK but something weird is going on the the public key. This is the complete ssh-keyscan result:

OpenSSH_6.0p1 Debian-4+deb7u2, OpenSSL 1.0.1e 11 Feb 2013
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to github.com [192.30.252.129] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version libssh-0.6.0
debug1: no match: libssh-0.6.0
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48
debug1: Host 'github.com' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
Warning: Permanently added the RSA host key for IP address '192.30.252.129' to the list of known hosts.
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: read PEM private key done: type <unknown>
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: No more authentication methods to try.
Permission denied (publickey).

I have tried chmod 600 and chmod 700 on the priv/public keys, this did not help.

Any clues? Has anyone succeeding in running go get that fetches from private repos on debian from docker?

Answer 1

go get is trying to use https, completely ignoring ssh.

You will have to setup ~/.netrc:

ADD priv/.netrc /root/.netrc

Where netrc looks like:

machine github.com login github-username password github-password

ref:

• https://stackoverflow.com/a/13724351/145587

Answer 2

I figured this out after a bit of hacking around. Not an ideal solution as it involves installing SSH, plus building a private key into the container. This example is based on the official Docker golang image (Debian Wheezy):

The main difference to your example is that you need a git config command to force ssh instead of the default https.

FROM golang

RUN apt-get update && apt-get install -y ca-certificates git-core ssh

ADD keys/my_key_rsa /root/.ssh/id_rsa
RUN chmod 700 /root/.ssh/id_rsa
RUN echo "Host github.com\n\tStrictHostKeyChecking no\n" >> /root/.ssh/config
RUN git config --global url.ssh://[e-mail]/.insteadOf https://github.com/

ADD . /go/src/github.com/myaccount/myprivaterepo

RUN go get github.com/myaccount/myprivaterepo
RUN go install github.com/myaccount/myprivaterepo

Answer 3

In the newest version of golang (v1.11) there are now modules.

To quote the source:

A module is a collection of related Go packages that are versioned together as a single unit. Most often, a single version-control repository corresponds exactly to a single module.

Using the latest version of golang will allow you to have dependencies that are in private repositories. Essentially by running the $ go mod vendor command will create a vendor directory locally for all external dependencies. Now making sure your docker image has Golang v1.11, you will update your Dockerfile with the following:

WORKDIR /<your repostiory>

COPY . ./

Answer 4

Elaborating on OneOfOne's ~/.netrc answer, this is what I am doing with Jenkins on linux:

FROM golang:1.6

ARG GITHUB_USER=$GITHUB_USER
ARG GITHUB_PASS=$GITHUB_PASS

# Copy local package files to the container's workspace.
ADD . /go/src/github.com/my-org/my-project
WORKDIR /go/src/github.com/my-org/my-project/

# Build application inside the container.
RUN echo "machine github.com\n\tlogin $GITHUB_USER\n\tpassword $GITHUB_PASS" >> ~/.netrc && \
    go get github.com/tools/godep && \
    go get github.com/onsi/ginkgo/ginkgo && \
    godep restore && \
    ginkgo -r --randomizeAllSpecs --randomizeSuites --failOnPending && \
    godep go install && \
    rm -f ~/.netrc

ENTRYPOINT /go/bin/my-project

EXPOSE 8080

The docker build command is:

docker build \
    --build-arg GITHUB_USER=xxxxx \
    --build-arg GITHUB_PASS=yyyyy \
    -t my-project .

The two ARG directives map --build-args so docker can use them inside the Dockerfile.

The first and last lines of RUN create and remove the ~/.netrc.

In Jenkins, I use the same creds from git pull in the build command.

In this strategy, the password is not echoed during the docker build process and not saved on any layer of your docker image. Also note that the gingko test results are printed to console during the build.