Setting up private Github access with AWS Elastic

Question

Going by a recent tutorial on setting up AWS Elastic Beanstalk for Ruby deployment using Git, I just set up a Elastic Beanstalk environment from my CI server. However, the application failed to start. I went through the logs to find that bundle install was failing with an error message.

Fetching [e-mail]:example/private-repository.git Host key verification failed. fatal: The remote end hung up unexpectedly [31mGit error: command git clone '[e-mail]:example/private-repository.git' "/var/app/ondeck/vendor/cache/ruby/1.9.1/cache/bundler/git/private-repository-e4bbe6c2b13bb62664e39e345c1b01d80017934c" --bare --no-hardlinks in directory /var/app/ondeck has failed.[0m

Gemfile of my Rails application contains references to gemified plugins hosted on a couple of my owned private repositories on Github. Something like

gem 'somegemname', :git => '[e-mail]:example/private-repository.git'

I had faced similar issues with Capistrano deployments which were resolved by setting up ssh_options[:forward_agent] = true.

AWS Elastic Beanstalk Ruby container supports custom configuration through custom .config files placed under .ebextensions. Would setting up an SSH forward agent help in this case? Are there any other alternatives to reach a private Github repository while starting an Elastic Beanstalk environment?

Update 1: I just checked for the user with which a bundle install is initiated. Found out that a script /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh starts bundle install as root user. I tried creating an SSH Key under /root/.ssh and added it's pub-key to Github Deploy keys for that repository. No luck so far. Will now try to add an SSH pub-key to my user account on Github so that it applies to all private repositories accessible through my Github account.

Answer 1

Remove the private github repos from your requirements.txt file and create a script to install them using environmental variables for usernames and passwords.

file: project-root/install-extra-requirements.sh

#!/bin/sh

source /opt/python/run/venv/bin/activate
python ".extra-requirements.py"

file: project-root/.extra-requirements.py

import os

def main():
    github_username = os.environ['GITHUB_USERNAME']
    github_password = os.environ['GITHUB_PASSWORD']
    repository = "git+https://%s:[e-mail]/yourgithubrepo" % (github_username, github_password)
    os.system("pip install %s" % repository)

if __name__ == '__main__':
    main()

file: project-root/.ebextensions/002_container.config

container_commands:
  01_install_extra_requirements:
    command: './install-extra-requirements.sh'

Now you can just set GITHUB_USERNAME and GITHUB_PASSWORD as environmental variables in your elastic beanstalk environment.

Answer 2

After a good day of effort, I finally enabled use of my organization's private GitHub repos with Elastic Beanstalk by just using a .config file. I am using Python and pip, but it should also work for other package installers on EB.

rhetonik's ssh-agent+ssh-add approach did not work for me at all, so I elected to set up an ssh configuration file instead.

Here is my .ebextensions/3-pip-install-from-github.config file:

files:
    "/root/.ssh/config":
        owner: root
        group: root
        mode: "000600"
        content: |
            Host github.com
                User git
                Hostname github.com
                IdentityFile /root/.ssh/github
    "/root/.ssh/known_hosts":
        owner: root
        group: root
        mode: "000644"
        content: |
            #
            # paste output of `ssh-keyscan -H github.com` here
            #

commands:
    01-command:
        command: sudo aws s3 cp s3://bucket-with-your-github-ssh-key/github /root/.ssh
    02-command:
        command: sudo chmod 600 /root/.ssh/github

Rough instructions:

• Set up an S3 bucket accessible by your EB instance. Inside of that bucket, store the SSH key allowing access to the GitHub repository you want to access via pip, npm, bundle, etc. Use sudo aws s3 cp to copy that key onto your EB instance on deploy. sudo is necessary because EB scripts use root and not ec2-user.

• This ebextensions config file also creates 2 files on your EB instance. /root/.ssh/config tells ssh (invoked by pip and git) to use the key you copied from S3. Pasting the output of ssh-keyscan -H github.com into /root/.ssh/known_hosts will pre-verify that ssh on your EB instance is actually communicating with GitHub to avoid MITM attacks. This is better than disabling StrictHostKeyChecking in /root/.ssh/config.

Here is my requirements.txt file for pip:

Beaker==1.7.0
Flask==0.10.1
Jinja2==2.7.3
MarkupSafe==0.23
# [...]
git+ssh://[e-mail]/myorganization/myprivaterepo.git@0.0.142

While running eb-deploy, you can tail -f /var/log/eb-activity.log to make sure everything runs smoothly.

Answer 3

There are two approaches to authenticating with GitHub. I recommend associating your personal GitHub account with the private GitHub repo in either case.

The first approach passes the same ssh credentials you use locally to push, pull, and so on from the remote GitHub repo -- you uploaded your public key for your personal account, and that's what GitHub uses. To make this work when running on another server, you need to have ssh-agent running and use ssh-add to add your key to the agent -- then your personal GitHub credentials can be used to perform git commands.

The second approach is to allow the remote server(s) you're deploying to to have access to GitHub -- this may be elastic beanstalk or your actual server(s). Create a passwordless ssh key on the server (ssh-keygen -t rsa, accept defaults, or perhaps EB has a way of its own) then copy the generated public key contents and create a new "deploy key" containing that key in your GitHub repo -- you'll need to be admin, which I assume you are. An installed deploy key will allow the EB users to log into the remote server and do git pull and related commands (read-only) from the server.

I think the first method is more elegant and more easily managed as the number of servers you're deploying to grows, but which method you use may depend on EB's options.

Answer 4

Here's how I finally did it. It's all about setting up an SSH Key for the user which is responsible for bundle install phase.

1. Start an environment for an application in AWS Elastic Beanstalk
2. Optional - Login to Amazon EC2 console and change instance type to a desired value
3. Update SSH Key pair name to enable remote SSH login. (I'm sure there must be a way to specify instance type and SSH key pair name while starting an environment)
4. Look for the newly launched instance either in EC2 console or through CLI, note the Fully Qualified Domain Name (FQDN) for this instance. EB instances are like any other instance you would create with Amazon EC2. Login via SSH to this instance.
5. Execute the following commands to create an SSH key for root user

   $ sudo su - root

   $ ssh-keygen -t rsa -C "[e-mail]"

6. Edit .bash_profile to explicitly start ssh-agent and add the newly generated SSH Key. Add the following lines (This might seem unnecessary, I did it just to be sure)

   eval `ssh-agent

   eval ssh-add ~/.ssh/id_rsa

7. Note the SSH public key E.g.: ~/.ssh/id_rsa.pub and add it to the set of SSH Keys for Github account which has access to private repositories

8. At this point, your instance has access to your private Github repositories. You could test this by issuing a git clone on those repositories by logging in as root user.

9. Create an AMI out of this instance using standard methods

10. Come back to your AWS Elastic Beanstalk Dashboard and look for Edit Configuration option in your application's environment. In the Server tab, look for an option which lets you specify a Custom AMI. Update this field with the newly created AMI ID E.g.: ami-4324fd4.

11. Save configuration by hitting Apply Changes. AWS Elastic Beanstalk would start deploying new instances across your environment and terminating the old ones. This is to ensure all your auto-scaled instances have the whitelisted SSH Key required for private Github access.

After the above steps are done, you could go ahead and deploy your Rails application with git aws.push

Hope this helps others who are stuck. I'd be glad to see a more graceful solution than this one though.

Answer 5

If you're in a hurry, and your application repo is also private, you can create an additional Github user account and assign it read-only privileges to the repo containing the gem.

Then give bundler the https url with the new account's credentials:

gem 'somegemname', git: "https://username:[e-mail]/example/privaterepository"