How to deploy private python pip dependency with A

Question

When I tried to set one of the services into AWS Elastic Beanstalk, the problem appeared. One of our python pip dependencies set up on private repository was causing errors during deployment process, since it was not accessible for pip process. Bellow I present description how we solved this issue.

It is worth to mention that there are other solutions that bind SSH keys used during deployment, to application project git repository. I find them a bit dirty, so I would like to share this one that allows to keep SSH keys separated in S3 bucket, separated from application git repository.

Answer 1

Works on: "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4" for dependency from private bitbucket repository. it may be modified for github, etc.

Add you private dependency to pip requirements.txt:

  -e [e-mail]:account_name/dependency-name.git#egg=dependency-name

Generate public & private SSH keys (HOWTO can be found elsewhere) so you have id_rsa (private) and id_rsa.pub (public) key files.

On bitbucket.org project tab, find settings and go under "Deployment keys". Use form to set your public SSH key there.

Upload both generated keys (just private should be enough) on S3 bucket using amazon AWS console:

  bucket-with-keys/bitbucket/:  
 - id_rsa
 - id_rsa.pub

Where bucket-with-keys/bitbucket - is [BUCKET_NAME]/[PATH].

Add packages file to your project: project_name/.ebextensions/packages.config

    packages:
      yum:
        git: []

Add configuration file to your project: project_name/.ebextensions/03-pip-install-from-bitbucket.config:

    files:
        "/root/.ssh/config":
            owner: root
            group: root
            mode: "000600"
            content: |
                Host *
                    StrictHostKeyChecking no
                    UserKnownHostsFile=/dev/null
        "/root/.ssh/known_hosts":
            owner: root
            group: root
            mode: "000644"
            content: |
                #
                # paste output of `ssh-keyscan -H github.com` here
                #
    commands:
        01-command:
            command: aws s3 cp --recursive s3://bucket-with-keys/bitbucket/ /root/.ssh
        02-command:
            command: chmod 600 /root/.ssh/id_rsa

If you modify this file, keep in mind that:

• commands are being executed in alphabetical order so "01-" starts before "02-" even if you would switch its order.
• file name also follows that rule.

Go to AWS console IAM (Identity & Access Management) "policies" tab and find AmazonS3FullAccess on a list and attache it for aws-elasticbeanstalk-ec2-role. If you cannot find policy you can create one like below:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "s3:*",
        "Resource": "*"
      }
    ]
  }

Look for "Create and Attach Your First Customer Managed Policy" Tutorial if needed. This policy is quite open and it is advised to create one that would be more narrow... aws-elasticbeanstalk-ec2-role is role created by default by Elastic Beanstalk but you can use your own as long as it is set with CLI tool eb config under: "IamInstanceProfile: aws-elasticbeanstalk-ec2-role"

Now you can create your environmant with eb CLI tool:

      eb create project_name-prod-env  \
        --instance_type t2.micro \
        --tier webserver \
        --region eu-west-1 \
        --cname project_name-prod-env \
        --keyname identifier-of-ssh-key-accessed-from-console-here \
        --platform "64bit Amazon Linux 2015.09 v2.0.6 running Python 3.4"

Should work now!

If sth goes wrong you may debug if SSH files got to its place:

  eb ssh
  sudo su
  ls -la /root/.ssh/

And check logs on AWS console or directly on instance:

  eb ssh
  sudo su
  less /var/log/eb-activity.log

Try manually execute commands from project_name/.ebextensions/03-pip-install-from-bitbucket.config as a root user bu in the way they appear in log file, use switches to get more verbose output.

Answer 2

I cannot comment so I'm answering the question. smentek's answer is very detailed and solves the issue. The only missing part is that you need to add the git package to the config:

packages:
    yum:
        git-all: ""

files:
    "/root/.ssh/config":
# ...

Git isn't installed on Amazon Linux by default.