When zipping a virtual env for AWS Lambda deployme

Question

Introduction

I'm just starting to use AWS Lambda and as much as I hate it, I freaking love it. I've created a Makefile to help me package my virtual env and ship to S3. After I figured out that cryptography requires a hidden file in the site-packages directory #GRRR, I started wondering how I can further improve my packaging process.

Context

This is what a new virtualenv on a new Amazon Linux AMI EC2 instance looks like.

$ uname -srvm
Linux 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64

$ cat /etc/system-release
Amazon Linux AMI release 2016.09

$ virtualenv --version
15.1.0

$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

$ virtualenv temp
New python executable in /home/ec2-user/temp/bin/python2.7
Also creating executable in /home/ec2-user/temp/bin/python
Installing setuptools, pip, wheel...done.

^{fig. 1}

$ ls -a temp/lib/python2.7/site-packages/
.                        packaging-16.8.dist-info   setuptools-34.3.2.dist-info
..                       pip                        six-1.10.0.dist-info
appdirs-1.4.3.dist-info  pip-9.0.1.dist-info        six.py
appdirs.py               pkg_resources              six.pyc
appdirs.pyc              pyparsing-2.2.0.dist-info  wheel
easy_install.py          pyparsing.py               wheel-0.29.0.dist-info
easy_install.pyc         pyparsing.pyc
packaging                setuptools

^{fig. 2}

I found that in order to do the python development I needed (using paramiko), I had to do this to prepare (prior to fig.1 & fig.2):

sudo yum install gcc python27-devel libffi-devel openssl-devel
sudo -H pip install --upgrade pip virtualenv

^{fig. 3}

Question

Of those site-packages in fig. 2, which ones can I omit from the zip I send to AWS?

For the sake of comparison, this is what my complete project's virtualenv has in it (and the only thing I pip installed was paramiko):

$ ls -a aws_lambda_project/lib/python2.7/site-packages/
.                             packaging
..                            packaging-16.8.dist-info
appdirs-1.4.3.dist-info       paramiko
appdirs.py                    paramiko-2.1.2.dist-info
appdirs.pyc                   pip
asn1crypto                    pip-9.0.1.dist-info
asn1crypto-0.22.0.dist-info   pkg_resources
cffi                          pyasn1
cffi-1.9.1.dist-info          pyasn1-0.2.3.dist-info
_cffi_backend.so              pycparser
cryptography                  pycparser-2.17.dist-info
cryptography-1.8.1.dist-info  pyparsing-2.2.0.dist-info
easy_install.py               pyparsing.py
easy_install.pyc              pyparsing.pyc
enum                          setuptools
enum34-1.1.6.dist-info        setuptools-34.3.2.dist-info
idna                          six-1.10.0.dist-info
idna-2.5.dist-info            six.py
ipaddress-1.0.18.dist-info    six.pyc
ipaddress.py                  wheel
ipaddress.pyc                 wheel-0.29.0.dist-info
.libs_cffi_backend

Answer 1

This works for me, please give it a try:

$ mkdir paramiko-lambda && cd paramiko-lambda
$ virtualenv env --python=python2.7 && source env/bin/activate
$ pip freeze > pre_paramiko.txt
$ pip install paramiko
$ pip freeze > post_paramiko.txt

I then put the following in a script to make sure it works locally:

from __future__ import print_function
import paramiko


def handler(event, context):
    print(paramiko.__version__)
    ssh_client = paramiko.SSHClient()

if __name__ == '__main__':
    handler(event=None, context=None)

The last two lines are optional, just a simple way to test the script locally. To see what was installed along with paramiko, I compared the two text files:

$ diff -u pre_paramiko.txt post_paramiko.txt
--- pre_paramiko.txt
+++ post_paramiko.txt
@@ -1,4 +1,13 @@
 appdirs==1.4.3
+asn1crypto==0.22.0
+cffi==1.10.0
+cryptography==1.8.1
+enum34==1.1.6
+idna==2.5
+ipaddress==1.0.18
 packaging==16.8
+paramiko==2.1.2
+pyasn1==0.2.3
+pycparser==2.17
 pyparsing==2.2.0
 six==1.10.0

The modules with a + were installed with paramiko so must be included with .zip archive that gets uploaded to AWS Lambda. It would be easy to write a bash script that takes the output of the diff command and automate the creation of the .zip archive, but I'm just going to enter them in manually.

$ cd env/lib/python2.7/site-packages
$ zip -x "*.pyc" -r ../../../../paramiko_lambda.zip packaging asn1crypto cffi cryptography enum idna ipaddress paramiko pyasn1 pycparser
$ cd ../../../../
$ zip -r paramiko_lambda.zip paramiko_lambda.py

I needed to add the packaging folder probably because of print(paramiko.__version__) so it may not be necessary. The paramiko_lambda.zip file was 2.5 MB and while not huge had a lot of unnecessary data, specifically *.pyc files. Excluding *.pyc files reduced the file to 1.5 MB.