Pandas & AWS Lambda

问题:

Does anyone have a fully compiled version of pandas that is compatible with AWS Lambda?

After searching around for a few hours, I cannot seem to find what I'm looking for and the documentation on this subject is non-existent.

I need access to the package in a lambda function however I have been unsuccessful at getting the package to compile properly for usage in a Lambda function.

In lieu of the compilation can anyone provide reproducible steps to create the binaries?

Unfortunately I have not been able to successfully reproduce any of the guides on the subjects as they mostly combine pandas with scipy which I don't need and adds an extra layer of burden.

回答1:

I believe you should be able to use the recent pandas version (or likely, the one on your machine). You can create a lambda package with pandas by yourself like this,

1. First find where the pandas package is installed on your machine i.e. Open a python terminal and type

   import pandas
   pandas.__file__
   

   That should print something like '/usr/local/lib/python3.4/site-packages/pandas/__init__.py'

2. Now copy the pandas folder from that location (in this case '/usr/local/lib/python3.4/site-packages/pandas) and place it in your repository.

3. Package your Lambda code with pandas like this:

   zip -r9 my_lambda.zip pandas/
   zip -9 my_lambda.zip my_lambda_function.py
   

You can also deploy your code to S3 and make your Lambda use the code from S3.

aws s3 cp  my_lambda.zip s3://dev-code//projectx/lambda_packages/

Here's the repo that will get you started

回答2:

After some tinkering around and lot's of googling I was able to make everything work and setup a repo that can just be cloned in the future.

Key takeaways:

1. All static packages have to be compiled on an ec2 amazon Linux instance
2. The python code needs to load the libraries in the lib/ folder before executing.

Github repo: https://github.com/moesy/AWS-Lambda-ML-Microservice-Skeleton

回答3:

I managed to deploy a pandas code in aws lambda using python3.6 runtime . this is the step that i follow :

1. Add required libraries into requirements.txt
2. Build project in a docker container (using aws sam cli : sam build --use-container)
3. Run code (sam local invoke --event test.json)

this is a helper : https://github.com/ysfmag/aws-lambda-py-pandas-template

回答4:

Another option is to download the pre-compiled wheel files as discussed on this post: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/

Essentially, you need to go to the project page on https://pypi.org and download the files named like the following:

• For Python 2.7: module-name-version-cp27-cp27mu-manylinux1_x86_64.whl
• For Python 3.6: module-name-version-cp36-cp36m-manylinux1_x86_64.whl

Then unzip the .whl files to your project directory and re-zip the contents together with your lambda code.