Unable to use pyodbc with aws lambda and API Gatew

Question

I am trying to build a AWS Lambda function using APi Gateway which utlizes pyodbc python package. I have followed the steps as mentioned in the documentation. I keep getting the following error Unable to import module 'app': libodbc.so.2: cannot open shared object file: No such file or directory when I test run the Lambda function.

Any help appreciated. I am getting the same error when I deployed my package using Chalice. It seems it could be that I need to install unixodbc-dev. Any idea how to do that through AWS Lambda?

Answer 1

Fisrt, install unixODBC and unixODBC-devel packages using yum install unixODBC unixODBC-devel. This step will install everything required for pyodbc module.

The library you're missing is located in /usr/lib64 folder on you Amazon Linux instance. Copy the library to your python project's root folder (libodbc.so.2 is just a symbolic link, make sure you copy symbolic link and library itself as listed): libodbc.so, libodbc.so.2 and libodbc.so.2.0.0

Answer 2

pyodbc uses some native libs. Therefore you cannot just copy the contents of your site-packages to Lambda, as your OS is likely not Amazon Linux.

So you need to install pyodbc on an Amazon Linux instance and use the generated libs:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Or you can get from here, if available there:

https://github.com/Miserlou/lambda-packages

Answer 3

try running this script to collect dependency into s3 bucket and then adding it to your lambda deployment package.

""" 
This lambda function collects python pip dependencies, and uploads them to S3 bucket 
as a single tar.gz file. Example input for Lambda event: 
    event = {
        "prefix"            : "myPackage",
        "saveToS3Bucket"    : "my-s3-bucket",
        "saveToS3Key"       : "package-jwt.tar.gz",
        "requirements"      : [ "cryptography==2.1.3",
                                "PyJWT==1.5.3" ]
    }

Minimal Lambda execution role:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1507151548000",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject"
                ],
                "Resource": [
                    "arn:aws:s3:::my-s3-bucket/package-jwt.tar.gz"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents"
                ],
                "Resource": "*"
            }
        ]
    }
"""

from subprocess import check_output
import uuid
import boto3

DEBUG_OUT_FILE = "/tmp/debug.txt"
S3 = boto3.resource('s3')


def lambda_handler(event, context):
    """ 
    """

    requirements = event.get('requirements', [])
    prefix = event.get('prefix', 'myPackage')
    saveToS3Bucket = event.get('saveToS3Bucket', None)
    saveToS3Key = event.get('saveToS3Key', None)
    location = "%s_%s" % (prefix, uuid.uuid4())
    destinationPath = '/tmp/%s' % location
    tarFileName = '/tmp/%s.tar.gz' % location

    for req in requirements:
        _exec(['pip', 'install', req, '-t', destinationPath])

    _exec(['tar', 'czvf', tarFileName, destinationPath])
    _persist_file_to_s3(tarFileName, saveToS3Bucket, saveToS3Key)
    return 'done!'


def _exec(statements):
    if statements and type(statements) == list:
        with open(DEBUG_OUT_FILE, "a") as f:
            try:
                f.write("\n$ %s \n" % " ".join(statements))
                rv = check_output(statements).decode("utf8")
                f.write(rv)
                print(rv)
            except Exception as ex:
                print(ex)
                f.write(str(ex))


def _persist_file_to_s3(filePathToUpload, s3Bucket, s3Key):
    if filePathToUpload and s3Bucket and s3Key:
        S3.meta.client.upload_file(filePathToUpload, s3Bucket, s3Key)