Issue using M2Crypto on lambda (works on EC2)

Question

I am trying to install a python function using M2Crypto in AWS Lambda.

I spun up an EC2 instance with the Lambda AMI image, installed M2Crypto into a virtualenv, and was able to get my function working on EC2.

Then I zipped up the site-package and uploaded to Lambda. I got this error

Unable to import module 'epd_M2Crypto': /var/task/M2Crypto/_m2crypto.cpython-36m-x86_64-linux-gnu.so: symbol sk_deep_copy, version libcrypto.so.10 not defined in file libcrypto.so.10 with link time reference

There are similar questions and hints here and here. I tried uploading the offending lib (libcrypto.so.10) in the zip file, but still get the same error. I am assuming the error means that the EC2 version of libcrypto.so.10 (used to install M2Crypto) is different than the version on Lambda (that I trying to run with), so M2Crypto complains.

If I look at the versions of openssl they are different:

• OpenSSL 1.0.0-fips 29 Mar 2010 (lambda version)
• OpenSSL 1.0.2k-fips 26 Jan 2017 (ec2 version)

I don't think the answer is to downgrade openssl on ec2 as the 1.0.0 version is obsolete (AWS applies security patches but the version still shows as 1.0.0). (Also the yum doesn't have versions this old)

Here's the steps i used on the EC2 instance to get it working on EC2:

$ sudo yum -y update
$ sudo yum -y install python36
$ sudo yum -y install python-virtualenv
$ sudo yum -y groupinstall "Development Tools"
$ sudo yum -y install python36-devel.x86_64
$ sudo yum -y install openssl-devel.x86_64

$ mkdir ~/forlambda
$ cd ~/forlambda
$ virtualenv -p python3 venv
$ source venv/bin/activate

$ cd ~
$ pip install M2Crypto -t ~/forlambda/venv/lib/python3.6/site-packages/

$ cd ~/forlambda/venv/lib/python3.6/site-packages/
$ (create python function that uses M2Crypto)
$ zip -r9 ~/forlambda/archive.zip .

Then added to the zip file

• /usr/bin/openssl
• /usr/lib64/libcrypto.so.10
• /usr/lib64/libssl.so.10

And uploaded to Lambda, which is where I am now stuck.

Do I need to do something to get Lambda to use the version of libcrypto.so.10 that I have included in the uploaded zip?

My function:

"""
Wrapper for M2Crypto
https://github.com/mcepl/M2Crypto
https://pypi.org/project/M2Crypto/
"""

from __future__ import print_function
from M2Crypto import RSA
import base64
import json

def decrypt_string(string_b64):
    rsa = RSA.load_key('private_key.pem')
    string_encrypted = base64.b64decode(string_b64)
    bytes = rsa.private_decrypt(string_encrypted, 1)
    string_plaintext = bytes.decode("utf-8")

    response = {
        's': string_plaintext,
        'status': "OK",
        'statuscode': 200
    };
    return response


def lambda_handler(event, context):

    response = ""
    action = event['action']

    if action == "decrypt":
        string_b64 = event['s']
        response = decrypt_string(string_b64)

    return response

Answer 1

First I ran this command on the EC2 instance to make sure I had included the correct .so file in my .zip:

$ ldd -v _m2crypto.cpython-36m-x86_64-linux-gnu.so 

The output of the ldd command (edited for brevity):

    libssl.so.10 => /lib64/libssl.so.10 (0x00007fd5f1892000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fd5f1433000)

Based on the output above, I included /lib64/libcrypto.so.10 in my .zip.

Also (at the suggestion of AWS Support), on the Lambda console, under 'Environment variables', I added a key 'LD_LIBRARY_PATH' with value '/var/task'.

I'm not sure if I needed both those changes to fix my problem, but it works right now and after three days of troubleshooting I am afraid to touch it to see if it was one or the other that made it work.

Answer 2

It is perhaps too brutal, but would it be possible to use LD_PRELOAD to force using your preferred version of OpenSSL library?

Answer 3

AWS lambda runs code on an old version of amazon linux (amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2) as mentioned in the official documentation https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

So to run a code that depends on shared libraries, it needs to be compiled in the same environment so it can link correctly.

What I usually do in such cases is that I create virtualenv using docker container. The virtualenv can than be packaged with lambda code.

Please note that if you need install anything using yum (in the docker container), you must use same release server as the amazon linux version:

yum --releasever=2017.03 install ...

virtualenv can be built using an EC2 instance as well instead of docker container (though, I find docker method easier). Just make sure that the AMI used for EC2 is same as the one used by lambda.

Answer 4

AWS support provided a resolution, upgrading to use Python 3.7 where the issue is resolved:

Our internal team has confirmed that the issue is with Lambda's Python runtime. In a few rare cases, when the Lambda function is being initialised, Lambda is unable to link against the correct OpenSSL libraries - instead linking against Lambda's own in-built OpenSSL binaries.

The team suggests trying this out in the Python3.7 environment where this behaviour has been fixed. Also, python3.7 is compiled with the newer openssl 1.0.2 and you should not have to include the binaries in the Lambda package. ... still had to include the OpenSSL binaries in the package and could not get it working with the default libraries.