Issue with imports when using WSGI in EC2 Instance

Question

I am trying to run a very simple Flask app off an EC2 instance using mod_wsgi. My apache error log keeps showing

"ImportError: No module named pandas, referer: http://xxxxx"

Despite the fact that I have pandas installed. For reference, pip freeze yields me

click==6.7
Flask==0.12
itsdangerous==0.24
Jinja2==2.9.5
MarkupSafe==0.23
numpy==1.12.0
pandas==0.19.2
python-dateutil==2.6.0
pytz==2016.10
scikit-learn==0.18.1
scipy==0.18.1
six==1.10.0
sklearn==0.0
virtualenv==15.1.0
Werkzeug==0.11.15

Previously, I tried using virtualenv (pip freeze of my venv is very similar to what I posed above) and then modifying the .wsgi file to use the virtualenv with the code:

activate_this = '/home/ubuntu/sklearn-env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

Doing this through the virtualenv gives me a flask app that just times out when I make a simple GET request with nothing useful in the apache error logs and using my local pip gives me ImportErrors with all my Python modules despite the fact that I am able to open a Python2.7 instance and import pandas, flask, etc. with no problem (Python 2.7.12 for reference). I'm completely stumped here, any advice?

EDIT: So I actually fixed the issue I was having with local pip as far as the import error, but now, both my local pip and using my virtualenv just gives me a flask app that hangs forever if I import anything besides flask. If I copy this code in

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
  return 'Hello from Flask!'

if __name__ == '__main__':
  app.run()

it works perfectly fine. I can literally just add import pandas and now my app just spins indefinitely. The only thing coming from the apache error log is

[Thu Feb 23 01:02:52.010864 2017] [wsgi:warn] [pid 11686:tid 140507506435968] mod_wsgi: Compiled for Python/2.7.11. [Thu Feb 23 01:02:52.010902 2017] [wsgi:warn] [pid 11686:tid 140507506435968] mod_wsgi: Runtime using Python/2.7.12. [Thu Feb 23 01:02:52.011483 2017] [mpm_event:notice] [pid 11686:tid 140507506435968] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations [Thu Feb 23 01:02:52.011498 2017] [core:notice] [pid 11686:tid 140507506435968] AH00094: Command line: '/usr/sbin/apache2'

which I was under the impression, wasn't actually breaking anything.

My .conf file is just modifying the default one.

<VirtualHost *:80>
        #ServerName www.example.com

        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        WSGIDaemonProcess classification-poc threads=5
        WSGIScriptAlias / /var/www/html/classification-poc/server.wsgi

        <Directory classification-poc>
    WSGIProcessGroup classification-poc
    WSGIApplicationGroup %{GLOBAL}
   Order deny,allow
    Allow from all
        </Directory>

        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined


        #Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

My server.wsgi file is just

import sys
sys.path.insert(0, '/var/www/html/classification-poc')
from testserver import app as application

when I am not using virtualenv. When I try it with virtualenv, I just add

activate_this = '/home/ubuntu/sklearn-env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

to the top of the file.

Answer 1

After a long and painful exercise, I was able to finally get my app running .The issue is with the pandas 0.19.2 built when the application is getting imported in the .wsgi file

To resolve it Remove your imports from the global level and insert them at the function level

import pandas as pd
....
@app.route('/getFunction', methods=["GET"])
def sample_get_function():
    movieData=pd.read_csv('someData.csv')

to

....
@app.route('/getFunction', methods=["GET"])
def sample_get_function():
    import pandas as pd
    movieData=pd.read_csv('someData.csv')

This is not a very good solution but it is working

Answer 2

You must set:

WSGIApplicationGroup %{GLOBAL}

in mod_wsgi configuration when using numpy and related packages. If you don't it can hang because numpy has not been implemented in a way that it will work in Python sub interpreters.

See:

• http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api