tensorflow deployment on openshift, errors with gu

I'm implementing a small app based on this git: gitlab.com/osevg/python-flask-modwsgi

All I've changed is the wsgi.py to execute a script that analyzes an uploaded file (the file is uploaded through a separate PHP-based app, sent to this Python app via http post). The script itself is pretty trivial and runs without any issues on my local Windows device (takes 5-6s to run, then send back the results via JSON). I'm working on getting the tool running on Openshift v3 now.

import os
from flask import Flask, request, redirect, url_for
from werkzeug.utils import secure_filename

os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import tensorflow as tf, sys

import json

UPLOAD_FOLDER = 'uploads'
ALLOWED_EXTENSIONS = set(['jpeg'])

application = Flask(__name__)
application.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
application.config['PROPAGATE_EXCEPTIONS'] = True

@application.route('/', methods=['GET', 'POST'])
def upload_file():
    if request.method == 'POST':
            filename = secure_filename(file.filename)
            file.save(os.path.join(application.config['UPLOAD_FOLDER'], filename))
            # change this as you see fit
            #image_path = sys.argv[1]
            image_path = os.path.join(application.config['UPLOAD_FOLDER'], filename)

            # Read in the image_data
            image_data = tf.gfile.FastGFile(image_path, 'rb').read()

            # Loads label file, strips off carriage return
            label_lines = [line.rstrip() for line
                               in tf.gfile.GFile("retrained_labels.txt")]

            # Unpersists graph from file
            with tf.gfile.FastGFile("retrained_graph.pb", 'rb') as f:
                graph_def = tf.GraphDef()
                graph_def.ParseFromString(f.read())
                _ = tf.import_graph_def(graph_def, name='')

            with tf.Session() as sess:
                # Feed the image_data as input to the graph and get first prediction
                softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')

                predictions = sess.run(softmax_tensor, \
                         {'DecodeJpeg/contents:0': image_data})

                # Sort to show labels of first prediction in order of confidence
                top_k = predictions[0].argsort()[-len(predictions[0]):][::-1]

                results = []

                for node_id in top_k:
                    human_string = label_lines[node_id]
                    score = predictions[0][node_id] * 100
                    # print('%s (score = %.5f)' % (human_string, score))
                    results.append({"guess" : human_string, "stat" : score.tolist()})

                return json.dumps(results)

if __name__ == "__main__":
    application.run(debug=True)

============== UPDATE: Works exactly 50% of the time. Succeeds the first time; fails the second; alternates success/failure.

On failure, logs show: "Truncated or oversized response headers received from daemon process." Driven from "terminated by signal 9."

---> Running application from Python script (app.py) ...
Server URL         : http://localhost:8080/
Server Root        : /tmp/mod_wsgi-localhost:8080:1030200000
Server Conf        : /tmp/mod_wsgi-localhost:8080:1030200000/httpd.conf
Error Log File     : /dev/stderr (debug)
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : en_US.UTF-8
[Tue May 23 22:06:46.439793 2017] [wsgi:debug] [pid 1:tid 139903948601472] src/server/mod_wsgi.c(8247): mod_wsgi (pid=1): Socket for 'localhost:8080' is '/tmp/mod_wsgi-localhost:8080:1030200000/wsgi.1.0.1.sock'.
[Tue May 23 22:06:46.440087 2017] [wsgi:debug] [pid 1:tid 139903948601472] src/server/mod_wsgi.c(8305): mod_wsgi (pid=1): Listen backlog for socket '/tmp/mod_wsgi-localhost:8080:1030200000/wsgi.1.0.1.sock' is '100'.
[Tue May 23 22:06:46.440465 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Starting process 'localhost:8080' with threads=5.
[Tue May 23 22:06:46.440605 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9947): mod_wsgi (pid=24): Setting lang to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 22:06:46.440611 2017] [mpm_event:notice] [pid 1:tid 139903948601472] AH00489: Apache/2.4.18 (Red Hat) mod_wsgi/4.5.15 Python/3.5 configured -- resuming normal operations
[Tue May 23 22:06:46.440623 2017] [mpm_event:info] [pid 1:tid 139903948601472] AH00490: Server built: Jul 13 2016 08:48:18
[Tue May 23 22:06:46.440635 2017] [core:notice] [pid 1:tid 139903948601472] AH00094: Command line: 'httpd (mod_wsgi-express)         -f /tmp/mod_wsgi-localhost:8080:1030200000/httpd.conf -D MOD_WSGI_MULTIPROCESS -D MOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'
[Tue May 23 22:06:46.440639 2017] [core:debug] [pid 1:tid 139903948601472] log.c(1546): AH02639: Using SO_REUSEPORT: yes (1)
[Tue May 23 22:06:46.440645 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9960): mod_wsgi (pid=24): Setting locale to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 22:06:46.440745 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Python home /opt/app-root.
[Tue May 23 22:06:46.440811 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Initializing Python.
[Tue May 23 22:06:46.440993 2017] [mpm_event:debug] [pid 25:tid 139903948596992] event.c(2094): AH02471: start_threads: Using epoll
[Tue May 23 22:06:46.465067 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(10116): mod_wsgi (pid=24): Process 'localhost:8080' logging to 'localhost'.
[Tue May 23 22:06:46.465193 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Attach interpreter ''.
[Tue May 23 22:06:46.473206 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Imported 'mod_wsgi'.
[Tue May 23 22:06:46.473277 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24, process='localhost:8080', application=''): Loading WSGI script '/tmp/mod_wsgi-localhost:8080:1030200000/handler.wsgi'.
[Tue May 23 22:06:48.916242 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9366): mod_wsgi (pid=24): Starting 5 threads in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916238 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8929): mod_wsgi (pid=24): Enable monitor thread in process 'localhost:8080'.
[Tue May 23 22:06:48.916251 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 1 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916252 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8933): mod_wsgi (pid=24): Startup timeout is 15.
[Tue May 23 22:06:48.916255 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8936): mod_wsgi (pid=24): Deadlock timeout is 60.
[Tue May 23 22:06:48.916257 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8939): mod_wsgi (pid=24): Idle inactivity timeout is 0.
[Tue May 23 22:06:48.916259 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8942): mod_wsgi (pid=24): Request time limit is 60.
[Tue May 23 22:06:48.916261 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8945): mod_wsgi (pid=24): Graceful timeout is 15.
[Tue May 23 22:06:48.916263 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8948): mod_wsgi (pid=24): Eviction timeout is 0.
[Tue May 23 22:06:48.916266 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8951): mod_wsgi (pid=24): Restart interval is 0.
[Tue May 23 22:06:48.916328 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 2 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916375 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 3 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916414 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 4 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916481 2017] [wsgi:info] [pid 24:tid 139903482156800] mod_wsgi (pid=24): Started thread 0 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916496 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 5 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916448 2017] [wsgi:info] [pid 24:tid 139903473764096] mod_wsgi (pid=24): Started thread 1 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916510 2017] [wsgi:info] [pid 24:tid 139903465371392] mod_wsgi (pid=24): Started thread 2 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916611 2017] [wsgi:info] [pid 24:tid 139903456978688] mod_wsgi (pid=24): Started thread 3 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916514 2017] [wsgi:debug] [pid 24:tid 139903490549504] src/server/mod_wsgi.c(8894): mod_wsgi (pid=24): Enable deadlock thread in process 'localhost:8080'.
[Tue May 23 22:06:48.916628 2017] [wsgi:info] [pid 24:tid 139903448585984] mod_wsgi (pid=24): Started thread 4 in daemon process 'localhost:8080'.
[Tue May 23 23:00:24.575348 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:24.575385 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:24.575496 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:24.575507 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:24.576463 2017] [wsgi:debug] [pid 25:tid 139903948330752] src/server/mod_wsgi.c(11650): mod_wsgi (pid=25): Request server was 'localhost|8080'.
[Tue May 23 23:00:24.576578 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12648): mod_wsgi (pid=24): Server listener address '|8080'.
[Tue May 23 23:00:24.576591 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12657): mod_wsgi (pid=24): Server listener address '|8080' was found.
[Tue May 23 23:00:24.576610 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12669): mod_wsgi (pid=24): Connection server matched was 'localhost|8080'.
[Tue May 23 23:00:24.576617 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12685): mod_wsgi (pid=24): Request server matched was 'localhost|8080'.
[Tue May 23 23:00:24.838739 2017] [mpm_event:debug] [pid 49:tid 139903948596992] event.c(2094): AH02471: start_threads: Using epoll
[Tue May 23 23:00:43.265971 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:43.266000 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:43.269467 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:43.269492 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:43.270535 2017] [wsgi:debug] [pid 25:tid 139903948064512] src/server/mod_wsgi.c(11650): mod_wsgi (pid=25): Request server was 'localhost|8080'.
[Tue May 23 23:00:43.270648 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12648): mod_wsgi (pid=24): Server listener address '|8080'.
[Tue May 23 23:00:43.270669 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12657): mod_wsgi (pid=24): Server listener address '|8080' was found.
[Tue May 23 23:00:43.270675 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12669): mod_wsgi (pid=24): Connection server matched was 'localhost|8080'.
[Tue May 23 23:00:43.270680 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12685): mod_wsgi (pid=24): Request server matched was 'localhost|8080'.
[Tue May 23 23:00:43.792919 2017] [wsgi:error] [pid 25:tid 139903948064512] [client 10.131.0.1:47450] Truncated or oversized response headers received from daemon process 'localhost:8080': /tmp/mod_wsgi-localhost:8080:1030200000/htdocs/
[Tue May 23 23:00:43.856204 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' has died, deregister and restart it.
[Tue May 23 23:00:43.856264 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' terminated by signal 9
[Tue May 23 23:00:43.856270 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' has been deregistered and will no longer be monitored.
[Tue May 23 23:00:43.868568 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Starting process 'localhost:8080' with threads=5.
[Tue May 23 23:00:43.868708 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9947): mod_wsgi (pid=70): Setting lang to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 23:00:43.868749 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9960): mod_wsgi (pid=70): Setting locale to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 23:00:43.876257 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Python home /opt/app-root.
[Tue May 23 23:00:43.877029 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Initializing Python.
[Tue May 23 23:00:43.972394 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(10116): mod_wsgi (pid=70): Process 'localhost:8080' logging to 'localhost'.
[Tue May 23 23:00:43.972537 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Attach interpreter ''.
[Tue May 23 23:00:43.986453 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Imported 'mod_wsgi'.
[Tue May 23 23:00:43.986556 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70, process='localhost:8080', application=''): Loading WSGI script '/tmp/mod_wsgi-localhost:8080:1030200000/handler.wsgi'.
[Tue May 23 23:00:46.405597 2017] [wsgi:error] [pid 70:tid 139903948601472] test1
[Tue May 23 23:00:46.405883 2017] [wsgi:error] [pid 70:tid 139903948601472] test5
[Tue May 23 23:00:46.406112 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8929): mod_wsgi (pid=70): Enable monitor thread in process 'localhost:8080'.
[Tue May 23 23:00:46.406126 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8933): mod_wsgi (pid=70): Startup timeout is 15.
[Tue May 23 23:00:46.406130 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8936): mod_wsgi (pid=70): Deadlock timeout is 60.
[Tue May 23 23:00:46.406132 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8939): mod_wsgi (pid=70): Idle inactivity timeout is 0.
[Tue May 23 23:00:46.406134 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8942): mod_wsgi (pid=70): Request time limit is 60.
[Tue May 23 23:00:46.406142 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9366): mod_wsgi (pid=70): Starting 5 threads in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406145 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8945): mod_wsgi (pid=70): Graceful timeout is 15.
[Tue May 23 23:00:46.406148 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 1 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406148 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8948): mod_wsgi (pid=70): Eviction timeout is 0.
[Tue May 23 23:00:46.406151 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8951): mod_wsgi (pid=70): Restart interval is 0.
[Tue May 23 23:00:46.406193 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 2 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406235 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 3 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406283 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 4 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406305 2017] [wsgi:info] [pid 70:tid 139903465371392] mod_wsgi (pid=70): Started thread 2 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406330 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 5 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406382 2017] [wsgi:debug] [pid 70:tid 139903490549504] src/server/mod_wsgi.c(8894): mod_wsgi (pid=70): Enable deadlock thread in process 'localhost:8080'.
[Tue May 23 23:00:46.406428 2017] [wsgi:info] [pid 70:tid 139903482156800] mod_wsgi (pid=70): Started thread 0 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406442 2017] [wsgi:info] [pid 70:tid 139903473764096] mod_wsgi (pid=70): Started thread 1 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406460 2017] [wsgi:info] [pid 70:tid 139903456978688] mod_wsgi (pid=70): Started thread 3 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406554 2017] [wsgi:info] [pid 70:tid 139903448585984] mod_wsgi (pid=70): Started thread 4 in daemon process 'localhost:8080'.

UPDATE #2: Second time through, it appears to fail at the following line:

        predictions = sess.run(softmax_tensor, \
                 {'DecodeJpeg/contents:0': image_data})

SEMI-UPDATE: As an FYI to anyone reading, I can switch back to gunicorn and the problem persists (fails every time with "booting worker with pid" on the TF side and 502 error in the sending side).

FINAL (MAYBE) UPDATE: Back to mod_wsgi. Embrace the madness. As the hackiest of all hack-ey workarounds, I just implemented a dummy 2nd curl on the call side so that the first call works and the second call causes the error that restarts the server. Ridiculous, yes; however, it works and you really can't tell the difference on the frontend (just a starting demo anyways).