uWSGI lazy-apps and ThreadPool

问题:

Running uWSGI with

$ cat /etc/uwsgi/uwsgi.cfg

[uwsgi]
callable = app
socket = /var/run/arivale-service/uwsgi.sock
chmod-socket = 666
pidfile = /var/run/arivale-service/uwsgi.pid
master = true
enable-threads = true
single-interpreter = true
thunder-lock
need-app
processes = 4

Without lazy-apps enabled, a request to the calling following endpoint hangs

import boto3
# ...irrelevant imports
from multiprocessing.dummy import Pool as ThreadPool


POOL = ThreadPool(6)

# ...irrelevant setup


def get_ecs_task_definitions(service_name):
    ecs_service_name, _ = get_ecs_service_name_and_cluster(service_name)

    def get_task_definition(task_definition_arn):
        formatted_task_definition = {}
        task_definition = ECS.describe_task_definition(taskDefinition=task_definition_arn)['taskDefinition']
        # ...
        # build formatted_task_definition from task_definition
        # ...
        return formatted_task_definition
    task_definition_arns = ECS.list_task_definitions(
        familyPrefix=ecs_service_name, status='ACTIVE')['taskDefinitionArns']
    return POOL.map(get_task_definition, task_definition_arns)


@service.api('/api/services/<service_name>/ecs/task-definitions')
def get_task_definitions(service_name):
    return {'service_name': service_name, 'task_definitions': get_ecs_task_definitions(service_name)}

NGINX is balancing uWSGI apps, and this is what it says in error.log (NGINX):

Jun 10 03:54:26 93e04e04e2cf nginx_error: 2017/06/10 03:54:26 [error] 49#49: *33 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.16.254.95, server: localhost, request: "GET /api/services/data-analysis-service/ecs/task-definitions HTTP/1.1",upstream: "uwsgi://unix:/var/run/arivale-service/uwsgi.sock", host: "devops-service.arivale.com"

Each request to the endpoint hangs a worker (below is the output of uwsgitop after 2 requests):

uwsgi-2.0.15 - Sat Jun 10 21:26:10 2017 - req: 0 - RPS: 0 - lq: 0 - tx: 0
node: localhost - cwd: /var/www/arivale-service/web - uid: 0 - gid: 0 - masterpid: 45
 WID    %       PID     REQ     RPS     EXC     SIG     STATUS  AVG     RSS     VSZ     TX      ReSpwn  HC      RunT    LastSpwn
 1      0.0     74      0       0       0       0       busy    0ms     0       0       0       1       0       0       21:23:20
 2      0.0     75      0       0       0       0       busy    0ms     0       0       0       1       0       0       21:23:20
 3      0.0     76      0       0       0       0       idle    0ms     0       0       0       1       0       0       21:23:20
 4      0.0     77      0       0       0       0       idle    0ms     0       0       0       1       0       0       21:23:20

Enabling lazy-apps fixes the issue. Does anyone know with certainty why?

回答1:

This happened because uWSGI workers were unable to access the thread pool created in the master, see https://stackoverflow.com/a/39890941/2419669.

Using @postfork fixes this:

global THREAD_POOL = None

@postfork
def _make_thread_pool():
    global THREAD_POOL
    THREAD_POOL = ThreadPool(8)