ZeroMQ + Django & uwsgi issues

Question

Using django, we need to send a message to another, separate python program. Zeromq seems to be lightweight and should fit the bill for this.

However, trying to get it to work and it always ends up with a ZeroMQ: Bad Address error, when setting the socket to zmq.PUSH (or anything else). This is the traceback:

Exception Type:     ZMQError
Exception Value:    Bad address
...
...
sock = context.socket(zmq.PUSH)
/usr/local/lib/python2.7/dist-packages/zmq/sugar/context.py in socket
s = self._socket_class(self, socket_type)
self <zmq.sugar.context.Context object at 0x200dc80>
socket_type 8

Context was made in the calling function in models.py, and just does:

context = zmq.Context()
sock = context.socket(zmq.PUSH)
< ^ crash here>
sock.bind('tcp://127.0.0.1:8921')
...

It is launched via

exec uwsgi_python \
    --master --pidfile=/tmp/blah.pid \
    --chdir=/path/to/site \
    --module=program.wsgi:application \
    --env DJANGO_SETTINGS_MODULE=program.settings \
    --uid user --gid 1000 \
    --socket=/tmp/uwsgi_program.sock \
    --chmod-socket \
    --vacuum \
    --harakiri=20 \
    --processes=2 \
    --max-requests=5000 \
    --die-on-term

Also have tried adding --lazy to the launch script, and that didn't help, same error.

The wsgi.py file has

import django.core.handlers.wsgi
from raven.contrib.django.middleware.wsgi import Sentry
application = Sentry(django.core.handlers.wsgi.WSGIHandler())

Of course, everything works fine with runserver or, another server not using uWSGI.

So it seems that the zeromq context it creates is invalid somehow. Trying to modify the wsgi.py file to generate a zeromq context there, using @postfork still don't solve this issue, exact same error. However, I also don't like using @postfork, since that would require separate codepaths depending on if we use uWSGI or something else, and rather do this more cleanly, if possible.

What am I overlooking ?

Answer 1

I tried all the different options and finally decided that the python uwsgi option would be the best for our setup. Instructions for the installation can be found at the following site

Answer 2

I also had such problem on uWSGI 1.9.13 from Ubuntu 13.10 repositoryes. But localy built 1.9.19 works fine.

Answer 3

You have already tried all the right approaches (@postfork, --lazy-apps, --lazy...), so unless you are using a really outdated uWSGI version i can only suppose (as i see the zmq.sugar wrapper in place) that you need --enable-threads in the uWSGI options (but it would be the first time seeing it).

The problem arise because zmq context create a background thread and this thread is not inherited after fork().

Have you tried removing the master and using a single process (so removing the fork() presence) if things go better ?