Django AJAX requests during regular request not go

Question

I have a Django site with a login webpage. When the login form on the page is submitted, it executes the login view, which runs a function inside it that takes a long time to process (30 seconds or so). So in order to show progress to the client during login, as soon as the form is submitted, a JS function on the login page starts making AJAX POST requests to the server (to a poll_state view), which returns the state of the login. It then updates the html of the login page to show the state (like a loading bar).

My question or problem is that when I do the regular python manage.py runserver with an NGINX proxy server it works flawlessly. But when I use Gunicorn instead of python manage.py runserver the AJAX requests don't go through until the login view has completely processed and returned a response (which is the next webpage). Instead of polling the state of the login, it just returns a bunch of errors for the poll_state view after the next webpage has loaded.

When the user submits the login form, the server executes the login view, and simultaneously the client starts polling the server using AJAX requests. Here is the JS code on the login webpage, which sends the requests:

      let willstop = 0
      var poll = function() {
        $.ajax({
          url:'http://<my server's ip>/poll_state/',
          type: 'POST',
          data: {
              csrfmiddlewaretoken: $("input[name='csrfmiddlewaretoken']").val(),
              id: $('#id').val(),
          },
          success: function(pollResult) {
            step = pollResult.data
            if (step == 'Done')
              willstop = 1
            $('#step').html(step)
          }
        })
      }
      let refreshIntervalId = setInterval(function() {
        poll()
        if(willstop == 1) {
          clearInterval(refreshIntervalId);
        }
      }, 500)

It's just sends a request to my server's poll_state view (I ommitted my server's address, but can share it if it helps!) every half-second until the login is finished (step == "Done"), and then it just clears the interval (which is kinda redundant since it redirects to a new page anyway).

And the poll_state view:

def poll_state(request):
    """ A view to report the progress to the user """

    data = 'Fail'
    if request.is_ajax() and request.method == 'POST':
        data = loggingInSteps[request.POST.get('id')]
    else:
        data = 'This is not an ajax request'

    result = {'data': data}
    return JsonResponse(result)

loggingInSteps is a global dict on the server with the steps for all the clients. Each client has a unique id, which is the key and the value is the step the client is on in the login process. The key-value pair gets deleted upon redirect.

The server I am running this on is a 64-bit Ubuntu DigitalOcean droplet, so it's supposed to be my production server. I followed this tutorial to set up the server, so my NGINX configuration is:

server {
  listen 80;
  server_name cume;

  location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

  location /static/ {
    alias /home/mikmaks/dev/CUNYsecond/cume/static/;
  }
}

And I start Gunicorn either like this gunicorn config.wsgi --bind 127.0.0.1:8000 or as a daemon like this: gunicorn config.wsgi --bind 127.0.0.1:8000 --daemon --log-file ~/dev/logs/cume_gunicorn.log --workers=3 Either version doesn't work.

Does anybody know if Gunicorn can take AJAX requests from the same client as it's processing a normal request from it? Maybe the problem isn't Gunicorn, but is my NGINX configuration. Thank you so much for the help. I'll gladly provide any code or additional info if necessary :)

Answer 1

loggingInSteps is a global dict on the server with the steps for all the clients may be a problem if you have more that one process handling the request. The dictionary can be updated in one process, but not in another. Perhaps switching to single-process mode (--workers=1 ?) will resolve your problem.

Alternatively, you can redesign your application and get rid of global dict to make it work in multiprocess environment

Answer 2

Using an asynchronous worker (in my case, I used gevent) worked! Now Gunicorn processes multiple requests at a time, i.e. the request that was blocking everything (to the login view) is being processed asynchronously, so the AJAX requests go through.

I start Gunicorn up now with: gunicorn config.wsgi --bind 127.0.0.1:8000 -k gevent --worker-connections 1001 --workers=3. The only thing is now, when the load is split amongst 3 workers, it doesn't update that well, but with 1 worker it works perfectly. I'll have to figure it out some more to see if I can get it to work with 3 workers.

Thank you, Alex, for helping!