I'm following this tutorial, and adjusting the Celery-background related code to my project.
In my case I am operating in a Docker environment, and I have a secured site (i.e. https://localhost).
which requires secured ssl communication.
I adjusted the code for secure connection.
I had initial connection problems, which created log error messages, but I was able to solve - see here.
Now the log file is quite, but I think that I still have connection problems. As a result, at runtime, when triggerring a task, nothing happens.
What could be the reason for no connection in the secured case?
Should I expect a message if the keys are incorrect?
Is there a way to test the connection from celery/web containers to redis container from the command line?
I was able to fix the problem by making changes to the configuration of the containers.
Specifically, I made the following changes:
In redis container:
- followed this tutorial to add stunnel, and create certs for the redis container
- used this git code to configure stunnel within a Docker container
In celery container:
- elevated the log level to debug
I first tested that I can connect from my localhost to the redis docker container over ssl. This is described here
Then, I tested that I can connect from the celery container to the redis container over ssl.
The docker-compose file is:
version: '3'
services:
web:
restart: always
build:
context: ./web
dockerfile: Dockerfile
expose:
- "8000"
volumes:
- /home/webServer/web:/home/flask/app/web
- /home/webServer/redis/ssl:/etc/certs
- data2:/home/flask/app/web/project/img
command: /usr/local/bin/gunicorn -w 2 -t 3600 -b :8000 project:app
depends_on:
- postgres
stdin_open: true
tty: true
nginx:
restart: always
build:
context: ./nginx
dockerfile: Dockerfile
ports:
- "80:80"
- "443:443"
volumes:
- /home/webServer/web:/home/flask/app/web
- data2:/home/flask/app/web/project/img
depends_on:
- web
postgres:
restart: always
build:
context: ./postgresql
dockerfile: Dockerfile
volumes:
- data1:/var/lib/postgresql/data
expose:
- "5432"
redis:
build:
context: ./redis
dockerfile: Dockerfile
restart: always
command: sh -c "stunnel /stunnel-redis-server.conf && /usr/local/bin/redis-server /etc/redis/redis.conf"
expose:
- '6380'
ports:
- "6380:6380"
volumes:
- /home/webServer/redis/ssl:/etc/certs
- /home/webServer/redis/conf:/etc/redis
celery:
build:
context: ./web
command: watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker -A project.celery --loglevel=debug
volumes:
- /home/webServer/web:/home/flask/app/web
- /home/webServer/redis/ssl:/etc/certs
- data2:/home/flask/app/web/project/img
depends_on:
- redis
volumes:
data1:
data2:
Other related files are:
files on the redis docker container: (the settings in these files are described here)
- redis container Dockerfile
- redis/conf/redis.conf
- redis/stunnel-redis-server.conf
settings on the celery docker container:
cat web/project/flask_celery.py
...
key_file = '/etc/certs/localhost.key'
cert_file = '/etc/certs/private.pem'
ca_file = '/etc/certs/myCA.pem'
...
celery = Celery(app.import_name,
backend=app.config['CELERY_RESULT_BACKEND'],
broker=app.config['CELERY_BROKER_URL'],
broker_use_ssl = {
'ssl_keyfile': key_file,
'ssl_certfile': cert_file,
'ssl_ca_certs': ca_file,
'ssl_cert_reqs': ssl.CERT_REQUIRED
},
redis_backend_use_ssl = {
'ssl_keyfile': key_file,
'ssl_certfile': cert_file,
'ssl_ca_certs': ca_file,
'ssl_cert_reqs': ssl.CERT_REQUIRED
})
------------------
cat project/__init__.py
...
app.config['CELERY_BROKER_URL'] = 'rediss://webserver_redis_1:6380/0'
app.config['CELERY_RESULT_BACKEND'] = 'rediss://webserver_redis_1:6380/0'