Celery is awesome tools to manage queue in Django. It’s can reduce processing time in serial / sync process. For instance, we have schema for registering user:
1. User click on Sign-up Button
2. We create and save into records (1 second)
3. Signals trigger send SMS to User (1 second)
4. Signals trigger send Email to User (1 second)
5. Signals trigger send Email to our Admin (1 second)
User will waiting processing 5 seconds to complete the process. At this condition, we require to put the step 2-5 into queue to reduce un-necessary process.
Celery can solve this problem by queue the process and using tasks management. At this point, I will focus on how to setup Celery with Django for Production.
1. requirements.txt
1 2 3 | celery==3.1.25 django-celery django-celery-beat |
Unfortunatelly, Celery 4 not working with Windows environment. We need to back to 3.1.25 to support compability.
We can start to install the given requirements
2. Setup settings.py
1 2 3 4 5 | # CELERY CELERY_BROKER_URL = ‘amqp://guest:guest@localhost:5672/’ CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler" CELERYBEAT_SCHEDULE = {} |
Put this two apps in INSTALLED_APPS
1 2 3 4 | … ‘djcelery’, ‘django_celery_beat’, .. |
3. Install Rabbit-MQ for Queuing Engine
1 | sudo apt-get install rabbitmq-server |
4. Using SystemCtl for Daemonizing
A. Create file “/etc/default/celery”
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | # Where to chdir at start. CELERYD_CHDIR="<your-django-project-path>" CELERYBEAT_CHDIR="<your-django-project-path>" ENABLED="true" CELERYD_NODES="celery_worker" CELERY_CONFIG_MODULE="celeryd" CELERY_BIN="/usr/local/bin/celery" CELERYD_OPTS="–time-limit=300 –concurrency=2" # How to call manage.py CELERYD_MULTI="multi" CELERY_APP="kargo" # %n will be replaced with the nodename. CELERYD_LOG_FILE="/var/log/celery/%n.log" CELERYD_PID_FILE="/var/run/celery/%N.pid" CELERYD_LOG_LEVEL="INFO" # Workers should run as an unprivileged user. # you can change with others user CELERYD_USER="ubuntu" CELERYD_GROUP="ubuntu" CELERY_CREATE_DIRS=1 CELERYBEAT_LOG_LEVEL="INFO" # Log files CELERYBEAT_LOG_FILE="/var/log/celery/celerybeat.log" # Celery Beat Log files CELERYBEAT_PID_FILE="/var/run/celery/celerybeat.pid" # Scheduler for celery CELERYBEAT_OPTS="–pidfile=/var/run/celery/celerybeat.pid –schedule=/var/run/celery/celerybeat-schedule" |
B. Create folder “/var/run/celery” and “/var/log/celery” and assign permission folder based with your users using “chown -R youruserhere”
C. Setup “celery.service” in “/etc/systemd/system”
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | [Unit] Description=Celery Service After=network.target [Service] Type=forking User=ubuntu Group=ubuntu EnvironmentFile=/etc/default/celery WorkingDirectory=<your-django-project-path> Restart=on-failure RestartSec=20 5 KillMode=control-group RemainAfterExit=yes ExecStart=/bin/sh -c ‘${CELERY_BIN} multi start ${CELERYD_NODES} \ -A ${CELERY_APP} –pidfile=${CELERYD_PID_FILE} \ –logfile=${CELERYD_LOG_FILE} –loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}’ ExecStop=/bin/sh -c ‘${CELERY_BIN} multi stopwait ${CELERYD_NODES} \ –pidfile=${CELERYD_PID_FILE}’ ExecReload=/bin/sh -c ‘${CELERY_BIN} multi restart ${CELERYD_NODES} \ -A ${CELERY_APP} –pidfile=${CELERYD_PID_FILE} \ –logfile=${CELERYD_LOG_FILE} –loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}’ [Install] WantedBy=multi-user.target |
D. Setup “celerybeat.service” in “/etc/systemd/system”
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | [Unit] Description=Celery Beat Service After=network.target [Service] Type=simple User=ubuntu Group=ubuntu EnvironmentFile=/etc/default/celery WorkingDirectory=<your-django-project-path> KillMode=control-group ExecStart=/bin/sh -c ‘${CELERY_BIN} beat \ -A ${CELERY_APP} –pidfile=${CELERYBEAT_PID_FILE} \ –logfile=${CELERYBEAT_LOG_FILE} –loglevel=${CELERYBEAT_LOG_LEVEL} ${CELERYBEAT_OPTS}’ ExecStop=/bin/systemctl kill celerybeat.service [Install] WantedBy=multi-user.target |
E. Register both services
1 2 | sudo systemctl enable celery sudo systemctl enable celerybeat |
5. Starting the services
1 2 | sudo service celery start sudo service celerybeat start |
To check the log
1 2 | sudo tail -f /var/log/celery/celery_worker.log sudo tail -f /var/log/celery/celerybeat.log |