Django

Install Celery 3.1.25 in Django 1.10 with Ubuntu 16.04 Production

1 May , 2017  

Celery is awesome tools to manage queue in Django. It’s can reduce processing time in serial / sync process. For instance, we have schema for registering user:

1. User click on Sign-up Button
2. We create and save into records (1 second)
3. Signals trigger send SMS to User (1 second)
4. Signals trigger send Email to User (1 second)
5. Signals trigger send Email to our Admin (1 second)

User will waiting processing 5 seconds to complete the process. At this condition, we require to put the step 2-5 into queue to reduce un-necessary process.

Celery can solve this problem by queue the process and using tasks management. At this point, I will focus on how to setup Celery with Django for Production.

1. requirements.txt

1
2
3
celery==3.1.25
django-celery
django-celery-beat

Unfortunatelly, Celery 4 not working with Windows environment. We need to back to 3.1.25 to support compability.
We can start to install the given requirements

2. Setup settings.py

1
2
3
4
5
# CELERY
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672/'
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"

CELERYBEAT_SCHEDULE = {}

Put this two apps in INSTALLED_APPS

1
2
3
4
...
'djcelery',
'django_celery_beat',
..

3. Install Rabbit-MQ for Queuing Engine

1
sudo apt-get install rabbitmq-server

4. Using SystemCtl for Daemonizing

A. Create file “/etc/default/celery”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Where to chdir at start.
CELERYD_CHDIR="<your-django-project-path>"
CELERYBEAT_CHDIR="<your-django-project-path>"

ENABLED="true"
CELERYD_NODES="celery_worker"
CELERY_CONFIG_MODULE="celeryd"

CELERY_BIN="/usr/local/bin/celery"

CELERYD_OPTS="--time-limit=300 --concurrency=2"

# How to call manage.py
CELERYD_MULTI="multi"
CELERY_APP="kargo"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"
CELERYD_LOG_LEVEL="INFO"

# Workers should run as an unprivileged user.
# you can change with others user
CELERYD_USER="ubuntu"
CELERYD_GROUP="ubuntu"

CELERY_CREATE_DIRS=1

CELERYBEAT_LOG_LEVEL="INFO"

# Log files
CELERYBEAT_LOG_FILE="/var/log/celery/celerybeat.log"
# Celery Beat Log files
CELERYBEAT_PID_FILE="/var/run/celery/celerybeat.pid"
# Scheduler for celery
CELERYBEAT_OPTS="--pidfile=/var/run/celery/celerybeat.pid --schedule=/var/run/celery/celerybeat-schedule"

B. Create folder “/var/run/celery” and “/var/log/celery” and assign permission folder based with your users using “chown -R youruserhere”

C. Setup “celery.service” in “/etc/systemd/system”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=ubuntu
Group=ubuntu
EnvironmentFile=/etc/default/celery
WorkingDirectory=<your-django-project-path>
Restart=on-failure
RestartSec=20 5
KillMode=control-group
RemainAfterExit=yes

ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

D. Setup “celerybeat.service” in “/etc/systemd/system”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[Unit]
Description=Celery Beat Service
After=network.target

[Service]
Type=simple
User=ubuntu
Group=ubuntu
EnvironmentFile=/etc/default/celery
WorkingDirectory=<your-django-project-path>
KillMode=control-group

ExecStart=/bin/sh -c '${CELERY_BIN} beat \
  -A ${CELERY_APP} --pidfile=${CELERYBEAT_PID_FILE} \
  --logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYBEAT_LOG_LEVEL} ${CELERYBEAT_OPTS}'

ExecStop=/bin/systemctl kill celerybeat.service

[Install]
WantedBy=multi-user.target

E. Register both services

1
2
sudo systemctl enable celery
sudo systemctl enable celerybeat

5. Starting the services

1
2
sudo service celery start
sudo service celerybeat start

To check the log

1
2
sudo tail -f /var/log/celery/celery_worker.log
sudo tail -f /var/log/celery/celerybeat.log

, , ,


Leave a Reply

Your email address will not be published. Required fields are marked *