Anything slow or unreliable does not belong on the request path. Sending an email, generating a report, calling a third-party API: each one can take seconds or fail outright. Celery moves that work to a background worker, and RabbitMQ is the broker that carries the jobs.
What belongs in a task
If a piece of work can be slow, can fail in a way worth retrying, or does not need to finish before the user sees a response, it is a task. Everything else can stay inline.
Wiring Celery to RabbitMQ
# trampoline/celery.py
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "trampoline.settings")
app = Celery("trampoline")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
In settings, CELERY_BROKER_URL points at RabbitMQ. The
broker holds the queue of pending jobs and hands them to whichever
worker is free.
A task that can fail safely
from celery import shared_task
@shared_task(bind=True, max_retries=3, default_retry_delay=10)
def sync_order_to_warehouse(self, order_id):
try:
order = Order.objects.get(pk=order_id)
warehouse_client.push(order.as_payload())
except WarehouseTimeout as exc:
raise self.retry(exc=exc)Pass ids, not objects
A task argument is serialized and may sit in the queue for a while. Pass a primary key and load the row fresh inside the task. A serialized model instance is a snapshot that is stale before the worker ever picks it up.
Make every task idempotent
Retries mean a task can run more than once. Design each task so that running it twice does no harm: check whether the work is already done, or key the side effect on a unique id before committing it.
acks_late and the worker that crashes
By default a job is acknowledged the moment a worker receives it, so a
crash mid-task loses the work. Setting acks_late=True holds
the acknowledgement until the task finishes, so a crashed task is
redelivered to another worker.
Give tasks a time limit
A soft_time_limit raises an exception inside a task that
runs too long, so one stuck job cannot hold a worker hostage forever.
Pair it with monitoring on queue depth: a queue that only grows is
telling you the workers cannot keep up.
The shape this gives you
The request handler does the minimum, fires a task, and returns. The worker does the slow part, retries on failure, and survives a crash. The user never waits for any of it.