Picture a production tracker where a vendor marks five hundred orders as packed in one action, or exports every order from the last quarter. These endpoints look ordinary until the data grows, and then the naive version times out. The fix is to stop thinking one row at a time.
First principles
An endpoint’s cost comes from three things it holds while it runs:
- Round trips. Each query waits for the database to answer. The total wait grows with the number of queries, not only with the amount of data.
- Memory. Anything built in full before it is sent stays in the web process until the response ends.
- Time. The request has a timeout, and every row lock it takes is held until its transaction ends.
A bulk endpoint has to keep each of those flat as the row count grows. So: batch writes into a few statements, stream exports instead of building them in memory, cap the input, and move work that cannot fit in one request into the background.
The naive version times out
A loop that calls save() on each object runs one query, and one database round trip, per row. At five hundred rows that is five hundred round trips inside a single web request, and the request will not survive it.
bulk_update for many writes
Load the objects, change them in memory, then write them all in a handful of statements.
orders = list(Order.objects.filter(id__in=ids))
for order in orders:
order.status = "packed"
Order.objects.bulk_update(orders, ["status"], batch_size=500)Validate before you write
Collect every error first, then write. Wrapping the write in a transaction means a bad row aborts the whole batch instead of leaving the data half updated.
from django.db import transaction
with transaction.atomic():
Order.objects.bulk_update(orders, ["status"], batch_size=500)Exports: stream, do not build in memory
Building a full comma-separated values (CSV) string before sending it means the whole file sits in memory at once. A StreamingHttpResponse sends each row as it is produced.
import csv
from django.http import StreamingHttpResponse
class Echo:
"""A file-like object that returns each written line."""
def write(self, value):
return value
def export_orders(request):
writer = csv.writer(Echo())
rows = Order.objects.values_list(
"reference", "status", "total"
).iterator()
return StreamingHttpResponse(
(writer.writerow(row) for row in rows),
content_type="text/csv",
)iterator() keeps memory flat
A normal queryset caches every row it loads. Calling iterator() tells the object-relational mapper (ORM) not to, so memory stays flat no matter how many rows the export covers.
When it is genuinely large, go async
Past a certain size, even a streamed response outlives a sensible request timeout. At that point the endpoint should accept the request, hand the work to a Celery task, and let the task build the file and email a download link when it is ready.
Guard rails
Cap how many ids a single bulk request may carry. An unbounded bulk endpoint is a denial-of-service waiting for one oversized payload. A clear limit, returned as a clean validation error, protects the database and tells the client exactly what to do.