A production tracker where a vendor marks five hundred orders as packed in one action, or exports every order from the last quarter. These endpoints look ordinary until the data grows, and then the naive version times out. The fix is to stop thinking one row at a time.
The naive version times out
A loop that calls save() on each object runs one query and
one database round trip per row. At five hundred rows that is five
hundred round trips inside a single web request, and the request will
not survive it.
bulk_update for many writes
Load the objects, change them in memory, then write them all in a handful of statements.
orders = list(Order.objects.filter(id__in=ids))
for order in orders:
order.status = "packed"
Order.objects.bulk_update(orders, ["status"], batch_size=500)Validate before you write
Collect every error first, then write. Wrapping the write in a transaction means a bad row aborts the whole batch instead of leaving the data half updated.
from django.db import transaction
with transaction.atomic():
Order.objects.bulk_update(orders, ["status"], batch_size=500)Exports: stream, do not build in memory
Building a full CSV string before sending it means the whole file sits
in memory at once. A StreamingHttpResponse sends each row
as it is produced.
import csv
from django.http import StreamingHttpResponse
class Echo:
"""A file-like object that returns each written line."""
def write(self, value):
return value
def export_orders(request):
writer = csv.writer(Echo())
rows = Order.objects.values_list(
"reference", "status", "total"
).iterator()
return StreamingHttpResponse(
(writer.writerow(row) for row in rows),
content_type="text/csv",
)iterator() keeps memory flat
A normal queryset caches every row it loads. Calling
iterator() tells the ORM not to, so memory stays flat no
matter how many rows the export covers.
When it is genuinely large, go async
Past a certain size, even a streamed response outlives a sensible request timeout. At that point the endpoint should accept the request, hand the work to a Celery task, and let the task build the file and email a download link when it is ready.
Guard rails
Cap how many ids a single bulk request may carry. An unbounded bulk endpoint is a denial-of-service waiting for one oversized payload. A clear limit, returned as a clean validation error, protects the database and tells the client exactly what to do.