You cannot fix what you cannot see. Before observability, a production problem looks like a customer email and a guess. After it, the service tells you what broke, where, and how often. Three tools cover most of the ground: Sentry, Prometheus, and Grafana.
First principles
You cannot attach a debugger to production or ask it what happened. All you will ever know about it is what it records while it runs. So the real decision is what to record, and it has to be made before you need the answer.
Running code can record three kinds of thing:
- Events. One thing happened, such as an exception, with the details around it.
- Measurements. Counts and timings, cheap to keep, added up across many requests.
- Change over time. The same measurements laid out across hours and days, so a shift is visible.
The three tools below map onto those three. And because users feel symptoms, not causes, alerts should fire on symptoms.
Three questions, three tools
- Sentry answers what broke: the exception, the stack trace, the request that caused it.
- Prometheus answers what the numbers say: request rate, latency, error counts.
- Grafana answers show me over time: the dashboards and trends built on those numbers.
Sentry catches the exceptions
import os
import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration
sentry_sdk.init(
dsn=os.environ["SENTRY_DSN"],
integrations=[DjangoIntegration()],
traces_sample_rate=0.1,
environment=os.environ.get("ENV", "production"),
)From here, every unhandled exception arrives grouped, counted, and attached to the request that triggered it.
Prometheus exposes the metrics
Add django-prometheus and the app exposes a /metrics endpoint with request latency, response counts by status, and database timing. Prometheus scrapes it on an interval and keeps the history.
Grafana turns metrics into a story
A dashboard with 95th percentile (p95) latency, error rate, and throughput side by side turns raw numbers into a shape you can read at a glance. When something is wrong, the shape changes before the emails start.
Alert on symptoms, not causes
Alert on what the user feels: rising error rate, slow responses, requests failing. High central processing unit (CPU) use is not an incident on its own. It is a detail you look at after a symptom alert fires. Alerting on causes produces noise that teams learn to ignore.
Add context to every error
An exception with no context is a puzzle. Attach the user, the tenant, the relevant ids, and a tag for the area of the system. A well-tagged error is often fixed from the report alone, with no need to reproduce it.
Instrument as you build
The worst time to add observability is during an incident, in the dark. Wire it in as each feature ships, so the first production problem is met with a dashboard and a stack trace rather than a guess.