observability-and-reliability

Observability and reliability

Monitoring tells you a thing you predicted is happening. Observability lets you ask a question you

did not anticipate. Production failures are mostly the unanticipated kind.

Instrument for questions you have not thought of yet

Emit structured events with enough context to slice afterwards — request identifiers, user or tenant,

version, dependency, outcome, duration. Free-text logs are unsearchable at volume and become

expensive noise.

Propagate a correlation identifier across every hop. Without it, a distributed system is a set of

independent stories and reconstructing one request is manual archaeology.

Measure what the user experiences at the percentile they experience it. A p50 latency graph is

mostly a graph of the people who were not affected.

Alert on symptoms, not causes

Alert when users are affected or imminently will be. High CPU is not an alert; requests failing or

slowing is. Cause-based alerting produces pages for conditions the system handled and no page for

novel failures that hurt.

Every alert must be actionable, urgent and specific. If the recipient's honest response is to

look and close it, delete the alert — it is training the on-call to ignore the page, and the ignored

page is eventually the real one.

Alert fatigue is the actual reliability risk in most organizations. Fewer, better alerts beat

coverage.

Objectives and error budgets

Set service level objectives from what users need, then treat the remainder as a budget to spend.

This converts a sterile argument between shipping and stability into arithmetic: budget remaining

means ship, budget exhausted means the next work is reliability.

Keep the internal objective tighter than any external commitment made through

operations:service-level-management, so you find out before the customer does.

Learn from incidents

Post-incident review exists to find what made the failure possible and hard to detect, not who

touched it last. Human error is a starting question, never the finding: what made the error easy,

and why did nothing catch it?

Track the time to detect separately from time to resolve. Long detection is an observability

defect, and it is the part that repeats.

Produce a small number of real actions with owners and dates. A review generating fifteen actions

generates none.

Tooling

Metrics and traces: Datadog, Grafana with Prometheus, New Relic, Honeycomb, and similar.

Errors: Sentry, Rollbar, and similar. Logs: Elastic, OpenSearch, Loki, Splunk, and similar.

On-call and incident management: PagerDuty, Opsgenie, incident.io, FireHydrant, and similar.

Instrument with OpenTelemetry wherever you can. Vendor-specific instrumentation is the part

that makes leaving expensive.

Never

• Page a human for something they cannot act on.
• Alert on a cause when you can alert on the symptom.
• Report reliability as an average when users experience the tail.
• Close an incident review with the finding that someone was careless.