Debugging production: a method, not a hunch
The fastest production debuggers aren't the ones who know the most. They stop the bleeding first, build a timeline, ask what changed, and narrow the search by halves. A repeatable method for incidents.
Production incidents reward method over brilliance. The fastest way to a fix is rarely knowing every corner of the system; it’s not skipping steps. When an alert fires, the instinct is to open the logs and start reading. That instinct costs hours.
Here’s the method I use.
1. Stop the bleeding first
Your first job is to reduce user impact, not to understand the problem. Understanding can happen after users are fine.
- Roll back the last deploy if the timing fits, even if you’re “pretty sure” it isn’t the cause. A rollback that doesn’t help is cheap information.
- Fail over to another region or replica.
- Disable the feature flag behind the new code path.
- Shed load: rate-limit the noisy client, pause the batch job, scale out.
Declare the incident, name one person to coordinate and one to communicate, and post updates on a fixed rhythm. A quiet incident channel makes everyone else start their own investigation in parallel.
2. Build a timeline
“It’s slow” isn’t a starting point; “p99 latency on checkout went from 300 ms to 12 s at 09:02 UTC” is. Pin down:
- When exactly it started (and whether it was a step or a ramp).
- What is affected: which endpoints, error codes, regions, customers.
- What isn’t affected. Healthy neighbours are as informative as broken ones.
A step change usually means something changed. A slow ramp usually means something filled up: a disk, a queue, a connection pool, a cache that’s slowly expiring.
3. Ask “what changed?”
Most incidents are caused by a change. Check, in roughly this order:
| Change | Where to look |
|---|---|
| Code deploys | Deploy log, around the start time |
| Configuration and feature flags | Flag history, config repo |
| Infrastructure | Scaling events, node replacements, certificate renewals, DNS changes |
| Traffic | Volume, shape, a new client, a bot |
| Dependencies | Status pages, error rates on outbound calls |
| Data | A new tenant, a huge record, a backfill job running |
A shared, searchable change log across all of these is one of the highest-value investments a platform team can make. Without it, step 3 becomes a poll of the whole engineering department.
4. Form a hypothesis you can test quickly
“Something’s wrong with the database” isn’t testable. “Connection pool exhaustion on the order service since the 09:00 deploy, which raised the default timeout” is: check pool metrics and the diff in five minutes.
Write the hypothesis in the incident channel before you test it. It keeps the team aligned, stops three people testing the same idea, and makes the timeline easy to write afterwards. When a hypothesis is disproved, say so and move on — don’t keep half-believing it.
5. Narrow by halves
When there’s no obvious change, bisect the problem space:
- One region or all of them?
- One endpoint or all of them?
- One customer or all of them? (A single tenant with unusual data is behind more incidents than people expect.)
- One host or pod or all of them? (One bad node behind a load balancer looks like a small, random error rate.)
- Before or after a dependency? Traces answer this directly: where does the time go?
Each question halves the search space. Five good questions narrow almost anything.
This is where observability earns its cost: structured logs with a trace ID, traces across service boundaries (including message queues), and metrics you can slice by endpoint, tenant and host. If you can’t slice, you can’t bisect. (I packaged my defaults for this into a Spring Boot starter.)
6. Write it down
After the incident, write a short, blameless review:
- Timeline — from first symptom to recovery, with timestamps.
- Impact — who was affected, for how long, how badly.
- Cause — the technical chain of events, as far as you know it.
- Detection — how did we find out, and could we have found out sooner?
- Follow-ups — a small number of concrete actions with owners: a missing alert, a guardrail, a runbook step.
The detection question is often the most valuable. “The customer told us” is a finding in itself.
Habits that make the method faster
- Dashboards built for questions, not decoration: the four golden signals (latency, traffic, errors, saturation) per service, sliceable by endpoint.
- Runbooks for the alerts you have, with the first three things to check.
- Practice: game days where you break something on purpose in a safe environment and run the method.
Takeaways
- Mitigate before you investigate.
- Build a precise timeline, then ask what changed.
- Work from testable hypotheses, said out loud.
- Bisect by region, endpoint, tenant, host and dependency.
- Finish with a blameless review that improves detection, not just the fix.