Debugging where you cannot deploy a fix

Debugging on a customer site is the same activity as debugging at home with two of its limbs removed. You usually cannot reproduce it, because you do not have their data. You usually cannot iterate, because a deploy needs a change ticket, a window and somebody's approval. What is left is the discipline of narrowing without running the code.

What is actually different

You cannot see the whole system. Logs are partial, retention is thirty days or thirty minutes, and the interesting request is in a component you do not own. The temptation is to ask for more logs; the better first move is to ask which decision the missing log would change, because half the time it would change none.

The clock is someone else's. Every hour of downtime is a business event in a room you are not in. That changes the shape of the work: you are not looking for the bug, you are looking for the cheapest next test.

The reproduction lives with a person. Some clerk in accounts receives the malformed record and the system logs nothing unusual. Getting that person's screen, or their three example records, is worth more than an hour of log reading.

Fixes have a shipping cost. A hotfix means a ticket, a rollback plan and a conversation. So you only ship a fix when you can name the cause; otherwise you ship an instrument.

The order of moves

Work these in sequence and stop as soon as one discriminates. The point of the sequence is that each step halves the space for less than the cost of the one before it.

1. Confirm the symptom, not the report. "It's slow" from three people can be three different things. Get one timestamp, one identifier, one screenshot. Then check whether the system agrees that something happened at that timestamp.
2. Find the boundary that still works. If the last successful case was at 10:00 and the first failure at 10:20, the change at 10:11 is a suspect regardless of what anyone says it did.
3. Check whether the request arrived. A 200 from your endpoint proves your handler ran. It does not prove the action completed, and it certainly does not prove the destination accepted it. This single distinction accounts for a large share of "the integration is broken" tickets.
4. Read the data, not the code. In customer environments the data is wrong far more often than the logic. Take the failing identifier and walk it through the pipeline by hand; the place where your expectations and the record diverge is the find.
5. Instrument once, then wait. Add the smallest observable thing that would separate your two remaining hypotheses, and put it where the customer can see it. An instrument that only you can read turns you into a permanent dependency.
6. Change one thing. On customer systems, two simultaneous changes produce an outcome nobody can attribute, and the second one always gets blamed.

The failures that look like bugs

Three patterns recur often enough to check before reading any stack trace.

Duplicate work. An event delivered twice, a retry that was not idempotent, a scheduled job that overlapped itself. Symptoms look intermittent and correlated with volume, which is why the search for a race condition runs for a week.

Stale configuration. The environment variable was updated; the process was not restarted. Or it was restarted on two of three nodes and the load balancer is round-robin.

The clock. Timestamps in different time zones, a certificate that expired at 02:00 UTC, a cache with an hour of skew. This one produces the most convincing "it's random" reports.

What to write while you are in it

Keep a timeline in the incident document while it is happening, with times and who said what. Two reasons: post-incident reviews fail mainly from missing sequence, and a customer who watched you write things down trusts the eventual explanation more than one who watched you type.

Separate observed from inferred in every line. "Requests to the queue dropped to zero at 10:11" is observed. "The worker died" is inferred, and it may be wrong — the worker may have been alive and unable to reach the queue.

The part that is not technical

The person reporting the problem is usually frightened, and the loudest person in the room is rarely the one with the information. Two sentences do most of the work: we have it, here is what we know, here is when you will hear from me next. A next-update time you keep is worth more than a fix you deliver late without one.

And when it is over, the review has to produce a change, not a narration. If the output is a document that says "human error", nothing has been learned, because human error is the description of every incident and the cause of none. What an incident does to the deployment's survival is the subject of the four-week rule; who gets told, and when, belongs in the escalation path.