Engineering field notes / Example 02

The queue that
would not drain.

A deployment reduced worker concurrency. The API stayed healthy while the background work quietly fell behind.

Fictional incident. This is a teaching example, not a real outage. All service names, times, metrics, and outcomes are invented.

38 minDegradation window · 09:02–09:40 UTC

1,280Peak queued jobs · sampled at 09:15

6 minTime to first alert · 09:08 UTC

One queue. One clear signal.

Request success did not describe the user experience. Queue age would have shown the problem sooner.

320

810

1,280

1,020

690

330

100

09:0009:2009:40 UTC

Invented five-minute samples. The CSV in this collection contains the exact values.

What happened

09:02 · Change reaches workersConcurrency falls from 64 to 8.

09:08 · Queue alarm firesThe responder checks worker configuration rather than restarting everything.

09:17 · Previous configuration restoredProcessing capacity recovers; the backlog still needs time to drain.

09:40 · Recovery verifiedQueue reaches zero and a synthetic export completes.

The learning, not the blame

What failed in the system?

A valid configuration value was operationally unsafe. Deployment checks confirmed that workers started, but not that they could keep up with the assumed workload.

What should change next?

Add a configuration floor for the agreed capacity envelope, alert on oldest-job age, and make a representative background job part of the deployment gate. Each action needs an owner role, a deadline, and a verification artifact.

How do I reuse this?

Open the Markdown postmortem, blank template, and CSV using the document list. Replace the story with evidence from your incident. Keep unknowns explicit and do not infer recovery from one green dashboard.