# Engineering Design Doc — Reliable Background Exports

## Document Directory

This paste contains 4 documents:

- [README.md](https://marke.st/p/01M33QY8MJHN2V7JJPT1MCD9D7/README.md) ← (current)
- [state-machine.js](https://marke.st/p/01M33QY8MJHN2V7JJPT1MCD9D7/state-machine.js) (code)
- [job.json](https://marke.st/p/01M33QY8MJHN2V7JJPT1MCD9D7/job.json) (code)
- [review-checklist.md](https://marke.st/p/01M33QY8MJHN2V7JJPT1MCD9D7/review-checklist.md)

---

## README.md

# Reliable background exports

### A small system design with the difficult parts left in.

**DESIGN REVIEW · FICTIONAL SERVICE · NOT A DEPLOYED SYSTEM**

> This proposal describes an invented document-export service. Names, targets, capacities, and implementation status are illustrative. It contains no production architecture or customer information.

## 1. The problem

A user asks for a report. Generating it can take longer than an interactive request should remain open. The user needs a stable job identifier, a truthful status, and a way to retrieve the result later.

**Proposed outcome:** accept a job quickly, process it asynchronously, and make completion visible without pretending that an accepted request is a completed export.

| Goal | Example acceptance criterion |
|---|---|
| Acknowledge work | Return a job ID after durable job creation |
| Survive retries | Repeated equivalent submissions with the same scoped key resolve to the same job |
| Explain progress | Expose queued, running, completed, and failed states |
| Control access | Every job lookup and result download checks the caller's authorization |
| Make failure visible | A job cannot remain running forever without an explicit timeout/recovery path |

These are proposed requirements, not measured service guarantees.

## 2. The smallest useful architecture

```text
Browser / API client
        |
        | create export + idempotency key
        v
API ----+----> Job database
                   |
                   | job record + outbox record in one transaction
                   v
              Outbox relay ----> Work queue
                                    |
                                    v
                                  Worker
                                    |
                           result object storage
                                    |
                         conditional job completion
                                    v
                          Authorized status / download
```

This is a **proposed transactional-outbox design**: the job and an outbox entry are written together. A separate relay publishes the work. The relay may retry, so the worker must tolerate duplicate deliveries.

## 3. State is a contract

| Current state | Allowed next state | Required condition in this proposal |
|---|---|---|
| queued | running | Worker obtains a valid claim |
| running | completed | Result exists and completion update matches the current claim |
| running | queued | Retryable failure; attempt budget remains |
| running | failed | Terminal failure or retry budget exhausted |
| completed | None | Terminal; a new export gets a new job |
| failed | None | Terminal; deliberate resubmission creates a new job |

The adjacent JavaScript demonstrates only the state-transition rules. It does **not** implement authentication, distributed claims, database transactions, or storage cleanup.

## 4. Idempotency and concurrency

Scope an idempotency key to the tenant and operation. Store a normalized request fingerprint with it. Reusing the key for different input should return a conflict instead of silently producing the wrong export.

A worker claim includes an attempt number or fencing token. Completing a job uses a conditional update that matches that claim; a stale worker cannot overwrite a newer result. Writing an output object is not, by itself, proof that the job has completed.

## 5. Access and data lifecycle

Treat queue messages as references to authorized job records, not as the source of permission. Avoid putting document bodies or credentials into routine logs. Define result retention and cleanup explicitly. When returning a temporary result URL, choose a lifetime appropriate to the application's risk and verify access before issuing it.

## 6. Failure cases worth reviewing

| Failure | Proposed response | Evidence needed before release |
|---|---|---|
| API process stops after database commit | Relay later publishes the outbox entry | Recovery test |
| Relay publishes twice | Claim and job rules tolerate duplication | Duplicate-delivery test |
| Worker stops after writing output | Lease recovery and conditional completion reconcile state | Crash-point test |
| Result expires | Explain expiry and offer deliberate regeneration | User-flow test |
| Caller guesses another job ID | Deny access without exposing the result | Cross-account authorization test |

## 7. Deliberate non-goals

No scheduler UI, cross-region processing, arbitrary user-supplied execution, or promise of exactly-once message delivery. Do not add a second queue or distributed cache until a requirement justifies it.

## 8. Rollout and decisions

Start with synthetic documents in a non-production environment. Exercise retry, duplicate, crash, timeout, and authorization paths. Only then consider a small opt-in rollout. See `review-checklist.md` for review gates.

**Open decisions:** expected job duration distribution, maximum result size, retention period, concurrency envelope, and operational ownership. Leaving these open is better than inventing requirements.

### Related examples

These are separate teaching artifacts, not a jointly versioned implementation. This design proposes idempotency and an outbox; the compact API example intentionally omits an idempotency contract.

[API contract](https://marke.st/p/01M33QJ9HPSR82BE86TARDH78G) · [Queue versus direct-call ADR](https://marke.st/p/01M33QKM91QDT2CMVXKJBSNYDF) · [Fictional incident review](https://marke.st/p/01M33QFHSCM3NXS3YB1YQ1G02K)

---
*Published with [Marke.st](https://marke.st). Keep the proposal, code, and review questions together.*