Replaying a failed load without doubling the warehouse bill

Bounded backfills, watermark discipline, and why “just rerun everything since Monday” is an expensive comfort phrase.

During incident weeks, the tempting move is a wide backfill. It feels decisive. It also multiplies warehouse compute and can reintroduce duplicates if idempotency was never designed in.

Prefer a bounded window

Identify the earliest watermark that failed verification. Replay from that stamp forward, not from an arbitrary “Monday.” Write the window into the incident note so the next person does not guess.

Check idempotency before the second attempt

If the load appends without a natural key or merge strategy, a replay will inflate counts. Stabilize with a temporary filter only after you know how duplicates would appear — and schedule the permanent fix in the remediation list, not in a sticky note.

Cost visibility

Ask finance or FinOps for a rough baht-per-slot figure before approving a three-day full rebuild. Operators make better calls when the cost of comfort is visible.

Incident Recovery Support engagements spend the first hours on this triage before touching transformation code.

← Back to Field Notes