How would you choose retention for an event stream, and when does compaction help?

Set event retention and compaction for consumer recovery

Chooses event retention and compaction from consumer outage, replay, rebuild, audit, and disaster-recovery requirements rather than storage cost alone.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I start from recovery requirements. Retention must exceed the longest supported consumer outage plus detection and repair time. If projections rebuild from the stream, the retained or archived history must cover a full rebuild. Compaction helps when consumers need the latest state per key, such as the newest customer profile, but it may remove intermediate changes and therefore cannot replace a full audit history.

Full answer below

Practice focus

Retention windows, replay, storage cost, log compaction, state reconstruction, and consumer recovery.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Retention defines how far consumers can rewind; compaction preserves a latest value per key, not a complete history. Neither choice automatically creates an audit log or a permanent source of truth.
I start from recovery requirements. Retention must exceed the longest supported consumer outage plus detection and repair time. If projections rebuild from the stream, the retained or archived history must cover a full rebuild. Compaction helps when consumers need the latest state per key, such as the newest customer profile, but it may remove intermediate changes and therefore cannot replace a full audit history.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I classify the stream as transport, replay log, state-transfer log, event-sourcing history, or compliance evidence because each role needs different guarantees. A compacted topic keeps tombstones and latest keyed records according to broker policy, while an append-only archive may preserve immutable history more cheaply. Consumer recovery also depends on compatible schemas and external data referenced by old events. Retention that outlives those dependencies does not by itself make replay successful.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
In production each stream has an explicit recoverable window, owner, storage budget, and tested restore path. Alerts fire when oldest consumer lag approaches retention, leaving time to pause cleanup, restore from archive, or rebuild from an authoritative snapshot. I monitor segment growth, compaction lag, tombstone behavior, archival failures, and time required to replay at safe downstream throughput. Disaster exercises start a new consumer from a known checkpoint and prove that current code can process retained schema versions. Access controls and deletion policy account for personal or regulated data copied into long-lived events. The final design often combines broker retention for operational replay, periodic snapshots for fast rebuild, and object storage for longer evidence where policy permits.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Choosing retention only from current storage cost.
Assuming compaction preserves every historical change.
Keeping events longer than their schemas, referenced data, or privacy policy support.
Discovering during an outage that replay overwhelms downstream systems.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Cannot explain replay window.
Does not know what compaction removes.
Ignores storage and recovery trade-offs.
No consumer-lag alert tied to retention.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

When is compaction useful?
What happens if a consumer is down longer than retention?
When would you archive events outside the broker?

Continue the section