Debugging scenarioHard~12 minObservability, Data & Background Failures / 7 of 7

Interview question

A customer reports duplicate reports, emails, or charges after a background worker retried messages. How would you debug and fix it?

Debug duplicate background processing after retries

Traces duplicate side effects across message delivery, lock ownership, settlement, durable job state, and external idempotency, then repairs affected users safely.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I identify one duplicated charge, email, or report and trace it back to the job id, message id, delivery attempts, worker instances, and external operation ids. I look for a crash, lock expiry, or failed acknowledgement after the side effect completed. Before replaying anything, I stop the unsafe path, preserve message and provider evidence, and determine which users need compensation or correction.

Full answer below

Practice focus

Idempotency, at-least-once delivery, message settlement, visibility or lock timeout, job state, and compensating actions.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
A worker can finish the business side effect yet fail before message settlement or durable completion. Redelivery is correct broker behavior, so duplicate safety must exist at the side-effect boundary.
I identify one duplicated charge, email, or report and trace it back to the job id, message id, delivery attempts, worker instances, and external operation ids. I look for a crash, lock expiry, or failed acknowledgement after the side effect completed. Before replaying anything, I stop the unsafe path, preserve message and provider evidence, and determine which users need compensation or correction.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
The fix depends on the side effect. A database write uses a unique business key and atomic job transition. A payment call reuses a provider idempotency key and stores the provider reference. Email may need a durable send record if duplicates carry real cost. The worker completes the broker message only after durable local state commits, but settlement can still fail, so redelivery checks that state and returns safely. Lock duration and renewal match measured work, while long jobs are split into checkpointed steps.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
In production I build a timeline from queue delivery and lock events, application traces, database transitions, and provider records. That distinguishes duplicate delivery from two producers creating distinct messages for one business operation. If an external timeout left the outcome unknown, a reconciliation query determines whether the side effect happened before another attempt is allowed. Recovery separates suppressing future duplicates from refunding, retracting, or notifying about effects already visible to users. Alerts track repeated attempts per job, lock loss, settlement failures, duplicate-key hits, and provider idempotency responses. Regression tests force a crash after the external success and before local completion, plus a successful local commit followed by failed message settlement.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Assuming every duplicate effect came from broker redelivery.
Replaying messages before checking external provider state.
Completing the message before the durable local transition.
Treating a longer lock timeout as a substitute for idempotency.

Red flagsCandidateRed flagsSignals that an interviewer may treat as risky.Open details

Details

Cannot explain at-least-once delivery.
Has no job state model.
Only fixes the timeout and ignores idempotency.
Cannot describe compensation for duplicates already created.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

How would you design idempotency for email? For payments?
What happens if the worker crashes after the side effect?
How would you test retry safety?

Continue practice

Move to the linked follow-up, next path step, prerequisite, or deeper variant.