Architecture scenarioMedium~10 minTesting & Quality / 18 of 20

Interview question

How would you test a background worker or message consumer that may receive the same job more than once and can crash partway through processing?

Test background workers and message consumers

Tests durable background processing across duplicate delivery, retries, acknowledgement timing, crashes, poison messages, and observable completion.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I would unit-test the handler's business decisions, then add integration tests around durable state and delivery behavior. The main cases are successful processing, the same message delivered twice, a transient failure followed by retry, a permanent invalid message, and cancellation during shutdown. Assertions would verify the business result and the stored processing state so duplicate delivery cannot create a second charge, record, or notification. Time, identifiers, and retry delays would be controllable rather than based on wall-clock sleeps.

Full answer below

Practice focus

Idempotency, retry behavior, broker semantics, durable state, fault injection, eventual assertions, dead-letter handling, and observability.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Background processing fails across time and process boundaries. Useful tests need to prove the durable outcome, not merely that a handler method returned without throwing.
I would unit-test the handler's business decisions, then add integration tests around durable state and delivery behavior. The main cases are successful processing, the same message delivered twice, a transient failure followed by retry, a permanent invalid message, and cancellation during shutdown. Assertions would verify the business result and the stored processing state so duplicate delivery cannot create a second charge, record, or notification. Time, identifiers, and retry delays would be controllable rather than based on wall-clock sleeps.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
Where broker behavior matters, I would use an isolated real broker or a test container instead of a mock that can only call the handler once. The tests would cover acknowledgement after commit, visibility or lock expiry, concurrent delivery, ordering assumptions, retry limits, and dead-letter metadata. They would wait on an observable state with a bounded timeout rather than sleeping for a guessed duration. Message contracts and database constraints would participate in the test because idempotency often depends on a durable message id or business key.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I would inject failures at the dangerous boundaries: after the database commit but before acknowledgement, after an external call but before local state is saved, and during process restart. Those cases reveal whether the design can resume or reconcile instead of relying on exactly-once execution. The test harness would capture correlation ids, attempts, final status, and emitted telemetry so failures remain diagnosable. I would keep a small number of these slower fault tests for critical consumers and leave exhaustive domain combinations in fast handler tests.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Calling the consumer method directly and treating that as broker coverage.
Using fixed sleeps to wait for eventual work.
Testing retry count without checking duplicate business effects.
Ignoring acknowledgement timing and process termination.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

A crash after commit causes the next delivery to repeat the effect.
Poison messages retry forever or disappear without evidence.
The worker has no durable processing or correlation state.
Tests pass only when consumers run serially in one process.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

How would you reproduce a crash after the database commit?
When is a real broker necessary in the test suite?
How would you assert eventual completion without making the test flaky?

Continue the section

Next in Testing & Quality: Migration Compatibility Tests

Step 19 of 20

Test database migrations and rolling compatibility

Opening drill

Architecture scenarioMedium~10 minTesting & Quality / 18 of 20

Interview question

How would you test a background worker or message consumer that may receive the same job more than once and can crash partway through processing?

Test background workers and message consumers

Tests durable background processing across duplicate delivery, retries, acknowledgement timing, crashes, poison messages, and observable completion.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Idempotency, retry behavior, broker semantics, durable state, fault injection, eventual assertions, dead-letter handling, and observability.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Background processing fails across time and process boundaries. Useful tests need to prove the durable outcome, not merely that a handler method returned without throwing.
I would unit-test the handler's business decisions, then add integration tests around durable state and delivery behavior. The main cases are successful processing, the same message delivered twice, a transient failure followed by retry, a permanent invalid message, and cancellation during shutdown. Assertions would verify the business result and the stored processing state so duplicate delivery cannot create a second charge, record, or notification. Time, identifiers, and retry delays would be controllable rather than based on wall-clock sleeps.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
Where broker behavior matters, I would use an isolated real broker or a test container instead of a mock that can only call the handler once. The tests would cover acknowledgement after commit, visibility or lock expiry, concurrent delivery, ordering assumptions, retry limits, and dead-letter metadata. They would wait on an observable state with a bounded timeout rather than sleeping for a guessed duration. Message contracts and database constraints would participate in the test because idempotency often depends on a durable message id or business key.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
I would inject failures at the dangerous boundaries: after the database commit but before acknowledgement, after an external call but before local state is saved, and during process restart. Those cases reveal whether the design can resume or reconcile instead of relying on exactly-once execution. The test harness would capture correlation ids, attempts, final status, and emitted telemetry so failures remain diagnosable. I would keep a small number of these slower fault tests for critical consumers and leave exhaustive domain combinations in fast handler tests.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Calling the consumer method directly and treating that as broker coverage.
Using fixed sleeps to wait for eventual work.
Testing retry count without checking duplicate business effects.
Ignoring acknowledgement timing and process termination.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

A crash after commit causes the next delivery to repeat the effect.
Poison messages retry forever or disappear without evidence.
The worker has no durable processing or correlation state.
Tests pass only when consumers run serially in one process.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

How would you reproduce a crash after the database commit?
When is a real broker necessary in the test suite?
How would you assert eventual completion without making the test flaky?

Continue the section

Next in Testing & Quality: Migration Compatibility Tests

Step 19 of 20

Test database migrations and rolling compatibility

How would you test a background worker or message consumer that may receive the same job more than once and can crash partway through processing?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue the section

How would you test a background worker or message consumer that may receive the same job more than once and can crash partway through processing?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue the section