Architecture scenarioHard~12 minMessaging & Event-Driven Essentials / 16 of 16

Interview question

A payment request timed out, the order is still pending, and no webhook arrived. How would you determine what happened and repair the workflow without charging twice?

Reconcile an ambiguous distributed workflow

Repairs a workflow when a remote operation may have succeeded but local state, messages, and callbacks disagree.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I do not retry the charge blindly because the timeout means the outcome is unknown, not necessarily failed. I look up the payment using the original provider idempotency key or operation reference. If the provider confirms success, I record that result and advance the pending order idempotently. If it confirms failure, I allow the workflow to retry under the same business operation. If the provider cannot decide, the order remains in an explicit pending-review state.

Full answer below

Practice focus

Unknown outcomes, sources of truth, provider lookup, idempotency, workflow states, reconciliation jobs, monotonic repair, compensation, observability, and manual review.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Distributed calls can succeed remotely while their response, callback, or follow-up message is lost. The system needs a durable way to discover truth and move the workflow to a valid state without repeating an irreversible effect.
I do not retry the charge blindly because the timeout means the outcome is unknown, not necessarily failed. I look up the payment using the original provider idempotency key or operation reference. If the provider confirms success, I record that result and advance the pending order idempotently. If it confirms failure, I allow the workflow to retry under the same business operation. If the provider cannot decide, the order remains in an explicit pending-review state.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I give each checkout a durable operation id and store local attempts, provider references, workflow state, and observed evidence. A reconciliation worker scans operations that have remained pending beyond an expected window, queries the authoritative provider with bounded rate and backoff, and applies a monotonic state transition such as pending to paid, failed, or manual review. That transition and any follow-up outbox message commit together. Late webhooks pass through the same idempotent transition, so reconciliation and callbacks cannot complete the order twice.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
In production I define the source of truth for each fact: the provider owns settlement status, while the order service owns fulfillment state. Reconciliation never guesses from absence of a webhook. It records evidence time, provider response, decision, and repair action, but avoids creating an audit row for every harmless poll. Conflicting evidence, expired lookup windows, refunds, chargebacks, or irreversible fulfillment move to a queue with clear operator controls and compensation rules. Metrics track oldest ambiguous operation, reconciliation success and error rates, state divergence, repairs, duplicate-suppression hits, and manual-review age. Failure tests drop the provider response, delay the webhook, deliver it twice, crash after local repair, and run two reconcilers concurrently to prove one final business outcome.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Treating a timeout as proof that the remote operation failed.
Charging again with a new idempotency key.
Using webhook absence as the only reconciliation signal.
Letting repair, late callbacks, and operator actions bypass the same state-transition guard.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

No durable operation or provider reference.
Pending workflows can remain invisible forever.
Reconciliation can move completed state backward.
Manual repair has no authorization, evidence, or duplicate protection.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Which system is authoritative for payment and fulfillment?
How does a late webhook interact with reconciliation?
When does the workflow require compensation or manual review?
How would you test two reconcilers processing the same pending order?

Continue practice

Move to the linked follow-up, next path step, prerequisite, or deeper variant.

2 linked

Next interviewer turn

Continue the conversation with a linked follow-up drill.

Sagas & Compensation

Go deeper

Practice the next layer of the same subject.

Outbox Pattern

Opening drill

Architecture scenarioHard~12 minMessaging & Event-Driven Essentials / 16 of 16

Interview question

A payment request timed out, the order is still pending, and no webhook arrived. How would you determine what happened and repair the workflow without charging twice?

Reconcile an ambiguous distributed workflow

Repairs a workflow when a remote operation may have succeeded but local state, messages, and callbacks disagree.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

Full answer below

Practice focus

Unknown outcomes, sources of truth, provider lookup, idempotency, workflow states, reconciliation jobs, monotonic repair, compensation, observability, and manual review.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Distributed calls can succeed remotely while their response, callback, or follow-up message is lost. The system needs a durable way to discover truth and move the workflow to a valid state without repeating an irreversible effect.
I do not retry the charge blindly because the timeout means the outcome is unknown, not necessarily failed. I look up the payment using the original provider idempotency key or operation reference. If the provider confirms success, I record that result and advance the pending order idempotently. If it confirms failure, I allow the workflow to retry under the same business operation. If the provider cannot decide, the order remains in an explicit pending-review state.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
I give each checkout a durable operation id and store local attempts, provider references, workflow state, and observed evidence. A reconciliation worker scans operations that have remained pending beyond an expected window, queries the authoritative provider with bounded rate and backoff, and applies a monotonic state transition such as pending to paid, failed, or manual review. That transition and any follow-up outbox message commit together. Late webhooks pass through the same idempotent transition, so reconciliation and callbacks cannot complete the order twice.
Finish
Advanced answer
Optional deeper angle for production judgment and nuance.
In production I define the source of truth for each fact: the provider owns settlement status, while the order service owns fulfillment state. Reconciliation never guesses from absence of a webhook. It records evidence time, provider response, decision, and repair action, but avoids creating an audit row for every harmless poll. Conflicting evidence, expired lookup windows, refunds, chargebacks, or irreversible fulfillment move to a queue with clear operator controls and compensation rules. Metrics track oldest ambiguous operation, reconciliation success and error rates, state divergence, repairs, duplicate-suppression hits, and manual-review age. Failure tests drop the provider response, delay the webhook, deliver it twice, crash after local repair, and run two reconcilers concurrently to prove one final business outcome.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Treating a timeout as proof that the remote operation failed.
Charging again with a new idempotency key.
Using webhook absence as the only reconciliation signal.
Letting repair, late callbacks, and operator actions bypass the same state-transition guard.

Red flagsCandidateSigns the approach is fragileSignals that an interviewer may treat as risky.Open details

Details

No durable operation or provider reference.
Pending workflows can remain invisible forever.
Reconciliation can move completed state backward.
Manual repair has no authorization, evidence, or duplicate protection.

Follow-up prompts

Quick interviewer prompts inside this drill.

Follow-up questionsCandidateFollow-up questionsLikely next questions after the main answer.Open details

Details

Treat these as the next interviewer turn. Answer one out loud, then save the gap in your practice notes.

Which system is authoritative for payment and fulfillment?
How does a late webhook interact with reconciliation?
When does the workflow require compensation or manual review?
How would you test two reconcilers processing the same pending order?

Continue practice

Move to the linked follow-up, next path step, prerequisite, or deeper variant.

2 linked

Next interviewer turn

Continue the conversation with a linked follow-up drill.

Sagas & Compensation

Go deeper

Practice the next layer of the same subject.

Outbox Pattern

A payment request timed out, the order is still pending, and no webhook arrived. How would you determine what happened and repair the workflow without charging twice?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue practice

Next interviewer turn

Go deeper

A payment request timed out, the order is still pending, and no webhook arrived. How would you determine what happened and repair the workflow without charging twice?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue practice

Next interviewer turn

Go deeper