How would you keep observability useful without making telemetry too expensive or noisy?

Control trace sampling, metric cardinality, and observability cost

Explains sampling, high-cardinality labels, retention, cost, and keeping enough signal for incident diagnosis.

1Say itAnswer out loud first

2CompareCompare with Core answer

3RefineRefine with Stronger and Advanced

TL;DR

I would control volume with log levels, sampling, retention, and careful metric labels. Metrics should use low-cardinality dimensions like route template, status code, dependency name, region, or environment, not raw user ids, full URLs, or arbitrary search terms. Logs should keep important errors and decision points without dumping every request body.

Full answer below

Practice focus

Sampling, cardinality, retention, noisy logs, metric labels, rare errors, cost control, and incident usefulness.

Answer first

Answer path

Core answer first, then checkpoints toward a stronger answer.

Start
Core answer
Start here. This is the direct spoken answer to practice first.
Why this question matters
Observability can become its own production problem when labels explode, logs flood storage, or sampling hides the one trace needed during an incident.
I would control volume with log levels, sampling, retention, and careful metric labels. Metrics should use low-cardinality dimensions like route template, status code, dependency name, region, or environment, not raw user ids, full URLs, or arbitrary search terms. Logs should keep important errors and decision points without dumping every request body.
Checkpoint
Stronger answer
Optional upgrade with examples, trade-offs, and sharper phrasing.
For traces, I would sample normal traffic but keep more detail for errors, slow requests, important endpoints, and known incident windows. Tail-based sampling can preserve interesting traces when the backend supports it. I would also make sure critical business events and security events are not sampled away if they are needed for audit or support. Cost control should not erase the signals that explain user impact.

Review signals

Use these to catch weak phrasing, shallow reasoning, and risky claims.

Common mistakesCandidateCommon mistakesMistakes that weaken the answer or signal shallow understanding.Open details

Details

Using user ids, full URLs, or search strings as metric labels.
Sampling away all error traces.
Leaving debug logging on permanently in hot paths.
Keeping high-volume telemetry forever without a reason.

How would you keep observability useful without making telemetry too expensive or noisy?

Practice focus

Answer path

Why this question matters

Review signals

Follow-up prompts

Continue the section

Tech stacks

Skill areas

Suggest an improvement