TL;DR: Instrument the four golden signals per endpoint, the saturation of everything that kills the service (pool depths, queue lag), and dependency calls as spans with context propagated into logs. Wire exemplars so a metric spike points at real traces, and add a synthetic probe before launch. Skip the cardinality traps.
How to approach it
Organise by question rather than by tool: what will someone need to know at 3am, and what evidence answers it fastest? Mentioning trace-context propagation and exemplars early signals you have debugged rather than merely installed dashboards.
A strong answer
Golden signals per endpoint, not per service. Service-level averages hide the endpoint that breaks. For each significant route: latency as a distribution (histogram buckets, never just an average), traffic rate, errors split by kind (client, server, dependency, timeout), and saturation of the internal resources that actually kill this service: connection-pool checked-out count and waiters, queue depth and age, cache eviction rate, thread-pool utilisation. RED for request-handling, USE for resources; scope both to the critical paths and validate coverage before launch.
The linking layer is the differentiator. Three data sources without correlation are three tabs open at 3am:
- Request-scoped logs carry available
trace_id,span_idand service version. Include a tenant identifier only where necessary and permitted by the data policy, because structured logging without request context is barely better than none. - Context propagates across every boundary: HTTP headers, message queues, scheduled job wrappers. One missing propagation point severs the chain exactly at the failing hop.
- Exemplars connect metrics to traces: a histogram bucket records which actual trace IDs fell in it, so a latency spike on the graph clicks through to a real slow request. This single feature collapses triage time from "query logs hoping" to "look at five examples".
Dependencies are first-class citizens. Every outbound call (database, cache, third-party API) becomes a span with peer name and outcome. The first incident of every service's life is almost always a dependency behaving differently than assumed, and per-dependency success/latency panels make that visible in seconds.
Events that explain state changes. Deploys, config changes, feature-flag flips and scale events annotated onto dashboards. These markers help test whether a regression began with a specific change.
Alerts from day one, even with placeholder thresholds. A burn-rate alert against a provisional SLO beats nothing for the first month, because the alternative is discovering gaps during the outage. Add one synthetic probe hitting the public endpoint every 30 seconds: it detects what internal metrics cannot, namely that users are failing while servers look busy.
What deliberately stays out. No per-user metric labels (cardinality death; use logs or traces for per-entity questions). No dashboard sprawl: one owned dashboard per service with template variables beats fifteen personal ones nobody maintains. Include critical business correctness on day one, such as successful order completion or duplicate-charge detection; HTTP success alone can hide a broken product.
The position worth stating: telemetry is part of the definition of done for the first release, not a hardening phase. The team that instruments day one spends its first incident fixing the cause instead of building the microscope.
What interviewers probe next
"How do you keep cardinality bounded?" Labels only for dimensions you will group by in alerts (route, method, outcome class); high-cardinality identifiers live in logs and traces where storage is linear, not combinatorial.
"Sampling strategy?" If retaining later errors is required, send candidate traces to a tail sampler before any head drop; capacity and decision deadlines still bound retention. Random head sampling is cheaper but can miss rare errors, a tradeoff to state explicitly.
"What makes a dashboard good?" Someone can diagnose the top three failure modes of this service from it in under five minutes. If a panel does not serve that test, it is decoration.
Common mistakes
Instrumenting CPU and memory and calling it observability. Host graphs say nothing about why requests fail.
Tracing enabled but context not propagated through the queue library, leaving orphan spans at the exact async boundary where the bug lives.
Dashboards built after the SLO argument. Metrics chosen for alerts force the useful subset; dashboards chosen first collect everything and answer nothing.
References