Datadog SRE / Platform Engineer interview questions

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

Most candidates can recite the flags. The scored part is proving you can capture on a saturated box without adding load, then read the file afterward like evidence.

Everything top, htop and your APM agent display is derived from procfs anyway. Knowing the raw files turns any bare SSH session into a monitoring console.

tcpdump copies packets to userland; eBPF aggregates inside the kernel next to the event. Know which hook runs where, what a verifier log is telling you, and when XDP drops for free.

X-Forwarded-For, PROXY protocol v1 and v2, and the trust boundaries between them. Get the layer wrong and your rate limiting, geo rules and audit trails all key off the balancer's address.

Anycast turns BGP convergence into your failover mechanism. Sessions stick to the site they landed on, monitoring must go per-site, and a route you cannot withdraw cleanly is worse than DNS.

Strip the marketing and Aurora is a rewrite of the storage layer: the log is the database, committed to a quorum across three AZs. Know when that machinery pays and when it just bills.

Quotas are invisible until launch day. Treating them as tracked inventory with utilization alerts beats discovering them in a postmortem.

Four-hour applies with throttling errors share one root cause more often than not. Diagnose which calls burn the quota, split the state along dependency seams, and touch the parallelism flag last, not first.

Everyone can recite the three. The scoring answer is what each failure causes, and why a badly configured liveness probe is more dangerous than none at all.

A practical question with a precise answer, and the size is not the real cost. Pull time on every node, registry storage, and attack surface all scale with it.

Cilium became the default CNI on major managed platforms for a reason that is mechanical rather than fashionable. The iptables datapath degrades with service count, and eBPF replaces a linear scan with a map lookup.

The friendliest DaemonSet question there is, and it still separates people who have operated one from people who have read about one. The scheduling rules hide the traps.

The kubelet is a local reconcile loop with its own event stream, probe workers and eviction brain. Staff-level candidates can narrate it and explain why Ready status can reflect either reported health failure or missing heartbeats.

Dockershim is gone and the CRI contract decides what a runtime must do. The senior-level part is the shim model: a supervisor process that owns exit codes and stdio so the daemon can die without taking workloads with it.

Controllers should never poll, yet most home-grown operators do. This question separates people who run controllers at ten-thousand-object scale from those who copied a tutorial: watch semantics, cache behaviour, and which LISTs touch etcd.

Exec into a distroless image and you get nothing, because there is nothing to exec into. Ephemeral containers inject tooling into a running pod without restarting it, and the security caveats are what interviewers really listen for.

Tenancy is a dial from policy-based soft isolation to hard cluster boundaries with a virtual-control-plane middle ground. The scored answer picks by blast radius requirements and names what each model still leaves shared.

Two cleanup mechanisms hide behind one taint: image garbage collection removes unreferenced images by LRU at threshold crossings, while eviction kills pods. Candidates who conflate them fail this question quietly.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Exit codes, the previous-container log, and a little-known file at /dev/termination-log that carries a container's last words into kubectl output. Scale triage starts with classification, not with opening pods.

Best-effort objects, one-hour TTL, deduplicated by count, invisible unless shipped out. Events narrate scheduling and lifecycle decisions; treating them as durable records is the expensive mistake.

Route attachment and cross-namespace backend access have different owners. Use route status to find the failing boundary before broadening any permission.

Reciting metric names scores low. Naming the definition decisions that corrupt each one, and the gaming that starts when targets appear, scores high.

Dependency bots fail as noise generators or silent auto-mergers. Tiering by risk, weekly batching, and treating bot PRs with a real review SLA is what keeps updates flowing.

The arithmetic takes ten seconds and most candidates stall on it. The stall is the signal: it means error budgets are something you have read about rather than used.

Every candidate can define chaos engineering. Far fewer can say what they would break first, and almost none volunteer the abort condition, which is the thing that makes it a programme rather than an incident.

The observability failure that scales with your success. Cardinality is the cost driver in every metrics system, and the fix has to be structural or it comes back next quarter.

A question with a right answer and a subtle reason. Histogram versus summary is the part that catches people, and it turns on whether you can aggregate across instances.

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

OpenTelemetry is an API and a wire protocol, not a product. Candidates who promise instant multi-vendor bliss have never run the collector in production or negotiated with a vendor who knows it.

Most candidates recite the pull model and stop. The interviewer asking this wants the storage engine story, because that is where Prometheus hurts in production: crash recovery, cold queries, disk death.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

Routing rots because orgs change faster than configs. The senior answer treats ownership as data resolved from a catalog at fire time, with drift detection on the routing itself.

Attribution is an incentive-design problem: own-team labels at ingest, unit pricing per signal, monthly showback before any chargeback, sanctioned levers so teams can actually respond.

Detectors earn their keep on seasonal aggregates and fleets of similar series, run in shadow mode with measured precision. They lie on low-volume series, bursty workloads and regime changes.

One surface cannot serve the responder and the review. Tier dashboards by the question each audience asks, and make every tile answer in five seconds or link one click deeper.

Sampling keeps proportions and loses rare individuals; filtering keeps chosen classes complete and loses whatever you misclassified. Decide retention at write time, keep counters always, rehydrate the rest.

Monitoring dies silently: stuck alert queues, full TSDBs, collectors killed inside the same blast radius. Layer independent checks and a heartbeat so absence of signal pages louder than any red dashboard.

You have added a dependency with a latency distribution an order of magnitude wider than anything else in your stack, whose failure mode is a plausible wrong answer rather than an error.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

The most-asked scenario question in DevOps interviews, and the one where candidates most often list commands instead of reasoning. Empty logs is the clue, not the obstacle.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Signing proves what shipped, not what runs. The strong answer covers syscall-level detection, the signals that matter, tuning against alert fatigue, and the response path once a sensor fires.

Discover, scan, deduplicate, prioritise with context, route with SLA clocks, verify, measure. Most programmes rot at dedup and ownership: five scanners produce five conflicting queues nobody trusts.

Everyone names token bucket. The scored part is the rest: drift between nodes, the 429 contract with clients, and whether your quota service should fail open or closed.

Every overloaded system dies the same way: queues turn load into latency, latency into timeouts, timeouts into retry storms. Tiered shedding, deadline propagation, and retry budgets are the design.

Everyone sizes a metrics system by samples per second and everyone is wrong. The constraint that actually kills these platforms is a single label added on a Friday.

A disputed invoice needs a reproducible calculation. Retain raw usage events and versioned pricing so each line can be explained and corrected.

The hard parts are not search. They are deciding what to keep before you pay to store it, isolating one team's query from everyone else's, and making the cost visible to the people generating it.

The difference between scaling node groups and provisioning nodes is the whole answer. One picks from shapes you defined in advance; the other reads the pending pods and buys what they need.

Everyone quotes the discount. Fewer candidates can describe what happens when a notice arrives late or a worker disappears without completing shutdown and why some workloads shrug it off while others lose hours of compute.

Pay-per-use sounds free until the invoice arrives. The candidates who impress decompose the bill into invocations, gigabyte-seconds and the lines nobody models, then locate the honest crossover.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Culture change with receipts. Interviewers score whether you built mechanisms, credited messengers, shipped postmortem actions, measured report latency, rather than reciting the blameless poster.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

Asked in nearly every senior Kubernetes loop, and most candidates answer with a feature list. The scoring answer names the three problems it solves and then argues for not adopting it.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

The question that finds out whether you know Kubernetes networking or just use it. The model has three rules and everything about CNI follows from them.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

The scheduler is behaving correctly, which is the part that confuses people. It optimises placement at admission time and never revisits, so a cluster drifts out of balance by design.

Both solve the same surface problem and they disagree about how. The answer that scores names what each one makes hard, and notices that the GitOps tool you use changes the calculus.