Netflix SRE / Production Engineer interview questions

A growing backlog of sockets waiting for the application to close its side usually points at lifecycle handling. Here is the hunt from ss output to the pool timeout that caused it.

Most candidates can recite the flags. The scored part is proving you can capture on a saturated box without adding load, then read the file afterward like evidence.

Everything top, htop and your APM agent display is derived from procfs anyway. Knowing the raw files turns any bare SSH session into a monitoring console.

Latency fine, loss low, link idle, yet a single copy takes hours. The arithmetic of bandwidth-delay product settles this question before any tool opens.

Multiplexing removes application-level queuing and hands the problem to TCP. Here is where the blocking really lives, the loss math behind it, and the four levers that help.

One CPU doing all the packet work is a topology failure, not a tuning mystery. RSS queues, MSI-X vectors, local_cpulist and irqbalance: the whole chain from symptom to pin.

Most throughput tests measure the tester, not the network. Single stream versus parallel, slow start, sender-side retransmits, and the CPU ceiling: how to produce a number you can act on.

Loss-based versus model-based congestion avoidance decides whether your WAN transfers fill the pipe or die on the first stray drop. Here is the operator's decision, not the RFC summary.

Absolute caps versus latency-target protection, systemd IO attributes versus raw io.max, and why ionice lies to you. The staff-level answer assigns the job its own slice with intent.

Spreading instances across availability zones is not the same as surviving an AZ failure. The candidates who know this can name the four coupling points that quietly re-unify your blast radius.

A busy date key can throttle while table capacity remains available. Check the actual key distribution and throttling reason before adding capacity or shards.

Fleet-wide replacement is a traffic-shaping problem: bake once, wave through in controlled batches, watch the alarms, and keep the exit open. In-place patching is the alternative and usually the regret.

Queue, log and router solve different problems that all get called events. Picking by workload shape beats picking by whichever service the team already knows.

Strip the marketing and Aurora is a rewrite of the storage layer: the log is the database, committed to a quorum across three AZs. Know when that machinery pays and when it just bills.

Four tiers, four bills. Stating RTO and RPO per tier turns a vague resilience wish into a decision leadership can actually price, and the testing story is where candidates fold.

Latency budgets decide architecture, and most teams buy edge compute before earning it with caching and protocol work. Know which rung of the ladder your problem actually needs.

A dead-letter queue with no owner is a landfill with a nicer name. Triage discipline separates teams that recover messages from teams that archive regret.

Data transfer is charged in several unrelated ways and reported as one line, so the first job is decomposition. Most surprises are cross-zone chatter between services that never needed to be on different nodes.

A green plan is evidence, not proof. Build the pyramid: lint in seconds, policy on every PR, real-cloud verification only where shared modules and money justify the burn.

Idempotence describes one run; convergence describes the fleet across time. Scheduled check-mode sweeps with diff reporting, alerts above a learned baseline, and a hard rule that incident hotfixes become pull requests the same day.

Four-hour applies with throttling errors share one root cause more often than not. Diagnose which calls burn the quota, split the state along dependency seams, and touch the parallelism flag last, not first.

Rotation is an expand-contract migration for credentials: stage the new secret, prove every reader picked it up, revoke the old one, and treat the revocation as the test. The outage risk lives in clients and connection pools, not in Terraform.

A registry stores repositories, which hold manifests, which point at content-addressed blobs. Candidates who can draw that hierarchy and the auth dance around it understand every pull failure they will ever meet.

The loop is easy; the statistics are not. This question finds engineers who can size a traffic slice against sample noise and name the false rollback that kills trust in automation.

Priority is one number with three separate effects: scheduling order, preemption victim selection, and eviction order under node pressure. Conflating those is the trap; designing tiers that survive both a full cluster and a hot node is the pass.

Utilization-based scaling is the wrong sensor when work arrives as backlog. The design questions are threshold math against drain rate, reaction-time bounds from polling, and why lag-driven fleets oscillate without generous targets and idempotent consumers.

Pods authenticate with JWTs minted per-pod now, not eternal Secrets. The scored answer separates authentication from authorization, explains audience binding, and treats automount as opt-in rather than ambient.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Hard versus soft rules, topology keys, weights that lose to everything else, and the IgnoredDuringExecution half nobody remembers. Placement shapes blast radius only when paired with disruption budgets.

A ladder from health-checked global DNS through endpoint mirroring to mesh east-west gateways. The deciding axis is trust boundary and failure isolation, not any mesh's feature matrix.

If every environment rebuilds, staging never validated the bytes production runs. The promotion model auditors and 2am responders both love, plus the costs nobody mentions until adoption.

Deployment and release are two decisions, not one. The flag patterns that buy you instant reversibility, and the retirement discipline most teams skip.

Rings, SLI gates and Git-expressed promotion: sync waves order resources inside a cluster, but fleet sequencing is an automation you design on top.

Reciting metric names scores low. Naming the definition decisions that corrupt each one, and the gaming that starts when targets appear, scores high.

For scheduled and stateful workloads the exposure unit is records and time, not requests: shadow runs on real input, row-level diff budgets, checkpoint compatibility, one-cycle warm standby.

One cadence for every team is the naive answer. Cadence follows constraints: store review for mobile, compatibility windows for APIs, contracts and embedded realities for trains.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

Deploying the application per pull request is a solved problem. Giving it data it can work against, and dependencies it can call, is where these projects stall and where the data-protection incident lives.

The arithmetic takes ten seconds and most candidates stall on it. The stall is the signal: it means error budgets are something you have read about rather than used.

Every candidate can define chaos engineering. Far fewer can say what they would break first, and almost none volunteer the abort condition, which is the thing that makes it a programme rather than an incident.

Alert fatigue is a design failure, not a discipline failure. The fix is a rule about what may page at all, and burn-rate alerting is how you implement it.

The roles question is really a delegation test: can you split commanding the incident from fixing it? Name the roles, the comms cadence, and the handoff discipline that keeps 3am coherent.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

Facilitation is mechanics: timeline before theories, blame converted into system questions, actions with owners tracked to closure. Include the defensive engineer and the exec who wants a head.

Where linear models break: compounding growth, step functions, peak days and saturation cliffs. Forecast per constrained resource, backtest, subtract lead time to get the real order-by date.

Detectors earn their keep on seasonal aggregates and fleets of similar series, run in shadow mode with measured precision. They lie on low-volume series, bursty workloads and regime changes.

MTTR is three clocks hiding in one number. Profile detection, diagnosis and fix separately from incident timestamps; attack whichever clock dominates, which is almost never rollback speed.

Their green page, your red SLO. Separate their true failure rate from the amplification your own client adds, buffer at the boundary, then escalate with a client-side scorecard instead of feelings.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

There is no status code and no caller waiting, so latency and error rate stop meaning what they meant. The signals that replace them are age, lag and completion, and they answer a question the request-response set never had to ask.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

A two-second dependency should cost you latency, not three services. The mechanics are arithmetic: Little's law plus retry amplification. The fix is a budget, not a ban.

NFS waits can stall shutdown and inflate load average while CPUs stay idle. Check the blocked kernel path before deciding whether signals or mount recovery can release it.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

Self-healing unavailability is a timeout arithmetic story. Pull the balancer's removal timestamps, lay them against the GC log, and the pause that ejects the instance will be sitting there in plain sight.

Purges act on exact cache keys, and the edge is full of variants the purge never named. Verify per variant at real POPs, then fix the key model instead of chasing URLs forever.

The automation stopped because it could not decide, which is the correct behaviour and an incomplete design. Somebody has to decide, and the interesting question is what evidence that decision needs.

Warn forever becomes wallpaper; block on day one gets the gate switched off. The durable answer is a severity-and-fixability matrix plus suppressions that expire.

Signing proves what shipped, not what runs. The strong answer covers syscall-level detection, the signals that matter, tuning against alert fatigue, and the response path once a sensor fires.

Break-glass exists to restore login when the identity provider is the thing that died. Few accounts, sealed hardware factors, an alert on every authentication, a printed runbook and a quarterly drill.

Escapes ride kernel bugs, privileged configuration and host mounts, not namespace magic. Cut the syscall and capability surface first, sandbox only what the trust boundary demands, and build nodes expecting compromise.

Choose by trust boundary, not fashion: mesh-issued mTLS inside a cluster, audience-bound tokens across teams and clouds, both together at scale. Static shared secrets are the thing being replaced.

Adoption is a product problem: templates with tested safe defaults, guardrails that start in audit mode, and metrics on paved-road share. Remove reasons to leave the road rather than writing memos.

The design round where the honest answer is mostly about data. Stateless traffic across regions is straightforward; the database is where active-active is either solved or pretended.

Every overloaded system dies the same way: queues turn load into latency, latency into timeouts, timeouts into retry storms. Tiered shedding, deadline propagation, and retry budgets are the design.

The app server that proxies uploads through itself is the design that fails this question. Direct-to-storage, validate after arrival, and make CDN invalidation almost never necessary.

Drawing an agent that polls a server is the easy third. The interview is about partial applies, fleet-versus-world state, and why rollback must be a pointer flip, never a reverse edit.

Ladders and player logic are the warm-up. Cache topology, egress arithmetic, and knowing which quality metric actually predicts churn are what separate the answers.

A question about method, not about tactics. Candidates who open with a list of levers lose to candidates who open with one question about the bill itself.

Four purchase instruments, one fleet, and a finance team asking why you did not commit to everything at maximum discount. The scoring rubric rewards the arithmetic of being wrong, not the discount table.

Percentage savings make every quarter look like a win and tell you almost nothing. The candidates who get hired bring a denominator tied to revenue and defend it under pressure.

Data transfer appears on nobody's architecture diagram and reliably appears on the invoice. The strong answer enumerates the taxed flows, ranks the levers, and knows which tax is worth paying.

The discount is real and so is the option you are selling. The question is what your usage looks like in year three, and the honest answer for most teams is that they do not know.

A budget miss can coexist with lower cost per request. Separate demand, usage intensity and price effects before assigning an efficiency target.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

Nobody is waiting, which changes every constraint. Latency stops mattering, throughput per dollar becomes the only metric, and the design converges on something that looks like a data pipeline rather than a service.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

An incident answer needs your actual decisions, the evidence behind them and the effect on users. A wrong turn belongs in the story only if it happened.

A question that punishes both extremes. A trivial mistake reads as evasion, a catastrophic one told carelessly reads as reckless, and the interviewer is really asking what you do in the twenty minutes after you realise.

Culture change with receipts. Interviewers score whether you built mechanisms, credited messengers, shipped postmortem actions, measured report latency, rather than reciting the blameless poster.

Arguing that reliability matters loses to a roadmap every time, because both sides agree it matters. What wins is converting it into the same currency features are argued in: a number, a cost, and a decision someone owns.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

Asked in nearly every senior Kubernetes loop, and most candidates answer with a feature list. The scoring answer names the three problems it solves and then argues for not adopting it.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

Everyone can recite the three. The scoring answer is what each failure causes, and why a badly configured liveness probe is more dangerous than none at all.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

The question that finds out whether you know Kubernetes networking or just use it. The model has three rules and everything about CNI follows from them.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

A practical question with a precise answer, and the size is not the real cost. Pull time on every node, registry storage, and attack surface all scale with it.