Databricks SRE / Platform Engineer interview questions

The swap-used graph is the least informative memory metric on the box. Here is what swappiness actually weights, which numbers to watch instead, and the policy worth defending.

Multiplexing removes application-level queuing and hands the problem to TCP. Here is where the blocking really lives, the loss math behind it, and the four levers that help.

Absolute caps versus latency-target protection, systemd IO attributes versus raw io.max, and why ionice lies to you. The staff-level answer assigns the job its own slice with intent.

Large object uploads are a reliability design problem, not a bandwidth one. Parts, retries and the completion call decide whether a flaky link costs one chunk or the whole night.

Queue, log and router solve different problems that all get called events. Picking by workload shape beats picking by whichever service the team already knows.

Quotas are invisible until launch day. Treating them as tracked inventory with utilization alerts beats discovering them in a postmortem.

A cluster boundary does not always create a new cloud identity. Inspect the workload pool and principal identifier before assuming a leaked key.

General-purpose IaC earns its complexity where application engineers own infrastructure and benefit from typed components and unit tests. It disappoints when the pitch was freedom: the engine still builds a graph, still keeps state, and still forbids most of the clever runtime logic people expect to write.

Provider upgrades are a scheduled platform product: automatic version inventory, a canary stack inside days, automation that fans out pinned bumps with plan output attached. Estates without this routine get upgraded by deprecation deadlines.

Registries live or die on upgrade ergonomics, not on publishing features. Green example plans gate every tag, automation delivers upgrade PRs with plan output to consumers, and retirement is a census plus a date.

The pitch is continuous reconciliation instead of a plan someone runs. The cost is that your cloud estate now depends on a cluster, and that drift correction without a plan step is a very different risk posture.

A mock can confirm that a module requests private storage while missing the IAM denial that stops creation. Separate configuration contracts from provider behavior, and make the costly tests deliberate.

The single most common Kubernetes interview question, and the one where a vague answer is most obvious. Every stage you name is a place the interviewer can ask you to debug.

Both run pods, so the answer sounds obvious until an interviewer asks what happens when you delete one. The guarantees are narrower and more specific than most candidates think.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

Almost always a finalizer, and the fast fix is the one that loses data. The safe answer explains what the finalizer is protecting against before removing it.

The scheduler is behaving correctly, which is the part that confuses people. It optimises placement at admission time and never revisits, so a cluster drifts out of balance by design.

A specific configuration trap with a clean explanation. Both controllers act on the same signal from opposite directions, and the result oscillates.

Default-deny egress breaks name resolution before it breaks anything else, and every connection times out confusingly. The practitioners are the ones who test with positive AND negative probes and know their CNI's fine print.

The senior tells are level-triggered reconciliation and finalizer semantics. The partition question separates people who have written a controller from people who have installed one.

Namespaces partition resources, not trust. This question finds the people who know which sysctls are namespaced, which ones change every tenant at once, and when only a second kernel will do.

Multiple schedulers are supported, rarely understood, and dangerous in a specific way. The strong answer names the deadlock that gang scheduling fixes and the capacity race it can open.

Controllers should never poll, yet most home-grown operators do. This question separates people who run controllers at ten-thousand-object scale from those who copied a tutorial: watch semantics, cache behaviour, and which LISTs touch etcd.

SSH plus yum is how outages start. The senior answer replaces nodes from versioned images on a cadence, orchestrates with surge capacity and PDBs, and can explain the kernel-skew problem that makes fleet homogeneity a feature.

Two artifacts hide under 'Kubernetes backup' and they recover different things. Interviewers grade the misses: volume bytes, encryption keys, and everything that never lived in etcd at all.

Tenancy is a dial from policy-based soft isolation to hard cluster boundaries with a virtual-control-plane middle ground. The scored answer picks by blast radius requirements and names what each model still leaves shared.

Driver, container toolkit, device plugin, DCGM and discovery must version-match on every node. The operator reconciles that ladder from one ClusterPolicy CR; MIG, upgrades and host kernels are where it bites.

Changing pod resources historically required recreation, which made vertical autoscaling disruptive for workloads with expensive recovery. In-place resize changes that, with limits worth knowing.

A billion-row backfill is a deployable, throttled, resumable job with checkpoints and verification gates. Ad-hoc scripts against prod are how it becomes an incident.

Build graphs identify affected targets only when their dependencies are complete. The ladder from build-everything to content-addressed caching, hermeticity audits, and where undeclared inputs rot correctness.

For scheduled and stateful workloads the exposure unit is records and time, not requests: shadow runs on real input, row-level diff budgets, checkpoint compatibility, one-cycle warm standby.

Most candidates discuss caches as a speed feature. The staff-level answer treats write access to a shared cache like deploy credentials and proves hermeticity with sampled cold rebuilds.

Runner migration questions hide two different exams: whether a PR job can steal your cloud credentials, and why the fleet idles at half price all night. Isolation first, economics second.

The code did not change, the data did. That inverts what triggers the pipeline, what the tests assert, and what a rollback means.

Request-shaped SLIs quietly fail on data platforms. The strong answer defines freshness, completeness and deadline SLIs on consumer-visible artifacts, not job success rates.

One disk metric explaining a whole cluster's weirdness is the point of this question. Candidates who restart etcd first turn a slow cluster into a down one.

More partitions should mean more throughput, so why did the backlog deepen? Because rebalance behavior depends on the group protocol, and the way consumers divide work decides who pays for it.

Fork failures can reflect PID limits or memory allocation failure even when host RAM looks available. Find which ceiling broke, cap PIDs per cgroup so one team's leak costs one pod, and fix the unbounded spawn.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A rebuilt index and a wrong answer means the staleness is somewhere else: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

Escapes ride kernel bugs, privileged configuration and host mounts, not namespace magic. Cut the syscall and capability surface first, sandbox only what the trust boundary demands, and build nodes expecting compromise.

Trace a retry through the broker, database and external provider. The guarantee changes at each boundary, and a recorded intent does not prove a payment or email completed.

Cron fires at wall-clock times; a scheduler reasons about data intervals and blocked frontiers. The scored details are per-partition idempotency and catch-up semantics.

Teams need comparable evaluations before they approve a model change. Design versioned test data, recorded runs and release thresholds that a pipeline can enforce.

A shared cluster produces one bill and no answer. Attribution is a measurement problem first and a cultural one second, and the number that changes behaviour is not the one finance asked for.

The difference between scaling node groups and provisioning nodes is the whole answer. One picks from shapes you defined in advance; the other reads the pending pods and buys what they need.

Everyone quotes the discount. Fewer candidates can describe what happens when a notice arrives late or a worker disappears without completing shutdown and why some workloads shrug it off while others lose hours of compute.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

Per-token pricing looks expensive until you compute what an idle GPU costs at 3am. Break-even depends on workload volume and duty cycle, measured throughput, and the engineering cost on both sides.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

Nobody is waiting, which changes every constraint. Latency stops mattering, throughput per dollar becomes the only metric, and the design converges on something that looks like a data pipeline rather than a service.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

Shared batching puts two tenants' data in one process on one device. That is defensible, and it is not what most customers assume they bought, so the answer has to be specific about which boundary exists where.

Use a real example if you have shipped an AI feature. If you have not, say so and offer a clearly hypothetical response about evaluation, containment and customer communication.

Asked in almost every cloud screen, and most candidates recite the table without ever naming the failure it causes. The ephemeral port range is the whole answer.

Turns the standard autoscaling definition into the problem it actually causes. Almost every candidate can define an HPA; far fewer can say why theirs did not save them.

The question that separates console users from people who have debugged a permissions failure across accounts. Evaluation order is deterministic and knowing it makes AccessDenied readable.

A design question with a clear answer at this scale, and the reasoning is what scores. The three options solve different problems and peering and Transit Gateway connect networks while PrivateLink exposes services.

Most candidates can sell serverless. The question asks you to argue against it, which needs a real understanding of the execution model rather than the marketing.

Trace the route and return path before changing rules. First establish whether the failing connection uses IPv4, IPv6 or a private service endpoint.

The junior-filter question that still separates candidates who understand the model from those who memorised a definition. The answer is three mechanical differences and one economic one.

S3 access failures are rarely about the bucket policy alone. The caller, object ownership and network path determine which authorization checks apply. Start with the failed request and trace changes across those boundaries.