MLOps & GPU Infrastructure Interview Questions

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.

A model provider is a single dependency with a shared failure domain across every customer they have. Treating it like a database you cannot fail over from is how a product goes down for four hours.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

The training part is a job. Everything around it is the pipeline: data snapshots that are immutable, an evaluation gate with a threshold set in advance, and adapters versioned so a bad one is a routing change away from gone.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A model is an artifact with a lineage, an evaluation record and an owner, or it is a file somebody copied to a bucket. The registry exists so that the second one cannot reach production.

Before blaming the model, check whether it is being fed the same thing it was trained on. Most of the gap is a pipeline difference, and pipeline differences are a platform problem with a platform fix.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.

A release-automation design question with a specific trust boundary. The novel risk is prompt injection: untrusted text reaching a privileged agent, where the text is the instruction and your usual input validation does not apply.

Untrusted text reaching a privileged agent is the whole vulnerability. It is the pull_request_target mistake again, with a model in the middle that will follow instructions it finds in the data.

The valuable part is the first ten minutes of context gathering, which is tedious and mechanical. The dangerous part is anything that changes production while a human is still working out what is wrong.

Generated infrastructure code is plausible, fluent and confidently wrong in specific ways. The volume goes up, the reviewer attention per line goes down, and the controls that used to be optional stop being optional.

Every team standing up its own tool server produces a permission surface nobody can enumerate. The gateway exists so that connecting an agent to a system is a reviewed act rather than a config file in somebody's repository.

Per-token pricing looks expensive until you compute what an idle GPU costs at 3am. Break-even depends on workload volume and duty cycle, measured throughput, and the engineering cost on both sides.

GPU cost attribution needs allocation records as well as utilization. Separate exclusive reservations from borrowable quota, then reconcile team charges and idle capacity to the actual bill.

Nobody is waiting, which changes every constraint. Latency stops mattering, throughput per dollar becomes the only metric, and the design converges on something that looks like a data pipeline rather than a service.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

GPUs are the default for reasons that hold for large generative models and not for everything else. A small classifier on CPU can be cheaper, more available and easier to operate, and knowing where the crossover sits is worth more than knowing the newest card.

A model artifact is code that runs in your cluster with GPU access. Most of the supply-chain controls you already run for containers apply, and the file format adds one of its own.

Shared batching puts two tenants' data in one process on one device. That is defensible, and it is not what most customers assume they bought, so the answer has to be specific about which boundary exists where.

A mandatory gateway can centralize limits and filtering, while applications authorize business actions. Neither a prompt nor a PII classifier guarantees that sensitive data stays private.

Most of the obligation lands on the product, but three parts land on infrastructure: an inventory of what is deployed, evidence of what it did, and the ability to turn it off. Those are platform capabilities and they take months to retrofit.