TL;DR: GPUs are indivisible, their memory cannot be overcommitted, and they are expensive enough that idle time dominates the bill. So the design is queue-first rather than scale-first: separate latency-sensitive serving from batch, keep a warm minimum for serving, scale batch to zero, and pack models onto devices with MIG where the model is small enough to share.
How to approach it
Open with the three properties that make GPU different from CPU, because every design decision follows from them. Then separate the two workload classes, since treating them the same is the main error.
A strong answer
Three constraints set the shape. A GPU is allocated whole unless you partition it explicitly, so bin-packing intuition from CPU does not apply. Without sharding, offload or a smaller representation, a model requiring 80 GB of device memory cannot load into 48 GB. Quantization and CPU offload may make it fit, at quality or latency costs that need testing. And the hourly cost is high enough that utilisation, not headroom, is the thing you optimise, which inverts the usual capacity instinct.
Split the workloads first. Latency-sensitive inference and batch or training have opposite requirements. Serving needs warm capacity because cold-starting a large model means pulling tens of gigabytes of weights and loading them into device memory, which is minutes, not seconds. Batch tolerates queueing and should scale to zero. Running them in one pool means batch starves serving at exactly the wrong moment, so separate node pools with taints, and priority classes so serving preempts batch if they ever share.
Serving tier. vLLM or a comparable server, and the reasons are worth naming: continuous batching keeps the device busy by admitting new requests into an in-flight batch rather than waiting for a batch to complete, and paged attention manages the KV cache in fixed blocks so memory is not fragmented by variable-length sequences. Those two are what take utilisation from poor to good.
The scaling signal must not be GPU utilisation, as the sole demand signal: a device can read 100 percent while doing very little useful work. Scale on queue depth or time-to-first-token, both of which track the experience you care about. Size warm replicas for both lead-time demand and failure tolerance. One replica avoids a cold start but is a single point of failure; use multiple replicas across suitable failure domains when the availability target requires it.
Multiple models. Three options by size. Large models get dedicated devices or tensor parallelism across several. Medium models share a device through MIG, which gives hardware-level memory isolation rather than time slicing, and isolation matters because one tenant OOMing another is otherwise possible. Small or infrequently used models go behind a multi-model server that loads on demand and evicts by least-recently-used, accepting a cold start for the long tail.
Cost ceiling. Scale batch to zero and run it on spot or preemptible capacity with checkpointing, which is where most of the savings are. Commit to the baseline serving capacity and burst on demand. Attribute cost per model and per team, because unattributed GPU spend is never reduced. And set a queue with admission control rather than autoscaling without limit, so the ceiling is enforced by refusing work instead of by a surprise invoice.
What interviewers probe next
"Why not scale on GPU utilisation?" Because it measures whether kernels are running, not whether they are doing useful work, and it does not reflect queueing at all.
"How do you handle a 70B model landing on the wrong hardware?" Node labels from GPU feature discovery plus affinity in the short term, resource claims describing memory and model in the longer term, which is what DRA exists for.
"What dominates the latency?" For generation, memory bandwidth rather than compute, which is why batch size and KV cache management matter more than raw FLOPS.
Common mistakes
Applying CPU autoscaling patterns unchanged, which produces a platform that is either always cold or always expensive.
Ignoring cold start. Model load time is the constraint that decides whether scale-to-zero is available to you.
Treating time slicing as equivalent to MIG. Slicing shares compute without isolating memory, so it is a possible choice for trusted workloads with measured interference, but unsuitable where hard memory isolation is required.