TL;DR: Treat the driver as a node-level dependency with its own rollout, not as part of the application deploy. Use the GPU Operator so the driver, toolkit and device plugin move as one versioned unit, roll node pools rather than nodes, and rely on CUDA backward compatibility so applications built against an older runtime keep working on a supported newer driver. Verify the operator, GPU generation and framework support matrix before rolling.
How to approach it
Lay out the stack before the plan, because the whole difficulty is that four layers version independently. Then split the fleet by what an interruption costs: inference drains in seconds, a training gang can lose hours. The interviewer wants to hear that you know which direction the compatibility guarantee points.
A strong answer
The stack, bottom to top: the kernel module and userspace driver on the host, the container toolkit that injects device nodes and libraries into the container, the Kubernetes device plugin that advertises nvidia.com/gpu, the CUDA runtime baked into the image, and the framework build (PyTorch, vLLM) compiled against that runtime. CUDA backward compatibility lets existing applications run on supported newer drivers, so the driver can move before the images. Minor-version compatibility is a different guarantee: within a CUDA major family, a newer runtime can work on an older driver that meets the minimum requirement, with feature and PTX limitations. Across major families, check the supported driver or forward-compatibility package; do not assume the image will work. A mismatch can fail at container startup or when CUDA initialises. NVIDIA documents the compatibility directions and restrictions.
Make the stack one unit. The GPU Operator packages driver, toolkit, device plugin, DCGM exporter and node feature discovery, versioned together and tested as a set. Hand-installed drivers with a separately installed toolkit is where fleets diverge, and a fleet where nodes differ is a fleet where a failure cannot be reproduced. Pin the operator version in Git and upgrade it like any other platform component, with the manifest change reviewed and the rollout observable.
Roll pools, not nodes. On cloud, the cleanest mechanism is a new node group with the new image: surge capacity in, cordon the old group, let workloads move, then delete. That gives an instant rollback because the old group still exists. In-place driver upgrades require unloading the kernel module, which requires that no process holds the device, which means every GPU pod on the node must stop first. It works and it is slower and less reversible.
Order by cost of interruption. Inference nodes go first: drain semantics are already solved for them, the blast radius is one replica, and a problem shows up in TTFT within minutes. Training goes last and moves at checkpoint boundaries, cordoning the node so no new gang lands and draining when the current job saves. A node holding a 40-hour job is a node you upgrade next week, and saying that out loud is the correct answer rather than an admission of defeat.
Verify with load, not with nvidia-smi. The command printing a version proves the driver loaded. Burn-in proves the node works: DCGM diagnostics, an all-reduce benchmark compared against the fleet, and one real workload at production concurrency. Most driver-upgrade incidents are found by throughput regression, not by a failed start.
What interviewers probe next
"What if a framework needs a newer CUDA major?" Check the framework and GPU support matrix, then upgrade the driver first where required. A supported forward-compatibility package can cover some deployments; test the actual workload before relying on that exception.
"How do you handle a fleet with mixed GPU generations?" Separate node pools with distinct labels and separate rollouts. A driver release can be fine on one architecture and regress on another, so mixing them into one rollout removes your ability to attribute a regression.
"Rollback plan?" Keep the previous node group or machine image alive until the new one has held an SLO for a full traffic cycle including the weekly peak. Rollback is deleting the new group, not reinstalling a driver at 3am.
Common mistakes
Upgrading drivers with a node-level DaemonSet edit and no canary, then discovering the regression across the fleet at once.
Assuming a successful pod start means success. Version mismatches fail loudly; performance regressions do not.
Treating training and inference nodes as one fleet with one rollout policy, which either strands training jobs or holds the whole upgrade hostage to the longest run.