DevOpsInterviewPrep logo
AI & GPU Infrastructure / 01
hardNewNVIDIADatabricksGoogle

Device plugin versus Dynamic Resource Allocation for GPUs: why did Kubernetes need DRA?

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: The device plugin model exposes GPUs as an opaque integer resource, with hardware distinctions supplied through resource names, labels and driver/topology integrations. DRA replaces the count with structured, requestable attributes, which is what lets the scheduler place a job on the right device instead of merely on a node that has one.

How to approach it

Start from the limitation rather than the feature. Name three concrete placements the old model cannot express, then describe what DRA changes. Interviewers here are checking whether you have felt the constraint or read the release notes.

A strong answer

Under the device plugin model, a GPU is advertised as an extended resource, nvidia.com/gpu: 1. The scheduler treats it exactly like a countable integer, the same way it treats a made-up resource. The count alone cannot express arbitrary device attributes, but existing deployments add model labels, MIG-specific resource names and topology hints.

rendering diagram…

Three constraints the count alone does not express. First, heterogeneity: a cluster with H100s and L40S nodes advertises the same resource name from both, so a job that needs 80GB of device memory can be scheduled onto a 48GB card and will fail at runtime with an out-of-memory error rather than staying Pending. The usual workaround is node labels from GPU Feature Discovery plus node affinity, which works but pushes scheduling logic into every workload manifest and drifts.

Second, sharing. NVIDIA device plugins can expose MIG profiles as named resources and advertise configured time-slicing replicas. Repartitioning changes available resources, so a request for an absent profile remains Pending until matching capacity returns or its request changes.

Third, topology. Multi-GPU training cares which GPUs are on the same NVLink domain or PCIe switch. The raw count does not express that relationship. Device plugins can supply topology hints to the Topology Manager, and driver allocation policies can help; evaluate which constraints the installed stack actually honors.

DRA replaces the count with a claim. Devices are published with structured attributes such as memory, product name and topology, and a workload references a ResourceClaim describing what it needs. The scheduler matches the claim against real device attributes rather than decrementing a number, so heterogeneity, partitioning and topology all become expressible in the same mechanism. Core DRA became stable in Kubernetes 1.34. Driver attributes and optional allocation features have their own support and feature-gate requirements; check the DRA documentation for the deployed version.

The operational consequences are worth naming. GPUs are indivisible without MIG and their memory cannot be overcommitted, so bin-packing intuition from CPU workloads misleads. Idle accelerators are expensive enough that queueing is preferable to over-provisioning, which is why Kueue and gang scheduling matter: a distributed training job needs all its workers or none, and partially scheduling it wastes the GPUs it did get while it waits.

What interviewers probe next

"When is time slicing the wrong answer?" Whenever the workloads have latency requirements or need memory isolation, because time slicing shares the device without partitioning memory and one tenant can OOM another. Use it only when the measured interference and weaker isolation meet the workload requirements; strict production latency or untrusted tenancy usually calls for a stronger boundary.

"A 70B model landed on an L40S instead of an H100. Diagnose it." The request was for a generic GPU resource, so the scheduler had no way to distinguish. Short term, node affinity on GPU Feature Discovery labels. Long term, a resource claim that names the memory requirement.

"How do you keep expensive GPUs busy?" Queue rather than over-provision, scale batch workloads to zero, keep a warm minimum only for latency-sensitive inference, and separate the two pools so batch cannot starve serving.

Common mistakes

Describing DRA as "the new way to request GPUs" with no account of what the old way could not express. The limitation is the answer.

Applying CPU intuition, particularly overcommitment. GPU memory is a hard wall.

Ignoring the interconnect for multi-node training. A job that is waiting on NCCL is not compute-bound, and treating it as compute-bound leads to buying the wrong hardware.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.