TL;DR: Assign nominal quota per team and allow borrowing through a cohort. Configure reclaim and measure how long a returning lender actually waits. Nominal quota is an accounting entitlement, not a promise of immediate GPU placement. Checkpoint-aware preemption needs workload integration beyond Kueue's built-in priority and fair-sharing policies.
How to approach it
State the failure mode of each naive design in one line before proposing the middle: hard quotas waste, no quotas starve. Then spend the answer on reclaim and preemption, because that is where the design gets decided and where the political problems live.
A strong answer
Nominal quota plus borrowing. Each team receives nominal quota for a resource flavor. A cohort lets queues borrow unused quota subject to borrowing and lending limits. Returning lenders may reclaim resources if preemption policy permits it. Admission can still wait for termination, eligible victims, healthy nodes or topology fit. Publish a measured reclaim target separately from the quota amount.
Reclaim is the hard half. When a lender submits work, the platform may need to preempt workloads in queues above nominal quota. Three parameters decide whether the system is usable:
- How fast? Immediate preemption maximises fairness and destroys work. Set a termination budget that the training controller and application can use to checkpoint, then test it. A fifteen-minute reclaim target is a platform SLO you must validate; Kueue cannot guarantee it from nominal quota alone.
- Whose work? Use GPU-hours since the last checkpoint to estimate restart cost. Kueue's preemption policies use quota, priority and fair sharing; choosing victims by checkpoint cost requires an additional controller or scheduler integration. Preempting the smallest job is the intuitive rule and it is wrong: a small job that has run for eight hours without checkpointing costs more to kill than a large one that saved five minutes ago.
- Can borrowed work opt out? Use a separate queue with borrowing disabled and an explicit preemption policy for protected work. Staying within nominal quota does not prevent every within-queue preemption or infrastructure interruption. Reclaim eligibility is based on queue usage, not a permanent borrowed label on one job.
Checkpointing is the precondition for the whole model. Without checkpoints, preemption loses all progress. Lending is still possible, but reclaim must either wait for completion or accept that loss. So the platform's job is to make checkpointing the default: a library, a template, a signal handler in the base image, and an admission rule that a job entering a borrowable queue declares a checkpoint interval. Framing it that way turns a nagging conversation into a capability the platform provides.
Set nominal quotas from measured steady-state demand, not from what each team requests. Teams ask for peak; the sum of peaks exceeds the cluster; the exercise stalls. Allocate from observed usage, leave a shared pool unallocated for bursts, and revisit quarterly.
Publish the numbers that end arguments. Per team: allocated hours, of which idle, queue time distribution, and preemption count. A team with high queue time and low idle time has a genuine capacity case. A team with low queue time and high idle time is holding capacity it does not need, and seeing that next to a peer's queue time changes behaviour faster than any policy.
What interviewers probe next
"How do you handle an urgent production retrain?" A high-priority class with a small reserved allocation, used rarely and audited. If everything is urgent the class is worthless, so the audit is the control that keeps it meaningful.
"Does this work for interactive notebooks?" Badly, because a notebook holds a GPU while somebody thinks. Give them a separate pool with idle timeouts and a smaller device or a MIG slice, and keep them out of the training cohort entirely.
"What if a team never uses its guarantee?" Reduce it at the quarterly review, with the idle report as the evidence. Quota that is never used is capacity the cluster cannot plan around.
Common mistakes
Hard quotas with no borrowing, which produces a cluster with idle GPUs and queued jobs at the same time.
Preempting the smallest job, which maximises wasted GPU-hours precisely when the cluster is contended.
Enabling borrowing before checkpointing is standard, so the first reclaim destroys a day of someone's work and the feature is turned off that afternoon.