TL;DR: Liveness checks answer whether the device responds, which is the wrong question. Run DCGM diagnostics plus a short collective benchmark on every node, compare each node against the fleet distribution rather than a fixed threshold, and automate cordon-and-drain on the signals that predict failure: Xid errors, rising ECC retirement, PCIe replay counts, sustained clock throttling and NVLink bandwidth below its peers.
How to approach it
Start from the observation that a slow node is worse than a dead node, because a dead node reschedules and a slow node caps every gang it joins at its own speed. Then structure the answer as detection, attribution and remediation, and say plainly that the detection signal has to be relative to the fleet, because absolute thresholds miss exactly this class of fault.
A strong answer
What actually degrades. Xid errors are the driver's fault codes and they are not all equal: some are application bugs (a bad memory access in a kernel), some are hardware (double-bit ECC, GPU falling off the bus). ECC memory retires pages over time, and a device retiring pages at a rising rate is on its way out. Thermal or power throttling shows up as clocks sitting below their target under load, often from a failing fan or a hot rack position rather than the GPU itself. On multi-GPU nodes the interconnect degrades independently: one NVLink lane down turns an all-reduce into a bottleneck while every device reports healthy. PCIe replay counts rising means the link is retrying, which shows up as slow host-to-device copies and a data loader that appears to be the problem.
Detection has to be comparative. DCGM exports the counters; the discipline is in how you read them. Run dcgmi diag -r 2 on drain and after every incident, and run a short NCCL all-reduce benchmark as part of node admission, recording achieved bandwidth. Then compare each node against the fleet: a node two standard deviations below the median on bus bandwidth is suspect even though every absolute number looks fine. This is why the answer is a distribution and not a threshold. The fleet tells you what healthy looks like this week, with this driver, at this ambient temperature.
Remediation is a policy question. Automate the cordon: a node crossing the rule set gets tainted immediately, because a human in the loop at 3am means the node keeps taking jobs. Draining is more delicate, since evicting one pod from a running gang kills the whole job. The workable pattern is to cordon on detection, let the current gang finish or hit its next checkpoint, then drain. Keep a quarantine pool where a suspect node runs one real job under observation before returning to service, because a node that passes diagnostics and fails under load is common enough to plan for. On cloud, the economics are simpler: terminate the instance and let the pool replace it. On owned hardware you are opening a support case, so the counters you captured are the evidence.
| Signal | What it usually means | Action |
|---|---|---|
| Xid 48 | Uncorrectable ECC event | Quarantine and follow architecture-specific recovery guidance |
| Xid 63 / 64 | Retirement or row-remapping recorded / failed | Interpret with GPU generation, associated errors and diagnostic state |
| Xid 79 | GPU became inaccessible over PCIe | Capture diagnostics; investigate power/link/driver causes before replacement |
| Rising retired pages | Memory wearing out | Schedule replacement |
| Clocks below target under load | Thermal or power limit | Check cooling, rack position |
| NCCL bandwidth below fleet p10 | Link or topology fault | Quarantine, benchmark again |
NVIDIA's Xid catalog distinguishes these codes and their recovery paths. Do not turn every retirement record or transient bus loss into an automatic RMA.
What interviewers probe next
"How do you avoid cordoning half the fleet after a driver upgrade?" Compare against the current fleet, not a historical baseline, and require a signal to persist across two checks. A regression that hits every node is a release problem, not a hardware problem, and your rule should be able to tell those apart.
"Who pays for the wasted GPU-hours?" Attribute them. Track GPU-hours lost to node faults separately from hours lost to job failures, because the two drive different fixes and the first one justifies replacement hardware.
"What about a node that only fails under a specific model?" That is usually not the node. Reproduce on a second node before opening a hardware case, or you will spend two weeks on an RMA for a memory-fragmentation bug.
Common mistakes
Treating GPU health as a liveness probe. The device answers, the job is slow, and nothing pages.
Ignoring the interconnect. A degraded link can limit a healthy GPU; include both link and compute diagnostics.
Draining a node out from under a running gang, which converts a 30 percent slowdown into a lost six-hour job.