TL;DR: Profile the step to split compute time from communication time before touching anything. If communication dominates, investigate placement across a slow boundary, a misconfigured NCCL path falling back to TCP, congestion or lossy behaviour on a RoCE fabric, and a single degraded link dragging the collective. Check placement early, while accounting for rescheduling and checkpoint costs.
How to approach it
Refuse to guess between compute and communication. Say you would instrument the step breakdown first, then work outward: within the node, across nodes, then the fabric itself. The interviewer is checking whether you know that a distributed training job is a networking system with GPUs attached.
A strong answer
Split the step. PyTorch profiler or Nsight gives you time in compute versus time in collectives. Long collective time can reflect fabric limits, late-arriving ranks, compute imbalance or poor overlap; it does not alone prove a network fault. Baseline the fabric independently with a NCCL all-reduce benchmark at the same message size and world size, and compare achieved bus bandwidth against what the hardware can do. That single number separates "our job is badly configured" from "this fabric cannot do better".
Check placement against the intended topology. GPUs inside one node talk over NVLink at hundreds of gigabytes per second. GPUs in different nodes talk over the fabric, which even at 400 Gbps InfiniBand is an order of magnitude slower per link. Sixteen GPUs spread across sixteen nodes, one GPU each, is the worst possible layout. A scheduler without the needed constraints may choose an unsuitable layout; inspect the actual placement. Ask for whole nodes, and ask for nodes within one network domain: a placement group on cloud, the same leaf switch or rail on owned hardware. Hierarchical collectives then do the heavy reduction inside the node over NVLink and exchange only the reduced result across the fabric.
The silent fallback. NCCL will quietly use TCP over the regular network if it cannot initialise RDMA, and the job runs, just slowly. Causes are mundane: the container missing the InfiniBand device nodes, the wrong interface selected, a missing plugin in a cloud environment. Set NCCL_DEBUG=INFO and read which transport it chose at startup. Two lines of log. Making that a startup assertion, failing the job if the transport is not what you expect, is worth more than any dashboard.
InfiniBand versus RoCE is the design-level version of the question. InfiniBand is lossless by design with credit-based flow control and hardware-managed congestion; it requires a compatible fabric and operational expertise; compare its total cost against the proposed Ethernet design. RoCE runs RDMA over Ethernet and can match it, but only on a fabric tuned for it: PFC and ECN configured consistently end to end, correct buffer sizing, and no misconfigured switch anywhere in the path. RoCE that mostly works produces exactly this symptom, throughput below spec with no errors anyone notices, because pause frames and retransmissions do not appear in application metrics.
Stragglers. A collective finishes when its slowest member finishes. One node with a degraded link, a thermally throttled GPU, or a noisy neighbour on shared storage sets the pace for the whole gang. Log per-rank step times and alert on the spread, not the mean.
What interviewers probe next
"How would you prove it is the network to someone who wants to blame the model code?" Run the NCCL benchmark alone on the same allocation. It is the same fabric, no model, and it either reaches spec or it does not.
"What changes at 1,000 GPUs?" Topology awareness stops being an optimisation. Rail-optimised designs and hierarchical algorithms are the difference between scaling and not, and failure becomes continuous rather than occasional, so checkpointing and elastic restart become the dominant design concern.
"Would you use gradient compression?" Only after placement and fabric are correct. It trades convergence quality for bandwidth, and paying that price to work around a scheduler bug is a bad trade.
Common mistakes
Assuming time inside all-reduce proves a fabric fault without checking rank arrival times and communication overlap.
Accepting a working job as a correct job. TCP fallback can run successfully with a large workload-dependent throughput penalty.
Treating GPU count as capacity. Sixteen scattered GPUs are not the same machine as sixteen colocated ones, and the difference shows up on the invoice.