TL;DR: Split the demand into a committed floor, a flexible middle and an interruptible tail, then buy each differently: reservations or capacity blocks for the floor, on-demand for the middle, spot or preemptible for anything that checkpoints. Plan across regions and generations from the start, because the constraint is usually availability rather than budget, and a workload pinned to one instance type in one region has no options when that type runs out.
How to approach it
Establish the demand shape before the purchase mix: what fraction of the 64 runs continuously, what is bursty, and what can be interrupted. Then treat availability as the primary constraint and cost as the secondary one, which inverts the usual cloud instinct and is the point of the question.
A strong answer
Segment the demand. Production inference with an SLO is the floor: it runs continuously, cannot be interrupted, and its capacity must exist before traffic arrives. Development, experimentation and evaluation are the flexible middle: they matter, they can wait an hour, and they should not hold reserved capacity overnight. Batch training, hyperparameter sweeps and offline inference are the interruptible tail, and they can absorb a great deal of price volatility if the jobs checkpoint.
| Segment | Purchase | Why |
|---|---|---|
| Inference with SLO | Capacity reservation, plus an eligible discount commitment | Reserve the required shape; price the discount separately |
| Bursty dev and eval | On-demand within a quota | Flexibility, no idle commitment |
| Checkpointing batch | Spot, preemptible, capacity blocks | Large discount, interruption is survivable |
| Known training runs | Capacity block for a fixed window | Guaranteed contiguous nodes on a date |
Separate capacity from discounts. An AWS On-Demand Capacity Reservation holds a matching instance shape in one Availability Zone, once AWS has accepted it; an immediate request can still fail when the zone is full. Savings Plans discount eligible usage but do not reserve GPUs. Regional Reserved Instances are also billing discounts; zonal Reserved Instances include capacity reservation benefits. Check the exact instrument and matching rules before promising availability. AWS distinguishes these purchasing options.
A capacity reservation is a reliability purchase, and it need not reduce the hourly rate. Price any eligible discount separately, including unused reserved hours. For a customer-facing SLO, reserve the required capacity before promising the launch date and retain a tested failure path for the zone itself.
Capacity blocks fit training specifically: contiguous GPUs, colocated for the interconnect, for a fixed window booked in advance. That maps well onto a planned training run and badly onto exploratory work, so it complements rather than replaces the other modes.
Design for substitution. A workload that only runs on one accelerator model in one region has one supplier and no leverage. Being able to run on the previous generation at lower throughput, or in a second region at higher latency, converts a hard outage into a degraded mode. This is engineering work (validating the model on a second configuration, keeping images built for both) and it is the difference between waiting three weeks for capacity and running slower for three weeks.
Do not over-commit the discount. Size a discount commitment near the usage floor you expect to sustain. Size capacity reservations separately against launch and failover requirements. A one-year billing commitment on usage you stop needing in month four still costs money, and accelerator generations turn over quickly. Ladder the commitments so they do not all expire together, which preserves the ability to move as prices and hardware change.
What interviewers probe next
"Spot for inference?" Only behind a warm on-demand base, with fast drain and enough replicas that losing several at once is absorbed. The failure mode is a correlated reclaim taking a third of the fleet in a minute, which is exactly when traffic does not stop.
"How do you forecast?" From current utilisation, planned launches, and a model-size trend, with a stated confidence band. Present a range with the reservation sized to the bottom of it, since under-committing is recoverable and over-committing is not.
"What if finance wants a three-year commitment for the deeper discount?" Push back with the hardware refresh cycle. Three years on a specific accelerator generation is a bet that the price per token will not fall, and it usually has.
Common mistakes
Treating GPU capacity as elastic. On-demand is what remains after everyone's reservations, and in a tight region that is frequently zero.
Reserving to peak, which pays for idle hardware for a year to serve a few hours a week.
One region, one instance type, no validated fallback, which leaves a team with no options exactly when capacity gets tight.