TL;DR: Design for abrupt loss as well as noticed interruption. Use a notice to stop new work and attempt a final checkpoint, but recover from the last completed durable checkpoint if shutdown cannot finish. Diverse capacity pools reduce concentration risk; idempotent redrive and fallback capacity determine recovery.
How to approach it
Split the problem into frequency and blast radius. Frequency is a portfolio question (diversification, pool breadth). Blast radius is a workload question (drain, checkpoint, redrive). Candidates who address only one half sound like they have read about spot rather than run it.
A strong answer
Use provider notices, but make recovery work without a completed shutdown hook. EC2 interruption notices are best effort, normally two minutes before stop/termination; hibernation starts immediately. Azure documents a minimum 30-second eviction notice through Scheduled Events. GCP documents a best-effort shutdown period of up to 30 seconds for Spot VMs. Those are different contracts, and handler delivery and checkpoint IO consume the available time. EC2 notices, Azure eviction, GCP Spot VMs.
In Kubernetes, configure the provisioner or interruption handler to cordon and drain, and ensure applications handle SIGTERM. A pod grace period or disruption budget cannot extend the provider's reclamation deadline. Test the installed handler path and abrupt node loss separately.
Everything else in this design depends on checkpointing. Publish complete, verifiable checkpoints to durable storage at a cadence chosen from acceptable recomputation and checkpoint cost. The periodic interval need not be shorter than the notice. An optional final checkpoint must finish within the time remaining; if it cannot, recover from an earlier completed checkpoint. Give each work unit an identity so retries can recognize completed effects. Queue-driven work recovers through the queue itself: SQS visibility timeout hands a dead worker's message back, Kafka offsets stay uncommitted until processing completes. Underneath both sits idempotency, because at-least-once delivery means the same item can complete twice, and completing twice must be harmless.
| Workload shape | Recovery mechanism | Loss when done right |
|---|---|---|
| Stateless request workers | Drain when possible; bounded safe retries | In-flight requests may fail; spare capacity controls recovery |
| Queue-driven jobs | Durable queue redrive and idempotent effects | Bounded by retention, retries and sink durability |
| Long sharded compute (Spark, video) | Completed checkpoints and worker replacement | Work since the last usable checkpoint, plus recovery time |
| Stateful single-copy | Do not place it on spot | Not applicable |
Frequency is the other half, and it is bought rather than hoped for. Use several compatible instance types across availability zones and inspect pool availability; interruptions can still be correlated. For EC2, evaluate a capacity-aware allocation policy such as price-capacity-optimized rather than selecting only the lowest price. Keep an on-demand baseline and a tested fallback for required throughput, subject to quotas and regional capacity. Diversity alone does not guarantee replacements. EC2 Spot best practices.
Then rehearse it. In staging, test a graceful drain, a notice with little time remaining, and an abrupt worker loss. Measure checkpoint recovery, duplicate suppression and queue drain time. A successful manual drain proves only that shutdown path; it does not establish no-notice recovery.
Reversal condition: keep irreplaceable single-copy state off interruptible nodes. Latency-critical services can use Spot for replaceable excess capacity only when their baseline and fallback satisfy the required availability. Replicated quorum stores need a failure-domain and quorum analysis before inclusion.
What interviewers probe next
"The checkpoint takes longer than two minutes." Do not depend on a final checkpoint. Use periodic or incremental checkpoints and price recomputation from the last completed one. The workload may still fit Spot if recovery meets its deadline and expected savings cover the lost work.
"An interruption storm hits one zone." Pool diversity reduces concentration but may not hold capacity. Measure backlog age and completion deadlines, activate the tested on-demand fallback where available, and throttle admission if replacement capacity remains constrained.
"How do you know the handler works?" Test the provider-notice integration and forced loss, measuring completed checkpoints, redelivery and duplicate effects as well as drain latency.
Common mistakes
Choosing checkpoint cadence without pricing recomputation, or assuming an unfinished final checkpoint replaces the last completed one.
Assuming Kubernetes graceful termination by default: without a real SIGTERM path and adequate grace period, the two minutes expire with pods mid-write.
No idempotency, so redriven jobs double-write results and quietly corrupt downstream aggregates.
Running one instance type because it benchmarked fastest, then wondering why reclamation arrives in waves.