Spot (preemptible) capacity is the best discount in cloud computing and the most common source of "why is my batch job stuck at 99% for six hours" tickets.
The deal is simple: up to 70–90% off, in exchange for the provider taking the capacity back with about two minutes' notice.
The question that decides everything
Not "can we afford it?" but "what happens when this node vanishes mid-work?"
| Workload | Spot-friendly? | Why |
|---|---|---|
| Stateless web tier behind a LB | Yes | Eviction drains connections; replicas absorb it |
| Batch with checkpointing | Yes | Resumes from last checkpoint |
| Batch without checkpointing | No | Loses the whole run on eviction |
| Single-replica database | Absolutely not | Data loss and split-brain risk |
| CI runners | Yes | Jobs re-queue; idempotent by design |
| Long training run, no checkpoint | No | Days of compute, gone |
The pattern that makes spot safe
Run redundant, interruptible capacity. The safety property you want is: no single eviction can lose work or drop a user request.
- Keep a small on-demand floor for capacity that must exist.
- Put the elastic burst on spot, spread across instance types and zones so one pool's reclamation does not take everything.
- Make work checkpoint or re-queue. AWS eviction notices give you roughly two minutes — enough to drain, not enough to finish.
- Handle the interruption signal explicitly: stop accepting work, flush, checkpoint, exit cleanly.
On Kubernetes this is exactly what node pools with mixed instance types plus a cluster autoscaler give you, and it is why "spot for everything" fails while "spot for the burst" saves real money.
Do the arithmetic honestly
If a job takes four hours and the pool reclaims every ninety minutes on average, an un-checkpointed job may never finish. The discount is only real if the work survives the reclamation.
Spot pricing is not a discount on compute. It is a discount in exchange for designing for interruption.
Teams that treat it as a pure cost cut end up paying for it in engineering time. Teams that treat it as a design constraint get the 70% and keep their weekends.
Conversation
1 comment