How do you use managed Spot training and what must your script handle?

Updated Aug 1, 2026

Short answer

Managed Spot uses spare EC2 capacity for up to about 90% savings; you enable it with a max wait time and an S3 checkpoint path, and your script must save and resume from checkpoints so an interruption resumes rather than restarts.

Deep explanation

Spot capacity can be reclaimed with two minutes' notice. SageMaker handles the mechanics — it detects the interruption, waits for capacity, restarts the job — but it cannot recover your in-memory training state. That is entirely on your script.

The contract: SageMaker syncs checkpoint_local_path to checkpoint_s3_uri during the run, and restores it into the container before a resumed job starts. So the pattern is always look for a checkpoint first.

Python

1estimator = PyTorch(
2    entry_point="train.py", role=role,
3    instance_type="ml.g5.12xlarge", instance_count=4,
4    use_spot_instances=True,
5    max_run=14400,        # max training seconds
6    max_wait=28800,       # must exceed max_run: training + waiting for capacity
7    checkpoint_s3_uri="s3://my-bucket/checkpoints/exp-42/",
8    checkpoint_local_path="/opt/ml/checkpoints",
9)

Python

1# train.py — resumable by construction
2import os, glob, torch
3
4CKPT_DIR = "/opt/ml/checkpoints"
5os.makedirs(CKPT_DIR, exist_ok=True)
6
7start_epoch = 0
8existing = sorted(glob.glob(f"{CKPT_DIR}/ckpt_*.pt"))
9if existing:                                   # a resumed spot job lands here
10    state = torch.load(existing[-1], map_location="cpu")
11    model.load_state_dict(state["model"])
12    optimizer.load_state_dict(state["optimizer"])
13    scheduler.load_state_dict(state["scheduler"])
14    start_epoch = state["epoch"] + 1
15    print(f"Resumed from epoch {start_epoch}")
16
17for epoch in range(start_epoch, args.epochs):
18    train_one_epoch(...)
19    # Atomic write: never leave a half-written checkpoint to be restored
20    tmp = f"{CKPT_DIR}/.tmp_{epoch}.pt"
21    torch.save({"model": model.state_dict(),
22                "optimizer": optimizer.state_dict(),
23                "scheduler": scheduler.state_dict(),
24                "epoch": epoch}, tmp)
25    os.replace(tmp, f"{CKPT_DIR}/ckpt_{epoch:04d}.pt")

Three details that separate a working answer from a theoretical one:

Save the optimizer and scheduler, not just weights. Restoring weights alone resets momentum and learning-rate schedule, and the loss curve visibly suffers.
Write atomically. An interruption mid-torch.save leaves a corrupt file that breaks the resume.
`max_wait` must exceed `max_run` — it bounds training plus waiting for capacity, and the API rejects the job otherwise.

Judgement about where Spot fits: excellent for tuning jobs, experiments and long pre-training with checkpoints; poor for a job that must finish by a deadline, since capacity waits are unpredictable.

Real-world example

A team pre-training a domain language model runs 60-hour jobs on ml.p4d.24xlarge Spot with checkpoints every 20 minutes. They see two or three interruptions per run; each costs about 20 minutes of recomputation. Cost fell roughly 70% versus on-demand. Their final fine-tune before a release deadline runs on-demand — the savings are not worth missing a launch.

Common mistakes

- Enabling Spot without checkpointing, so an interruption restarts from zero and repeated interruptions can make a job never finish.
- Saving only model weights, losing optimizer and scheduler state and degrading convergence after resume.
- Setting `max_wait` equal to or below `max_run`, which fails validation.
- Non-atomic checkpoint writes producing corrupt files.
- Using Spot for deadline-bound work, where capacity waits are unbounded in practice.

Follow-up questions

How does the checkpoint sync actually work?
What savings should you expect, and how do you see them?
How do you reduce interruption impact besides frequent checkpoints?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More AWS Machine Learning interview questions