Initializing Techhub.cafe

midAWS Machine Learning

What is script mode and when would you build your own container instead?

Updated Aug 1, 2026

Short answer

Script mode supplies your training script to an AWS-managed framework container, so you write model code and AWS handles the image; a custom container is for unusual dependencies, non-supported frameworks or system-level requirements the managed image cannot satisfy.

Deep explanation

There is a ladder of increasing control and increasing maintenance burden:

  1. Built-in algorithm — no code.
  2. Script mode — your training script, AWS's image.
  3. Extend a managed image — FROM an AWS DLC and add packages.
  4. Bring your own container — fully custom image meeting the SageMaker contract.

Most teams should live on rung 2 and climb only when blocked. The managed Deep Learning Containers are non-trivially valuable: patched, optimised for AWS hardware, with correct CUDA/NCCL builds and EFA support already wired.

Script mode:

Python
from sagemaker.pytorch import PyTorch


estimator = PyTorch(
    entry_point="train.py",
    source_dir="src",                     # whole dir uploaded; requirements.txt honoured
    role=role,
    framework_version="2.3", py_version="py311",
    instance_type="ml.g5.2xlarge", instance_count=1,
    hyperparameters={"epochs": 20, "lr": 3e-4},
)

The script reads SageMaker's environment contract:

Python
# src/train.py
import argparse, os, torch


p = argparse.ArgumentParser()
p.add_argument("--epochs", type=int, default=10)
p.add_argument("--lr", type=float, default=1e-3)
# SageMaker injects these env vars — never hard-code paths
p.add_argument("--model-dir",   default=os.environ["SM_MODEL_DIR"])
p.add_argument("--train-dir",   default=os.environ["SM_CHANNEL_TRAIN"])
args = p.parse_args()


# ... training ...
torch.save(model.state_dict(), os.path.join(args.model_dir, "model.pth"))

Extending an image — the pragmatic middle ground:

DOCKERFILE
FROM 763104351884.dkr.ecr.eu-west-1.amazonaws.com/pytorch-training:2.3-gpu-py311
RUN apt-get update && apt-get install -y libgeos-dev   # system dep pip can't provide
COPY requirements.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements.txt

Full BYOC must honour the contract: respond to train for training, run a server answering /ping and /invocations for inference, read from /opt/ml/input, write to /opt/ml/model.

Reach for BYOC when you need a framework AWS does not ship, a specific CUDA or compiler toolchain, licensed native libraries, or a bespoke serving stack. Accept that you now own base-image patching and CVE response.

Real-world example

A genomics team needed bioinformatics binaries with no pip equivalent. Rather than full BYOC, they extended the managed PyTorch DLC with an apt install and pinned Python packages — keeping AWS's CUDA and NCCL configuration while adding what they needed. Their Dockerfile is about fifteen lines, rebuilt weekly by CI against the latest DLC tag so security patches flow in automatically.

Common mistakes

  • - Jumping to BYOC when script mode with a `requirements.txt` would have worked, and inheriting image maintenance forever.
  • - Hard-coding `/opt/ml/...` paths instead of reading `SM_MODEL_DIR` and `SM_CHANNEL_*`, which breaks under different channel configurations.
  • - Building a custom GPU image without matching CUDA, driver and framework versions — a slow, painful class of bug the DLCs exist to prevent.
  • - Pinning to a `:latest` image tag, making builds unreproducible and pipeline caching useless.
  • - Forgetting `/ping` must return 200 quickly, causing endpoint creation to fail health checks with an opaque error.

Follow-up questions

  • How does SageMaker pass hyperparameters into your script?
  • What is SageMaker local mode and why is it useful?
  • How do you manage dependencies reproducibly in script mode?

More AWS Machine Learning interview questions

View all →