How does SageMaker Automatic Model Tuning work and which strategy should you choose?

Updated Aug 1, 2026

Short answer

AMT runs many training jobs across a hyperparameter search space and optimises a metric you emit. Bayesian search learns from previous trials and is the sensible default; random parallelises perfectly; Hyperband stops weak trials early and is best for deep learning.

Deep explanation

AMT is a managed search loop. You declare ranges, an objective metric and a budget, and it schedules training jobs.

Strategies, and when each wins:

Bayesian — builds a surrogate model of the objective and picks promising points. Most sample-efficient; the default for expensive jobs. Because it learns sequentially, high parallelism reduces its advantage — keep max_parallel_jobs modest (roughly 10% of total) so each batch informs the next.
Random — samples independently. Embarrassingly parallel, surprisingly strong, and a good baseline. Use when you can run everything at once and wall-clock matters more than cost.
Hyperband — allocates a small budget to many configurations, kills the worst, promotes survivors. Dramatically cheaper for deep learning where a bad configuration is obvious after a few epochs. Requires your script to emit the objective metric repeatedly per epoch.
Grid — exhaustive over categorical values. Only sensible for small discrete spaces.

Python

1from sagemaker.tuner import HyperparameterTuner, ContinuousParameter, IntegerParameter
2
3tuner = HyperparameterTuner(
4    estimator=xgb,
5    objective_metric_name="validation:auc",
6    objective_type="Maximize",
7    hyperparameter_ranges={
8        # log scaling: learning rates matter multiplicatively, not additively
9        "eta":              ContinuousParameter(0.01, 0.3, scaling_type="Logarithmic"),
10        "max_depth":        IntegerParameter(3, 10),
11        "min_child_weight": ContinuousParameter(1, 10),
12        "subsample":        ContinuousParameter(0.5, 1.0),
13        "lambda":           ContinuousParameter(0.1, 10, scaling_type="Logarithmic"),
14    },
15    strategy="Bayesian",
16    max_jobs=40,
17    max_parallel_jobs=4,          # deliberately low so Bayesian can learn
18    early_stopping_type="Auto",
19)
20tuner.fit({"train": train_input, "validation": val_input})
21print(tuner.best_training_job())

For a custom script, AMT reads the metric from stdout via a regex:

Python

1estimator = PyTorch(
2    entry_point="train.py", role=role, instance_type="ml.g5.xlarge",
3    metric_definitions=[
4        {"Name": "validation:auc",  "Regex": "val_auc: ([0-9\\.]+)"},
5        {"Name": "validation:loss", "Regex": "val_loss: ([0-9\\.]+)"},
6    ],
7)
8# and in train.py, printed once per epoch so Hyperband can act on it:
9#   print(f"val_auc: {auc:.5f}")

Logarithmic scaling is the detail interviewers probe. Learning rate 0.001 → 0.01 is a tenfold change; 0.1 → 0.109 is trivial. Linear sampling wastes almost every trial on the uninteresting end of the range.

Real-world example

A vision team tuning a fine-tuned ResNet moved from Bayesian to Hyperband and cut tuning cost by roughly 70%. Most bad learning rates are visibly hopeless after three epochs; Hyperband killed them there instead of running all thirty. The prerequisite was emitting val_loss every epoch — with a metric printed only at the end, Hyperband has nothing to prune on and silently degrades to random search.

Common mistakes

- Using linear scaling for learning rate or regularisation, wasting the search on a narrow effective range.
- Setting `max_parallel_jobs` equal to `max_jobs` with Bayesian, which removes the sequential learning that justifies it — use random if you want full parallelism.
- Tuning on the test set. AMT optimises what you give it
feed it validation and keep test untouched.
- Emitting the objective metric only at the end while using Hyperband.
- Searching a huge space with a tiny budget
better to fix unimportant hyperparameters and search four or five that matter.

Follow-up questions

What is warm start and when is it useful?
How does early stopping in AMT differ from Hyperband?
How do you keep tuning costs under control?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More AWS Machine Learning interview questions