Initializing Techhub.cafe

midAWS Machine Learning

How do you track and compare experiments in SageMaker?

Updated Aug 1, 2026

Short answer

SageMaker's experiment tracking groups runs under an experiment name, logging parameters, metrics, and artefacts per run so results are comparable in Studio and reproducible later — replacing spreadsheets and notebook output as the record of what was tried.

Deep explanation

The failure this solves is mundane and universal: three weeks into a project nobody can say which combination produced the best result, or on which data. Experiment tracking makes the run — not the notebook — the unit of record.

The model is straightforward: an experiment contains runs; each run logs parameters, metrics over time, and artefacts, and is automatically associated with the training job that produced it.

Python
from sagemaker.experiments.run import Run, load_run


with Run(experiment_name="fraud-model-2026q3",
         run_name="xgb-depth6-lr02",
         sagemaker_session=session) as run:


    run.log_parameters({"max_depth": 6, "eta": 0.2, "features": "v3"})
    run.log_artifact(name="feature-list", value="s3://my-bucket/features/v3.json")


    estimator.fit({"train": train_input, "validation": val_input})


    run.log_metric(name="validation:auc", value=0.842)
    run.log_metric(name="validation:pr_auc", value=0.611)
    # Confusion matrix and ROC render natively in Studio
    run.log_confusion_matrix(y_true, y_pred, title="Validation confusion")

Inside a training script, attach to the surrounding run and log per epoch:

Python
# train.py
from sagemaker.experiments.run import load_run


with load_run() as run:
    for epoch in range(args.epochs):
        loss, auc = train_one_epoch(...)
        run.log_metric(name="val_auc", value=auc, step=epoch)

Many teams pair this with MLflow on SageMaker, which AWS now offers as a managed tracking server — useful when the organisation already standardises on MLflow's API and UI, while keeping the artefacts and lineage inside AWS.

The discipline that makes any of this work is logging the things that actually vary and are hard to reconstruct: the git commit, the data window or dataset version, and the feature set version. Hyperparameters are easy to recover from the job description; "which data was this?" is the question that later becomes unanswerable.

Real-world example

A team ran roughly 200 experiments over a quarter across four people. Because every run logged its git commit and feature-set version, reproducing the best model months later took minutes: check out the commit, point at the recorded feature version, re-run. On their previous project — spreadsheets and notebook output — an equivalent request had taken two days and ended in an approximate answer.

Common mistakes

  • - Logging metrics but not the data version, making runs impossible to reproduce.
  • - One run per notebook execution with no naming convention, producing an unsearchable pile.
  • - Logging only final metrics, so you cannot see whether a run was still improving or had diverged.
  • - Tracking experiments while training in an unversioned notebook, so the code behind a run no longer exists.
  • - Treating the tracker as a dashboard rather than a record — the value is answering "what produced this model" six months later.

Follow-up questions

  • How does experiment tracking relate to lineage tracking?
  • When would you use managed MLflow instead of native tracking?
  • What should you log that people usually forget?

More AWS Machine Learning interview questions

View all →