What are SageMaker Pipelines and why use them instead of orchestrating jobs yourself?

Updated Aug 1, 2026

Short answer

Pipelines is SageMaker's native workflow service: you define a DAG of processing, training, evaluation, condition and registration steps as code, and it handles execution, caching, lineage and parameterisation.

Deep explanation

A production ML workflow is a dependency graph, not a script. Pipelines lets you declare that graph and gives you three things a hand-rolled script cannot: step caching, lineage tracking, and parameterised re-execution.

Step caching is the one that changes daily life. If preprocessing inputs and arguments are unchanged, the cached output is reused and the step is skipped — so iterating on a training hyperparameter does not re-run a forty-minute preprocessing job.

The canonical shape is: process → train → evaluate → condition → register.

Python

1from sagemaker.workflow.pipeline import Pipeline
2from sagemaker.workflow.steps import ProcessingStep, TrainingStep, CacheConfig
3from sagemaker.workflow.condition_step import ConditionStep
4from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
5from sagemaker.workflow.functions import JsonGet
6from sagemaker.workflow.parameters import ParameterFloat, ParameterString
7from sagemaker.workflow.step_collections import RegisterModel
8
9min_auc     = ParameterFloat(name="MinAuc", default_value=0.78)
10input_data  = ParameterString(name="InputData", default_value="s3://lake/raw/")
11cache       = CacheConfig(enable_caching=True, expire_after="7d")
12
13step_process = ProcessingStep(name="Preprocess", processor=processor,
14                              inputs=[...], outputs=[...], code="preprocess.py",
15                              cache_config=cache)
16
17step_train = TrainingStep(name="Train", estimator=xgb, inputs={
18    "train": TrainingInput(
19        step_process.properties.ProcessingOutputConfig.Outputs["train"].S3Output.S3Uri,
20        content_type="text/csv"),
21}, cache_config=cache)
22
23step_eval = ProcessingStep(name="Evaluate", processor=eval_processor,
24                           code="evaluate.py", property_files=[evaluation_report])
25
26# Only register if the model clears the bar
27step_register = RegisterModel(
28    name="Register",
29    estimator=xgb,
30    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,
31    content_types=["text/csv"], response_types=["text/csv"],
32    inference_instances=["ml.m5.large"], transform_instances=["ml.m5.xlarge"],
33    model_package_group_name="fraud-models",
34    approval_status="PendingManualApproval",
35)
36
37step_cond = ConditionStep(
38    name="AucGate",
39    conditions=[ConditionGreaterThanOrEqualTo(
40        left=JsonGet(step_name=step_eval.name, property_file=evaluation_report,
41                     json_path="metrics.auc"),
42        right=min_auc)],
43    if_steps=[step_register],
44    else_steps=[],
45)
46
47pipeline = Pipeline(
48    name="fraud-training",
49    parameters=[min_auc, input_data],
50    steps=[step_process, step_train, step_eval, step_cond],
51)
52pipeline.upsert(role_arn=role)
53pipeline.start(parameters={"MinAuc": 0.80})

The critical mechanism is property references: step_train.properties.ModelArtifacts.S3ModelArtifacts is not a value at definition time — it is a placeholder resolved at execution. That is how the DAG edges are inferred; you never declare dependencies manually.

The condition step is what makes this a quality gate rather than a conveyor belt. A model that fails the AUC threshold is simply never registered, so it cannot be deployed.

Real-world example

A lending platform retrains weekly on an EventBridge schedule. The pipeline recomputes features, trains, evaluates against a held-out month, and registers only if AUC ≥ 0.80 and the fairness gap from Clarify is within tolerance. Registration is PendingManualApproval, so a risk officer reviews the model card before an EventBridge rule on the approval event triggers deployment. Caching means a re-run after a hyperparameter tweak takes eight minutes rather than fifty.

Common mistakes

- Treating pipeline definition code as runtime code — you cannot `print()` a property reference or branch on it in Python
use a ConditionStep.
- Skipping the condition step, so a degraded model sails through to registration.
- Leaving caching off and paying repeatedly for identical preprocessing.
- Hard-coding S3 paths instead of Parameters, making the same pipeline unusable across dev and prod.
- Forgetting that a `property_file` must be declared on the step that produces it before `JsonGet` can read it.

Follow-up questions

When would you use Step Functions instead of SageMaker Pipelines?
How does step caching decide a step can be skipped?
How do you trigger a pipeline automatically?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More AWS Machine Learning interview questions