Initializing Techhub.cafe

juniorAWS Machine Learning

What is a SageMaker Processing job and when should you use one?

Updated Aug 1, 2026

Short answer

A Processing job runs a containerised script on managed, ephemeral compute for data preprocessing, feature engineering, validation or model evaluation — the steps around training that still need real compute.

Deep explanation

Processing jobs solve a specific problem: preprocessing that is too heavy for a notebook but is not training. Running it in a notebook means your work is tied to a machine you must keep alive, and nothing is reproducible. A Processing job packages the script, runs it on a right-sized ephemeral cluster, and records inputs and outputs.

The container contract is simple and worth memorising:

  • Inputs are copied from S3 to /opt/ml/processing/input/<name>.
  • Your script reads there and writes to /opt/ml/processing/output/<name>.
  • Outputs are copied back to S3 when the script exits 0.
Python
from sagemaker.sklearn.processing import SKLearnProcessor
from sagemaker.processing import ProcessingInput, ProcessingOutput


processor = SKLearnProcessor(
    framework_version="1.2-1",
    role=role,
    instance_type="ml.m5.4xlarge",
    instance_count=2,          # data is split across instances
)


processor.run(
    code="preprocess.py",
    inputs=[ProcessingInput(
        source="s3://my-lake/raw/",
        destination="/opt/ml/processing/input",
        s3_data_distribution_type="ShardedByS3Key",
    )],
    outputs=[
        ProcessingOutput(source="/opt/ml/processing/output/train",
                         destination="s3://my-lake/features/train"),
        ProcessingOutput(source="/opt/ml/processing/output/test",
                         destination="s3://my-lake/features/test"),
    ],
    arguments=["--train-ratio", "0.8"],
)

And the script itself:

Python
# preprocess.py
import argparse, pandas as pd
from pathlib import Path


p = argparse.ArgumentParser()
p.add_argument("--train-ratio", type=float, default=0.8)
args = p.parse_args()


df = pd.concat(pd.read_parquet(f) for f in Path("/opt/ml/processing/input").glob("*.parquet"))
df = df.dropna(subset=["target"])


cut = int(len(df) * args.train_ratio)
Path("/opt/ml/processing/output/train").mkdir(parents=True, exist_ok=True)
Path("/opt/ml/processing/output/test").mkdir(parents=True, exist_ok=True)
df.iloc[:cut].to_parquet("/opt/ml/processing/output/train/train.parquet")
df.iloc[cut:].to_parquet("/opt/ml/processing/output/test/test.parquet")

The same mechanism runs evaluation after training — load the model, score the test set, write evaluation.json — which a Pipelines condition step then reads to decide whether the model is good enough to register.

Real-world example

A bank's fraud pipeline runs a Processing job over 200 GB of nightly transactions on ten ml.m5.4xlarge instances, joining reference data and writing partitioned Parquet features. Because it is sharded, it finishes in twelve minutes instead of two hours on one box, and costs roughly the same — ten instances for a tenth of the time. The identical script runs in dev against a 1% sample by pointing at a different S3 prefix.

Common mistakes

  • - Hard-coding S3 paths in the script instead of using the input/output mounts, which breaks reuse across environments.
  • - Forgetting `ShardedByS3Key` with `instance_count > 1`, so every instance processes the whole dataset and you silently produce duplicated output.
  • - Doing preprocessing in a notebook and calling it reproducible — nothing is versioned or re-runnable.
  • - Under-provisioning memory and hitting OOM
  • Processing instances have no swap and the job simply dies.
  • - Writing outputs outside `/opt/ml/processing/output`, so nothing is uploaded and the job "succeeds" with no result.

Follow-up questions

  • How does a Processing job differ from an AWS Glue job for the same work?
  • How do you make preprocessing consistent between training and inference?
  • Can a Processing job read from somewhere other than S3?

More AWS Machine Learning interview questions

View all →