How does SageMaker Model Monitor detect data and model quality drift?

Updated Aug 1, 2026

Short answer

Model Monitor captures live endpoint traffic to S3, compares it against a baseline computed from training data, and emits CloudWatch metrics and violation reports for data quality, model quality, bias drift and feature attribution drift.

Deep explanation

Models degrade because the world moves, not because the code broke. Monitoring must therefore watch the data, not just the service.

Model Monitor offers four monitor types:

Data quality — schema, missing values, and distributional statistics per feature versus baseline.
Model quality — actual accuracy/AUC, which requires ground-truth labels to be delivered later and merged with captured predictions.
Bias drift — Clarify metrics such as DPPL recomputed on live traffic.
Feature attribution drift — SHAP importance shifting, which catches cases where distributions look stable but the model's reliance on features has changed.

Two prerequisites people miss: data capture must be enabled before you need it (you cannot retroactively capture yesterday's traffic), and model quality monitoring needs a ground-truth pipeline — you must write labels to S3 in the expected format once they arrive.

Python

1from sagemaker.model_monitor import DefaultModelMonitor, DataCaptureConfig
2from sagemaker.model_monitor.dataset_format import DatasetFormat
3
4# 1. Capture at deploy time
5model.deploy(..., data_capture_config=DataCaptureConfig(
6    enable_capture=True, sampling_percentage=100,
7    destination_s3_uri="s3://my-bucket/capture/fraud"))
8
9monitor = DefaultModelMonitor(role=role, instance_count=1,
10                              instance_type="ml.m5.xlarge", volume_size_in_gb=20)
11
12# 2. Baseline from the SAME data the model trained on
13monitor.suggest_baseline(
14    baseline_dataset="s3://my-bucket/features/train/train.csv",
15    dataset_format=DatasetFormat.csv(header=True),
16    output_s3_uri="s3://my-bucket/baseline/fraud",
17)
18
19# 3. Schedule comparison
20monitor.create_monitoring_schedule(
21    endpoint_input="fraud-scorer",
22    output_s3_uri="s3://my-bucket/monitor/fraud",
23    statistics=monitor.baseline_statistics(),
24    constraints=monitor.suggested_constraints(),
25    schedule_cron_expression="cron(0 * ? * * *)",   # hourly
26    enable_cloudwatch_metrics=True,
27)

Each run writes constraint_violations.json:

JSON

1{"violations": [
2  {"feature_name": "transaction_amount",
3   "constraint_check_type": "baseline_drift_check",
4   "description": "Kolmogorov-Smirnov distance 0.31 exceeds threshold 0.10"},
5  {"feature_name": "merchant_category",
6   "constraint_check_type": "completeness_check",
7   "description": "Completeness 0.62 below threshold 0.99"}
8]}

That second violation is the realistic one: an upstream change started sending nulls. The model kept returning confident scores from a broken feature — silent failure that only monitoring catches.

Wire violations to action: CloudWatch alarm → SNS for paging, and optionally EventBridge → retraining pipeline.

Real-world example

A lender's income feature silently changed unit from monthly to annual after an upstream migration. Predictions stayed within a plausible range, so nothing alerted for eleven days and thousands of decisions were made on a mis-scaled feature. After adding Model Monitor, an equivalent change now fires a KS-distance violation within the hour and pages the on-call data scientist.

Common mistakes

- Enabling data capture only after an incident, leaving no baseline or history.
- Baselining on a different dataset than the model trained on, producing violations from day one that everyone learns to ignore.
- Monitoring inputs but never labels, so accuracy decay is invisible until the business notices.
- Setting thresholds so tight that alerts become noise, or so loose they never fire — tune against a replay of historical traffic.
- Capturing 100% of high-volume traffic and paying significant S3 cost
sample when volume is large.

Follow-up questions

Why monitor feature attribution drift as well as data drift?
How do you monitor model quality when labels arrive weeks later?
Should a drift alarm trigger automatic retraining?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More AWS Machine Learning interview questions