What is SageMaker Clarify and how do you use it for bias detection and explainability?

Updated Aug 1, 2026

Short answer

Clarify computes pre-training bias metrics on data, post-training bias metrics on model predictions, and SHAP feature attributions for global and per-prediction explanations — usable as pipeline steps and as ongoing endpoint monitors.

Deep explanation

Clarify answers two regulated-industry questions: is this model treating groups differently? and why did it produce this decision?

Pre-training bias (data, before any model exists):

Class Imbalance (CI) — is the facet under-represented?
Difference in Proportions of Labels (DPL) — do positive outcomes differ by group in the training labels?

Post-training bias (model predictions):

DPPL — difference in proportion of positive predictions between groups.
Disparate Impact (DI) — ratio of favourable outcome rates; the "four-fifths rule" heuristic lives here.
Recall/accuracy difference — is the model simply worse for one group?

Explainability — Kernel SHAP attributions, globally and per prediction.

Python

1from sagemaker import clarify
2
3processor = clarify.SageMakerClarifyProcessor(
4    role=role, instance_count=1, instance_type="ml.m5.xlarge")
5
6bias_config = clarify.BiasConfig(
7    label_values_or_threshold=[1],       # 1 = favourable outcome
8    facet_name="age_band",               # the attribute under scrutiny
9    facet_values_or_threshold=["18-25"], # the group being compared
10    group_name="region",                 # for conditional metrics
11)
12
13processor.run_bias(
14    data_config=clarify.DataConfig(
15        s3_data_input_path="s3://my-bucket/features/train/train.csv",
16        s3_output_path="s3://my-bucket/clarify/bias",
17        label="approved", headers=headers, dataset_type="text/csv"),
18    bias_config=bias_config,
19    model_config=clarify.ModelConfig(
20        model_name="loan-model-v4", instance_type="ml.m5.xlarge",
21        instance_count=1, accept_type="text/csv"),
22    model_predicted_label_config=clarify.ModelPredictedLabelConfig(probability_threshold=0.5),
23    pre_training_methods="all", post_training_methods="all",
24)
25
26# SHAP explainability
27processor.run_explainability(
28    data_config=..., model_config=...,
29    explainability_config=clarify.SHAPConfig(
30        baseline=[baseline_row],   # the reference point attributions are measured against
31        num_samples=200, agg_method="mean_abs"),
32)

The SHAP baseline is the conceptual crux: attributions answer "how did this prediction differ from the baseline, and which features explain the difference?" A poorly chosen baseline (all zeros for features where zero is impossible) yields attributions that are mathematically valid and practically meaningless. Use a representative sample — often the median row or a small cluster-representative set.

Interpretation discipline matters too: bias metrics are diagnostics, not verdicts. A DPPL gap may reflect genuine differences in the underlying population, or it may encode historical discrimination. Clarify tells you a gap exists; determining whether it is unlawful or unjustified is a human and legal judgement.

Real-world example

A lender found DI of 0.71 against a protected group — below the 0.8 heuristic. SHAP showed the driver was a postcode-derived feature acting as a proxy for a protected attribute the model never saw directly. They removed the feature, retrained, and DI moved to 0.94 with a negligible AUC drop. The Clarify report attached to the registered model became the evidence in their fair-lending review.

Common mistakes

- Assuming dropping the protected attribute removes bias — proxies are the norm, not the exception.
- Using an all-zeros SHAP baseline, producing attributions no one can interpret.
- Treating a single metric as pass/fail
the metrics disagree by construction and you should report several.
- Running Clarify once at launch and never again, missing bias drift as populations change.
- Confusing SHAP with causality — attributions explain the model's behaviour, not the world's mechanism.

Follow-up questions

Why can removing a protected attribute fail to remove bias?
What is the difference between global and local explanations?
How do you monitor bias continuously in production?

Short answer

Deep explanation

Real-world example

Common mistakes

Follow-up questions

More AWS Machine Learning interview questions