How do you A/B test machine learning models in production correctly?

Split traffic with production variants using a consistent hashing key so a user always sees one model, size the test from a power calculation on a business metric, run for full business cycles, and avoid peeking — with shadow mode first for high-risk changes.

Offline metrics measure a proxy; A/B tests measure the outcome. They frequently disagree, which is the reason to test at all.

Traffic splitting. SageMaker production variants make this straightforward:

```python sm.create_endpoint_config( EndpointConfigName="ranker-experiment-v12", ProductionVariants=[ {"VariantName": "control", "ModelName": "ranker-v11", "InitialInstanceCount": 4, "InstanceType": "ml.c5.2xlarge", "InitialVariantWeight": 90}, {"VariantName": "treatment", "ModelName": "ranker-v12", "InitialInstanceCount": 1, "InstanceType…