How do you reduce inference latency for a model that is too slow in production?

Measure where the time goes first, then apply the applicable levers: model compilation and quantisation, dynamic batching, right-sized hardware, removing per-request overhead such as model loading or serial feature fetches, and caching repeated work.

Optimise from measurement, not intuition. SageMaker's CloudWatch metrics already split the problem: ModelLatency is time inside your container, OverheadLatency is SageMaker's handling. If OverheadLatency dominates, the model is not the problem — payload size, serialisation or queueing is.

Instrument inside the container to break ModelLatency down further:

```python import time, boto3 cw = boto3.client("cloudwatch")…