How would you reduce inference cost by an order of magnitude for a large model fleet?

Attack it in layers: right-size and compile the model, consolidate models onto shared endpoints, choose accelerator-optimised instances such as Inferentia, move non-interactive work to batch or async, then apply autoscaling and Savings Plans to what remains.

Inference typically dominates total ML spend because it runs continuously while training is bursty. The levers, in the order that usually pays best:

1. Do you need real-time at all? Moving a workload to batch transform is routinely a 10–20× saving because you stop paying for idle. This single question beats every micro-optimisation below.

**2.…