What is autoscaling in ML inference systems?

Autoscaling dynamically adjusts compute resources based on traffic load.

In ML systems, autoscaling ensures low latency under peak load and cost savings during low traffic. It uses CPU/GPU utilization or request rate triggers to scale pods.