What is cold start problem in ML inference systems?

Cold start refers to high latency when initializing models or services after inactivity.

Cold start occurs when models must be loaded into memory, dependencies initialized, or containers spun up after scaling events or inactivity. This is especially problematic in serverless ML systems. Solutions include warm pools, model preloading, and persistent inference services.