What is online vs offline inference in classification systems?

Online inference provides real-time predictions, while offline inference processes large batches of data asynchronously.

Online inference is optimized for low latency and is used in APIs serving individual requests. Offline inference processes datasets in bulk, often using distributed systems like Spark. Online systems prioritize speed, caching, and lightweight models, while offline systems prioritize throughput and cost efficiency.