How do you handle catastrophic failure in LLM production systems?

Catastrophic failures are handled using fallback models, circuit breakers, cached responses, and system-wide rollback mechanisms.

Catastrophic failures include model outages, broken prompt updates, corrupted retrieval indices, or API-wide latency spikes. LLMOps systems mitigate these using multi-layer fallbacks: switching to smaller models, disabling RAG, serving cached responses, or rolling back to previous stable versions. Circuit breakers prevent cascading failures across dependent services.