How do you design a production-grade LLM debugging system?

A debugging system reconstructs full LLM request pipelines to diagnose failures across prompts, retrieval, and model inference.

LLM debugging requires full trace reconstruction: input prompt, retrieved documents, model version, decoding parameters, intermediate tokens, and output. Debugging tools allow replaying requests and comparing against previous versions. This helps identify whether failures are due to prompt changes, retrieval drift, or model updates.