How do you evaluate and monitor an LLM application in production?

Evaluating an LLM in production is primarily about observability, not just offline benchmark scores. A production-grade system continuously measures quality, latency, cost, safety, and business outcomes using a combination of automated evaluators, human review, canary deployments, and telemetry, while treating prompts, models, and retrieval pipelines as versioned software artifacts that can regress independently.

Most engineers think of evaluation as "run a benchmark before deployment." Senior engineers know that production evaluation is a continuous feedback system.

An LLM application has at least five independently changing components:

Any one of them can silently degrade quality without changing the others.…