What is LLM-as-a-judge evaluation and its limitations?

It uses another LLM to evaluate outputs, but suffers from bias and inconsistency.

LLM-as-a-judge uses a large language model to score or rank outputs from another model. It is scalable and correlates moderately with human judgment, but suffers from positional bias, verbosity bias, and self-preference bias. It can also propagate evaluator model errors. Calibration against human labels is necessary for reliability.