A model can grade, and brings its own biases
Using a model to score another model's output is practical and scales, but the judge is a model: it has position biases, length biases and a tendency to prefer text that looks like its own. A judge has to be validated against human labels before its scores mean anything.
Why it is like this
Human grading does not scale to every change, and many qualities — helpfulness, tone, faithfulness to a source — have no programmatic test. A judge is the only thing that covers them cheaply.
Not to be confused with
An objective grader. An unvalidated judge measures agreement with itself, and a judge given a vague rubric produces confident numbers with no referent.
In practice
Validate on a few dozen human-labelled cases, measure the agreement, and re-check it when you change the judge's model or its rubric.
Source: NIST AI 600-1 — Generative AI Profile (AI RMF) — — Measure function, GAI evaluation. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Say it back Check yourself
Name three things to do before trusting an LLM judge's scores.
Answer out loud or in writing first, then open the points below.
The points a good answer makes
- Validate it against human labels on a sample and measure agreement
- Give it a specific rubric rather than a vague quality question
- Control for known biases — position, length, self-preference
- Re-validate when the judge's model or rubric changes
- Keep the judge separate from the model being judged where possible
A judge is a measurement instrument, and instruments get calibrated before they get believed.