"It looks good" is not a measurement
Without a fixed set of inputs and a repeatable way of scoring the outputs, you cannot tell whether a change helped. Sampling means two runs of the same prompt differ, so eyeballing a handful of outputs measures the sample, not the system.
Why it is like this
Every other part of software has regression tests because nobody can hold a system's behaviour in their head. Probabilistic output makes that more true, not less.
Not to be confused with
A reason to test manually but carefully. The problem is not care, it is variance: a change that improves most cases and breaks a few looks like an improvement in any small sample.
In practice
The cheapest version that works: twenty real inputs, the expected properties of a good answer, and a score you can re-run. Teams that skip it are not moving faster, they are moving without instruments.
Source: NIST AI 600-1 — Generative AI Profile (AI RMF) — — Measure function, GAI evaluation. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Predict Check yourself
A prompt change makes the five examples a developer tried noticeably better. What can be concluded?
Show the answer
Very little — with sampled output and five cases, that is indistinguishable from variance, and it says nothing about cases they did not try.
The point is not statistical purity; it is that five samples from a varying system cannot distinguish a real gain from noise, or reveal what broke.