The output is sampled, not looked up
A model produces a probability distribution over the next token and then one is sampled from it. Settings such as temperature and nucleus sampling change the shape of that distribution — flatter means more varied, sharper means more predictable.
Why it is like this
Always taking the single most likely token produces repetitive, degenerate text. Sampling buys fluency and variety; the price is that the same input can give different outputs.
Not to be confused with
A dial between "accurate" and "creative". Lowering temperature makes output more predictable, not more correct — a confidently wrong answer gets more confidently wrong, not truer. And even at the lowest setting, identical output is not guaranteed in practice.
In practice
Any test that asserts an exact string will be flaky. Tests on these systems assert properties — contains, matches a schema, passes a grader — not equality.
Source: NIST AI 600-1 — Generative AI Profile (AI RMF) — §2 — GAI system characteristics. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Predict Check yourself
A team sets temperature to zero to make a classifier 'deterministic and accurate'. What have they actually bought?
Show the answer
More predictable output, not more correct output — and still not a guarantee of identical results run to run.
Predictability and correctness are different axes. A wrong answer at low temperature is simply a wrong answer you will get repeatably.