Fine-tuning teaches behaviour, not facts
Fine-tuning adjusts weights on examples, so it is good at form, tone, structure and task behaviour. It is a poor and expensive way to install facts: facts learned in weights cannot be cited, audited, corrected or expired, and parameter-efficient methods such as LoRA adapt a small number of added parameters rather than rewriting what the model knows.
Why it is like this
Weights are a lossy, opaque store. Retrieval keeps facts in a database you can query, update and show a source for — which is also what makes an answer defensible.
Not to be confused with
A way to give the model your data. If a fact changes, a fine-tuned model is wrong until you retrain; a retrieval system is right as soon as the row changes.
In practice
Fine-tune for "always reply as a structured object in our house style". Retrieve for "what is the current SLA for this customer".
Source: Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (arXiv:2106.09685) — §1 and §4. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Odd one out Check yourself
Three of these are good reasons to fine-tune. Which is not?
Show the answer
A body of company facts the model should know
Facts belong where they can be updated and cited. Everything else on the list is behaviour, which is what weights are good at.