It reads tokens, not words or characters
A model does not see text. It sees tokens — sub-word fragments produced by a tokeniser specific to that model family. A word may be one token or several, and the same sentence costs a different number of tokens on different models.
Why it is like this
Working in sub-word units lets a fixed vocabulary cover any string, including words the model has never seen. The cost is that nothing about token counts is intuitive from looking at the text.
Not to be confused with
Words, or characters, or anything you can count in your head. Non-English text and code typically consume more tokens per character than English prose, so a limit that is comfortable in one language can be tight in another.
In practice
Every budget, rate limit and context size is denominated in tokens. If you estimate them by word count, your estimate is wrong by a factor that varies with the content — count with the provider's own tokeniser or counting endpoint.
Source: NIST AI 600-1 — Generative AI Profile (AI RMF) — §2 — GAI system characteristics. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Predict Check yourself
A team sizes its prompt budget by assuming roughly one token per word, validated on English marketing copy. Where does that first break?
Show the answer
On other languages and on code, which generally use more tokens per character — the ratio is a property of the content and the tokeniser, not a constant.
The ratio is not a constant and is not portable between models. This is why providers publish a token-counting endpoint rather than a conversion factor.