The context window is not memory
Each request is stateless. What looks like memory is the entire conversation being re-sent every turn, inside the context window. The model remembers nothing between calls; your application is doing the remembering.
Why it is like this
Statelessness is what makes the service horizontally scalable and what makes caching and replay possible. The conversation is a client-side construct.
Not to be confused with
A session the provider is holding for you. Two consequences people meet the hard way: cost and latency grow with conversation length because you resend everything, and anything you want remembered next week must be stored by you.
In practice
This is also why "it forgot what I told it" is usually a truncation or summarisation decision in your own code rather than anything the model did.
Source: NIST AI 600-1 — Generative AI Profile (AI RMF) — §2 — GAI system characteristics. Read on .
The OWASP Top 10 for LLM Applications is © the OWASP Foundation and published under a Creative Commons licence; NIST AI 600-1 is a US Government work in the public domain. These cards are independent commentary written in our own words. ytyk.org is not affiliated with or endorsed by the OWASP Foundation or NIST.
Say it back Check yourself
If each API call is stateless, what is actually happening when a chat 'remembers' earlier turns, and what two costs follow?
Answer out loud or in writing first, then open the points below.
The points a good answer makes
- The application re-sends the prior conversation in the context window each turn
- The model holds no state between calls
- Cost grows with conversation length, because the history is re-sent and re-charged
- Latency grows with it too
- Anything that must persist beyond the window has to be stored by the application
Almost every surprise about memory, cost growth and 'forgetting' follows from this one property.