You've probably heard that goldfish have a three-second memory. It's a popular myth, but it's not true. Goldfish can actually remember things for months.
Large language models, on the other hand? They really do forget. Not because they're simple, but because they have a hard limit on how much they can hold in mind at once. It's called a context window, and understanding it will save you a lot of frustration.
In Ted Lasso, the advice is "be a goldfish" because forgetting mistakes helps you move on. With AI, the forgetting isn't optional or therapeutic. It's just how the technology works.
The conversation's short-term memory
Think of a context window as a whiteboard. Every message you send, and every response the AI generates, gets written on that whiteboard. The AI can only see what's currently on the board when deciding what to say next.
The whiteboard has a fixed size, measured in tokens (roughly words or word-pieces). Modern models like Claude have large context windows, somewhere between 128,000 and 200,000 tokens, which is roughly the length of a novel. But in a long conversation or complex task, even that can fill up.
The animation below shows what happens as a conversation progresses:
What happens at the limit
When the context window fills up, something has to give. Older messages get dropped or compressed to make room for new ones. The AI literally loses access to the early parts of your conversation.
There's also a phenomenon called "lost in the middle." Even within a large context window, models tend to recall information at the beginning and end better than what's in between. It's a known quirk of how attention mechanisms work.
This explains why a long chat session can start to feel like the AI "forgot" important details you mentioned earlier. It might have.
Working with the limits
A few practical tips:
- Put important information at the beginning of your prompt. The start of the context window gets the most reliable attention.
- For long sessions, periodically summarize the conversation and start fresh. Think of it as clearing the whiteboard and writing a clean set of notes.
- Understand that the longer you chat, the more unpredictable things can get. If responses start drifting, it might be time to reset.
None of this means AI tools aren't useful for extended work. It just means you'll get better results if you work with the limitations instead of being surprised by them.