Context Overflow: What Happens When Your RAG Pipeline Runs Out of Space
How context overflow silently truncates the most relevant content, why it's hard to catch in testing, and the config changes that prevent it.
Your RAG system has been running for six months. Users are happy. Then one day, a power user with 200 turns of chat history sends a message. The context fills. The oldest retrieved chunks get dropped. The model loses track of the original question. The answer is incoherent. The user thinks the product is broken.
Context overflow is the failure that grows with usage. Day one users never see it. Power users hit it constantly. If you haven't designed for it, you've built a system that degrades for your best users.
The token budget allocation problem
A 200K token context sounds enormous. But fill it with: a 2,000-token system prompt, 50 turns of chat history at 500 tokens each (25,000 tokens), 10 retrieved chunks at 800 tokens each (8,000 tokens), tool call results (variable), and you're looking at 35,000+ tokens before the model says a word. At 500 turns of history, you're over 200K.
| Component | Tokens | Grows with? |
|---|---|---|
| System prompt | 1,000–5,000 | Product maturity |
| Chat history | 500 per turn | Usage — no ceiling if unmanaged |
| RAG context | 3,000–15,000 | Query complexity |
| Tool results | 500–10,000 | Agent step depth |
| Available tokens for output | Remainder | Shrinks as everything else grows |
The fixes
1. History summarisation at a threshold
When chat history exceeds X tokens, summarise the oldest half into 500 tokens and replace those turns with the summary. Preserves the essential context in a fraction of the tokens. Implement this proactively — don't wait until overflow, trigger at 60% of the window.
2. Tiered history
Keep the last 5 turns verbatim (they're most relevant). Summarise turns 6–20 into a 500-token block. Archive everything older into a memory store, retrieval-indexed. The model always has recent context and can query older context when it needs it.
3. Dynamic RAG budget
Instead of always retrieving 10 chunks, make retrieval context-aware. Shorter history → more RAG budget. Longer history → fewer, more precise RAG chunks. Retrieve by relevance score with a hard token cap, not a fixed chunk count.
4. Prompt caching for static context
If your system prompt is large and static, prompt caching turns it from a cost problem into a non-problem. The KV cache is reused; you only pay for the first request with that prefix. This doesn't reduce token count but eliminates the cost of the most stable component.
Design your token budget upfront, not when overflow happens. Allocate: system prompt X tokens, history budget Y tokens (with compaction at Z), RAG budget W tokens. Enforce these as hard limits at request construction time. Never discover overflow in production.
Context budget management →: Build tiered history and dynamic RAG budgeting in the Systems module.
Try it interactively
GenAI Systems Lab is a free platform for AI engineers — configure real failure modes, break things, and build the judgment that gets you hired.
Open GenAI Systems Lab →