Context limits and silent truncation

The most dangerous failure in local models is the one that looks like success: silent truncation. The model runs out of context, drops the start of your input, and keeps answering confidently as if nothing was lost.

What actually happens

When the conversation or the document exceeds the context window, the runtime cuts the oldest tokens. There is no warning, no error, and the model is not "aware" it lost information. It just answers a shorter, incomplete question. Your output looks coherent and is subtly wrong.

Why it fools people

• The answer is fluent, so you trust it.
• The mistake is in what is missing, not in what is wrong, and missing is hard to notice.
• Longer, better models make it worse, because their confident tone covers the gap.

The two ways it bites

1. The conversation is too long. After enough turns, the model forgets the beginning, including the instructions you gave it at the start.
2. The document is too big. You feed a 200-page report into a 32k window; the model saw the first third and answered as if it read it all.

How to guard against it

1. Know your real context length. Set it explicitly and confirm the runtime honours it.
2. Watch the token count. Most tools show how many tokens the input used. If it is near the limit, assume truncation.
3. Chunk deliberately. For large documents, feed them in pieces or use RAG, rather than hoping the whole thing fits.
4. Test the boundaries. Ask the model a question whose answer lives at the very start of a long input. If it misses, you are truncating.

The deeper lesson

This is the same principle as the KV cache and the VRAM budget: context is a finite resource you manage, not an invisible quality you hope for. The setups that work reliably are the ones where someone checked the window size instead of assuming.

In the method track, this becomes a concrete check in your own evaluation: always include a "does it still see the beginning?" test.