Context, explained

Context is the window of text a model can "see" at once: your prompt, the conversation so far, and any documents you feed it. It is measured in tokens, not characters, and it is the single most important number about a local setup, because it decides how much the model can hold in mind.

What context actually is

A model does not remember your previous messages the way a person does. Every turn, the entire conversation is sent again, up to the context limit. When the window is full, the oldest part gets cut off. The model never "forgets" halfway through; it simply stops seeing the beginning.

This is why a model with a 4k window looks smart in a one-line chat and useless in a real project: the project does not fit in the window.

What context costs

Context is not free. The KV cache, which stores the state the model needs to attend over the whole window, grows linearly with context length. The practical effect:

• 4k context is cheap, 128k context is expensive in VRAM.
• A long context can use more memory than the model weights themselves.
• A runtime with a quantized KV cache (usually q8_0) cuts that cost roughly in half with almost no quality loss.

The three context mistakes

1. Never setting it. Most runtimes default to a small window. A 27B at 4k is a waste.
2. Silent truncation. The model does not warn you when it drops the start of your input. Your output looks fine but is answering only part of the question.
3. Budgeting only the weights. People buy VRAM for the model and then find the context they wanted does not fit.

The rule of thumb

Decide the context you actually need for your task, set it explicitly, and verify the runtime honours it. For coding agents and RAG over documents, that usually means 32k to 128k. Then budget VRAM for weights plus KV cache, not weights alone.