The KV cache, and why it is the real bottleneck

When people hit a wall with local models, they blame the model. Most of the time the wall is the KV cache, and it is the least understood part of the stack.

What it stores

To generate each new token, the model must attend over everything it has seen so far. Instead of recomputing that state every time, the runtime caches it: one key-value pair per token, per layer. That is the KV cache.

The point: it grows with context. At 4k context it is small. At 128k it can be several gigabytes, on top of the model weights.

Why it bites

1. It is invisible. The model "fits" on paper, then you set a long context and it overflows or offloads.
2. It is the main reason "I have 24 GB" does not simply mean "I run a 24 GB model." The cache needs room too.
3. It grows regardless of whether you use the context, once the window is set.

The two levers

1. Set the context deliberately. A smaller window means a smaller cache. Do not leave the runtime default if it is larger than you need.
2. Quantize the KV cache. Most runtimes let you store the cache in 8-bit (often the q8_0 flag) instead of 16-bit. This roughly halves the cache size with almost no quality loss. It is the highest-value flag most people never touch.

A quick budget

A rough rule: a 27B at Q4 is about 16 GB of weights. At 64k context with a quantized cache, budget several more gigabytes for the cache. The exact number depends on the model and runtime, but the lesson is constant: budget weights plus cache, never weights alone.

Why it matters for choosing

Two setups can run the same model and one feels great while the other stutters, purely because one set the context and quantized the cache and the other left defaults. This single variable explains a surprising share of "local is too slow" complaints. Check the catalog: the setups that report smooth long-context use are the ones that got this right.