Tuning the KV cache

The KV cache is the invisible memory cost that grows with context. Tuning it is one of the highest-value, lowest-effort optimisations you can do, and it is almost always measured wrong or not at all.

The one flag that matters most

Quantize the KV cache (usually the q8_0 cache type). Storing the cache in 8-bit instead of 16-bit roughly halves its size, with a quality loss that is almost always negligible. For long-context work, this is the difference between fitting and overflowing.

This is the single highest-return setting most people never touch. Do this before anything else.

What else to adjust

• Cache type. Beyond q8_0, some runtimes offer smaller cache quants (q4) at a larger quality risk. Start at q8_0, go smaller only if you measure that you need to and that it still works.
• Context size. Set the smallest window that fits your task. A window you do not use is still a cache you pay for.
• Flash attention. If available, it reduces the cache's memory and compute. Turn it on when the runtime supports it and it does not change your results.

The method: measure both size and quality

1. Find how much VRAM the cache actually uses at your target context, before and after the change. Runtimes report it, or watch the memory.
2. Verify quality did not move: run your own task at both cache settings and compare.
3. Keep the setting that saves memory without hurting your task.

Why "measure quality" matters

A smaller cache quant can subtly degrade long-context reasoning, and you will not see it in a single demo. The only way to know is to run your own evaluation (next chapter) at both settings. Without that, you are optimising blind.

The takeaway

The KV cache is a dial, like quantization. Tune it for your task, measure the memory and the quality, and keep the best trade. It is the cheapest real win in local inference, and it is sitting in your runtime's flags right now.