How far to quantize

You know quantization compresses a model. The method question is: how far do you go for your task, and how do you prove you did not go too far? This is where the guesswork ends and measurement begins.

The starting point

Begin at Q4 (usually Q4_K_M). It is the community default for a reason: roughly half the size of Q8, with a quality loss most people cannot detect in daily use. Do not start more aggressive than that without a reason.

When to go higher

If your task is fact-heavy, mathematical, or requires long reasoning chains, the loss from quantization shows up first there. For those tasks, prefer Q5 or Q6 if the model still fits. The cost is VRAM, and it is usually worth it for work where a wrong answer is expensive.

When to go lower

If the only way to run the model you need is to quantize harder (Q3, IQ3, Q2), go lower, but treat it as an experiment. Aggressive quantization hurts reasoning and exact recall most, so it is fine for summarisation and extraction and risky for math and logic.

The method: measure, don't assume

The benchmark number will not tell you if the quant broke your task. The right way:

Pick a set of real examples for your task.
Run the same examples at Q4, Q5 and Q8.
Score the outputs on the thing you actually care about (correctness, tone, format).
Pick the highest quant that still fits and still passes your bar.

You are not choosing a quant; you are choosing the cheapest quant that keeps your task working. That is a measurement, not a preference.

The takeaway

Quantization is a dial, and "how far" is an empirical question. The dial that matters is not the one on the leaderboard, it is the one on your own evaluation. That evaluation is the next chapter, and it is the thing that makes every other decision here defensible.