Quantization, explained

Quantization is compression for model weights. A model trained in 16-bit floating point is a large file; quantization stores the same weights in fewer bits, so the model shrinks, loads faster and runs faster, at a small cost in quality. It is the single biggest lever in the whole local game.

The ladder, from big to small

• FP16 / BF16 — the full, uncompressed weights. Best quality, largest file.
• Q8 — 8-bit. Loss is essentially imperceptible for most uses.
• Q6, Q5 — very small loss, a meaningful size saving.
• Q4 (Q4_K_M) — the default sweet spot. Roughly half the size of Q8, quality that most people cannot tell apart for daily use.
• Q3, Q2, IQ3, IQ2 — aggressive. Fits big models in small cards, but the loss starts to be visible, especially on reasoning and math.

The rule of thumb: Q4 is the default, Q5 or Q6 if you have the VRAM, Q3 and below only when the alternative is not running the model at all.

What the name means

A filename like Q4_K_M is not random. Q4 is the bit width. K means it uses the K-quant method (a smarter scheme that keeps important weights at higher precision). _M and _S are medium and small variants of that scheme. You do not need the details; you need to know that Q4_K_M is the safe default and that _S variants are slightly smaller but sometimes slightly worse.

What quantization actually costs

Quantization does not make the model "dumber" uniformly. It mostly adds a little noise to the weights. For summarisation, extraction and routine coding, the difference at Q4 is usually invisible. For hard math, long reasoning chains and exact facts, the loss shows up first. That is why the aggressive quants are a last resort, not a default.

The honest guidance

1. Start at Q4_K_M.
2. If quality matters and you have the room, go Q5 or Q6.
3. If you must fit a big model in a small card, quantize hard and test your actual task, because the benchmark number will not tell you if the quant broke your use case.

Quantization is a trade, not a trap. Used well, it is why a 24 GB card can do what used to need a server.