Reading prefill vs decode

When someone quotes "50 tokens per second," they are almost always quoting the wrong number, or only half of it. Inference has two phases with very different speeds, and conflating them is how benchmarks lie to you.

The two phases

Prefill (prompt processing) — reading your prompt. The model processes all the input tokens at once, in parallel. This is fast, often hundreds or thousands of tokens per second.
Decode (generation) — writing the answer. The model produces one token at a time, each depending on the previous. This is the slow part, and it is the number that decides how the interaction feels.

So "2000 tokens per second prefill, 50 tokens per second decode" is a normal, healthy setup. "50 tokens per second" quoted alone usually means decode, which is fine, but you should always know which one you are looking at.

Why the distinction matters

Short questions, long answers are decode-bound. Decode speed is what you feel.
Long documents, short answers are prefill-bound. A big report takes a while to read before the model even starts writing, even on a fast card.
Agentic work alternates between the two constantly: it reads (prefill), writes (decode), reads again. Both numbers matter, and a setup that is fast at one and slow at the other will feel uneven.

How to read the logs

Runtimes print both numbers. llama.cpp shows "prompt eval" (prefill) and "eval" (decode). When you see a setup report, check whether they give both. A single number is a partial picture.

The practical lesson

When you benchmark your own setup, record both. When you compare two setups in the catalog, do not compare one person's prefill to another's decode. And when you tune, know which phase you are trying to fix: context size and prompt length affect prefill, while model size and bandwidth dominate decode.

Knowing the difference is what separates "it runs" from "I understand my machine."