Profiling prefill vs decode

Before you change anything to make a setup faster, find out what is actually slow. Most "optimisations" are applied blind, and they fix the phase that was never the problem. Profiling is the discipline of measuring first.

The two phases, again

• Prefill reads the prompt, all tokens at once. It is compute-bound and usually fast.
• Decode writes the answer, one token at a time. It is bandwidth-bound and usually the slow part you feel.

A setup can be slow at either, or at both, and the fixes are different.

How to profile

1. Time each phase separately. Runtimes report prefill and decode times. Record them for a real workload, not a toy prompt.
2. Vary one thing at a time. Longer prompt and you see prefill grow. Bigger model and you see decode slow. Longer context and the cache grows.
3. Find the dominant cost. For interactive use it is usually decode; for document processing it is often prefill; for long conversations it can be the cache.

The fixes, by phase

• Prefill is slow → your prompt or context is large. Reduce the input, enable flash attention, or check the CPU/GPU split during prompt processing.
• Decode is slow → the model or quant is the cost. Smaller model, higher quant is not the fix (that is bigger), better bandwidth, or accept the speed.
• Everything is slow and memory is full → you are offloading or swapping. See the offload lesson.

The rule

Never optimise on a guess. Profile, find the bottleneck, change one thing, measure again. If the number did not move, you fixed the wrong thing and you should revert.

The connection to the method

This is the same loop as evaluation: hypothesise, measure, change one variable, compare. Profiling is just evaluation applied to speed instead of quality. Both are the difference between tuning and guessing.

The catalog reports the numbers people actually measured. That is a starting point; your own profile is the truth.