Sampling: temperature, top-p, top-k

Sampling parameters control how the model picks the next token. They do not change what the model knows; they change how focused or creative the output is. Most people leave them at defaults and then wonder why the output is either robotic or unhinged.

The three knobs

• Temperature — the main one. Higher means more randomness and variety; lower means more focused and predictable. Zero makes it deterministic. For factual work and coding, go low (0.1–0.3); for brainstorming and prose, a little higher (0.6–0.9).
• Top-p (nucleus sampling) — instead of a fixed number of candidates, it picks from the smallest set of tokens whose combined probability reaches p. Lower p is tighter, higher p is more varied. A common default is 0.9.
• Top-k — limits the choice to the k most likely tokens at each step. Lower is more conservative.

You do not need to tune all three aggressively. The practical rule is to treat temperature as the primary control and top-p/top-k as fine adjustments.

The failure modes

• Too low temperature on creative work → repetitive, boring text.
• Too high temperature on factual work → confident nonsense, because the model is now choosing unlikely tokens.
• Tuning them without measuring → you changed the output but have no idea what you changed.

What actually matters

For most people, one setting change fixes 90% of problems: lower the temperature for deterministic tasks. Coding, extraction, summarisation and anything where a wrong answer costs you all improve when the model stops gambling on rare tokens.

A note from the method track

The honest way to choose these is not a blog's "best settings" list, it is to test on your own task. The method track teaches exactly that: change one parameter, measure the output on your own data, keep the one that wins. Defaults are a starting point, not an answer.

Sampling is the cheapest lever you have. Learn what each knob does, then verify it on your real workload instead of trusting a benchmark.