vram.wiki
My space Add your setup
All lessons
CHAPTER 05 · CONFIGURING

Threads, batch and parallelism

Intermediate~2 min read

Threads, batch and parallelism

Once the model runs, people chase "more tokens per second" by buying hardware. Before you spend, check the runtime settings: three of them move throughput more than a new card sometimes, and they cost nothing.

Threads

For CPU-based work (or CPU offload), the number of threads controls how many cores the runtime uses. More threads is not always better: past the point where the work saturates, extra threads just add overhead. The usual advice is to match threads to physical cores, then nudge down if it stutters. This matters most when you run partly or fully on CPU.

Batch size

Batch size is how many independent sequences the runtime processes at once. A larger batch uses the GPU more efficiently and raises throughput, at the cost of more VRAM and sometimes slightly higher latency per request. For a single user, batch size is a small win. For a server with many users, it is the difference between a GPU sitting idle and a GPU earning its keep.

Parallel sequences

Some runtimes let you generate several completions in parallel (parallel decoding). This can look like a speedup because it fills the hardware, but it also multiplies memory use by the number of sequences. For interactive use, it is rarely worth the VRAM.

The honest guidance

  1. For one user, start with defaults and only touch threads if you are CPU-bound.
  2. Batch is a server concern. Do not tune it for a single interactive session and expect a miracle.
  3. Parallel sequences trade VRAM for wall-clock speed, and usually lose that trade for interactive work.

The real bottleneck

Remember the earlier lessons: decode speed is mostly bandwidth-bound, and long context is mostly KV-cache-bound. Settings like threads and batch help at the margins; they do not change the physical limits. Fix the budget (weights + cache) first, then tune these.

The catalog is full of setups that list their runtime settings. Compare two setups with the same card but different settings and you will see how much of the "speed" is actually configuration, not hardware.