The words behind
the numbers.
A short, practical glossary for reading a setup: what the terms mean, and where each one shows up in the dataset. Concepts, not benchmarks.
VRAM
The GPU's own memory pool.
Model weights, the KV cache and the runtime's working buffers all have to fit in VRAM. When they do not, layers spill to system RAM over a much slower link and throughput collapses.
VRAM capacity matters, but so does bandwidth: a smaller card with fast memory can beat a larger pool that is bandwidth-starved.
Quantization
Storing weights at lower precision so a model fits.
Weights are commonly stored at 4-bit or 8-bit instead of 16-bit (FP16). A smaller quant fits a smaller card, at some cost in quality depending on the model and the level.
Weights and the KV cache are quantized separately. A model running at Q4 can keep its cache at Q8 for better quality — on vram.wiki, quantization is its own field, never folded into the model name.
Context length and KV cache
The token window, and the memory it costs.
The context is the number of tokens a model can attend to at once. A longer context needs a larger KV cache, which competes with the weights for the same VRAM.
There are two numbers: the configured limit, and the depth at which a speed was actually measured. A big advertised context is not free, and quality often degrades well before the limit.
Prefill and decode
Reading the prompt, then writing one token at a time.
Prefill processes the whole prompt in parallel and is fast (measured in prompt-processing tokens per second). Decode generates the answer one token at a time and is slower (measured in generation tokens per second).
When a setup reports a single speed, it is usually decode. Cache reuse makes a big difference: repeated prefixes can avoid re-running prefill.
Mixture of experts (MoE)
Only part of the model runs per token.
A MoE model routes each token through a subset of its parameters. A label like 35B-A3B means roughly 35B total parameters with about 3B active per token — fast, but the full weights still have to be resident in memory.
MoE models tend to be efficient on bandwidth-limited hardware for exactly this reason.
Runtimes
The inference engine that actually runs the model.
llama.cpp, vLLM, SGLang, Ollama, LM Studio and MLX are engines, not interfaces. The same model can be dramatically faster on one engine and card than another.
Engines differ on concurrency: vLLM and SGLang are built for serving many requests at once, while llama.cpp often wins on a single fast stream.
Harness
The agent loop or interface around the model.
A harness is the layer that manages tasks, tools, context and memory: OpenCode, Cline, Hermes, Open WebUI, or custom tooling. It is not the engine.
The same model can be reliable inside a strict harness and messy without one, which is why vram.wiki records workflow and harness separately from runtime.
RAG and embeddings
Search first, then answer.
Retrieval-augmented generation splits documents into chunks, embeds them into vectors, retrieves the most relevant chunks for a question, and gives them to the model as context.
Embeddings are a deterministic single pass, so they are cheap and safe to run locally. Retrieval quality depends mostly on how documents are chunked, not on the model size.
Batching and concurrency
Serving several requests at once.
Concurrency is how many requests run in parallel. Engines built for serving can keep several streams going and reuse cached prefixes; a single llama.cpp process favours one fast stream.
Reported tokens-per-second is usually single-stream. A setup that serves a whole office is a different kind of measurement.
Multi-GPU and offload
Spreading a model across cards or RAM.
Tensor parallelism splits one model across several GPUs and scales both capacity and speed. Layer offload keeps some layers on the CPU and RAM instead, trading speed for capacity.
Offload is what makes a model 'run' on too little VRAM; it is also where throughput usually falls off a cliff.
Unified memory
One pool shared between CPU and GPU.
Apple Silicon and AMD Strix Halo share a single memory pool. Large pools are relatively cheap and fit big models, but memory bandwidth is lower than a discrete card's VRAM.
This is why these machines run large models comfortably and still feel slow on long prompts.
Speculative decoding and MTP
Draft and verify to speed up decode.
A small draft model — or a model's own multi-token prediction head — proposes several tokens, and the main model verifies them in one pass. Accepted tokens make decode faster without changing the output.
It shows up as an MTP flag or a draft model alongside the main one.
Reasoning modes
Explicit thinking budgets, and their cost.
Some models expose a reasoning or thinking mode with a budget. More reasoning helps hard problems and hurts easy ones, where it produces long, looping answers — what people call overthinking.
Turning reasoning off, capping the budget, or stepping back a model version are common fixes recorded in the dataset.
Source score (upvotes)
The Reddit upvotes on the original comment, frozen at capture.
It reflects how the community reacted to the comment, not how complete or reliable the setup is: a high-scoring comment can omit its hardware, and a low-scoring one can describe every detail.
It is captured once at import and never updated, because it is a dated provenance signal rather than a live ranking.
Put the terms to work: open Compare, browse a pattern, or read a real setup.