The words behind the numbers

Every field has jargon, and local LLMs have more than most. This is a fast map of the terms you will meet again, from the fundamentals to the runtimes. The full glossary on vram.wiki goes deeper; this is the reading path.

The model itself

• Parameters (7B, 27B, 70B) — the size of the model, in billions of weights. More parameters is usually smarter, always hungrier.
• Weights — the numbers the model learned during training. Their file size is the main VRAM cost.
• MoE (Mixture of Experts) — a model where only some weights activate per token, so a big "parameter count" runs like a smaller model. The A3B models are this.

Making it fit

• Quantization — compressing the weights to fewer bits (FP16 → Q8 → Q4 → Q3 → IQ3). Less VRAM and more speed, a little less quality.
• GGUF — the file format most local runtimes load, bundling a quantized model.

Running it

• Runtime — the engine that loads the model and produces tokens (llama.cpp, Ollama, vLLM…).
• KV cache — the memory that holds conversation state; it grows with context.
• Offload — putting some layers on CPU instead of GPU when VRAM runs short.
• Prefill vs decode — prefill reads your prompt (fast, parallel), decode generates one token at a time (slower). The "tokens per second" you see quoted is usually decode.

Tuning the output

• Temperature, top-p, top-k — knobs that control how random or focused the generation is.
• Context length — the token window the model can see.

The layers on top

• Harness — the wrapper that gives the model tools, memory and workflow. It matters more than the model for real work.
• RAG — retrieval-augmented generation: searching your own documents and feeding the results into the context.
• Embeddings — the vector representation of text that makes search and RAG work.

Read any term here and the later chapters unpack it. The glossary is the same knowledge, organised for lookup rather than for learning.