vram.wiki
My space Add your setup
All lessons
CHAPTER 04 · RUNNING MODELS

Which runtime for whom

Beginner~2 min read

Which runtime for whom

A runtime is the engine that loads a model and produces tokens. The model is the same file no matter which runtime serves it, but the runtime decides speed, concurrency and how much control you have. The choice is less about features than about who uses it and how.

The map

  • llama.cpp — the foundation. Fast on CPU and GPU, runs GGUF quants, extreme control through flags. Most other local tools are built on top of it.
  • Ollama — a friendly layer over llama.cpp. One command to pull and run, a clean API. The default for getting started and for scripting.
  • LM Studio — a desktop app with a visual UI. Best for people who want to click, not type flags.
  • vLLM / SGLang — server engines built for concurrency and throughput. Overkill for one person, essential when several clients share one model.
  • MLX — Apple Silicon native, leverages unified memory well. The choice on Mac when you want speed and the ecosystem around it.
  • ExLlamaV2 / TensorRT-LLM — GPU specialists that squeeze a specific card hard.

The decision that matters most: how many users

  • One user, one machine → Ollama or LM Studio. You want convenience, not a server.
  • One user who wants every knob → llama.cpp directly.
  • Several concurrent users or agents → vLLM or SGLang, which batch requests so the card is not sitting idle between turns.
  • Apple Silicon → MLX or LM Studio with MLX models.

Almost every "which is best?" thread is really a "for how many people?" question wearing a disguise.

Why the wrapper matters more than the model

For real work, the runtime is half the answer. The same 27B is near-useless in a bare chat box and excellent in a good harness with clean context. The runtime is where context, KV cache and batching are controlled, so the runtime you pick sets the ceiling on how well your model behaves.

A practical recommendation

Start with Ollama to get running in minutes, then graduate: if you hit a wall with context or speed, read the llama.cpp flags underneath it (Ollama is mostly passing them through). Move to vLLM only when you actually have concurrent users, not because a benchmark said so.

Check against real setups

The catalog lets you filter by runtime and see what each one is actually used for, on what hardware, with what reported speed. If you are deciding, look at the setups that look like yours: they already answered this question for you.