Sizes, families and MoE

Models come in sizes from under a billion parameters to several hundred billion. The size is the first thing to understand, because it sets both what the model can do and what hardware it needs.

The rough ladder

• 0.5B – 3B — tiny, fast, single-purpose. Good for classification, extraction, simple assistants, and for running on phones or CPUs.
• 7B – 9B — the everyday workhorse. Fits in 8–16 GB, fast, good enough for many real tasks. The most common local model class.
• 14B – 27B — the sweet spot for quality. The 27B models are the point where local output starts to feel close to a good cloud model for coding and reasoning.
• 35B – 70B — high quality, needs 24 GB and up, or unified memory. Slower, but meaningfully smarter on hard tasks.
• 100B+ — datacenter territory. Rarely worth it at home, except on a large unified-memory machine or a server.

Size is not the whole story. A well-made 27B can beat a mediocre 70B on a specific task, and a heavily quantized big model often loses to a smaller model at a higher quant.

MoE: the middle path

A mixture-of-experts model (often written like "35B-A3B") has a large total parameter count but only activates a small fraction per token. The practical effect: it thinks like a much bigger model while running at the speed and memory cost of a smaller one. The A3B models are exactly this, and they are the most interesting middle ground for modest hardware.

When a spec says "35B-A3B", read it as: "35B of knowledge, 3B of active weights per token."

How to choose

Match the size to two things: your hardware (see the VRAM lesson) and your tolerance for waiting. A model that fits comfortably and runs fast will get used daily; a model that barely fits will be a toy you stop using. Choose the biggest model that stays comfortably within your VRAM and your patience.

Then check the catalog: filter by model family and see the sizes people actually run on hardware like yours, with the real numbers.