Reading a GPU spec

When people compare cards for local inference, they fixate on the model name. The name is the least useful part. What actually decides whether a card is good at this job is three numbers, and one compatibility question.

The three numbers

1. VRAM — the hard ceiling. A model either fits or it does not. Nothing else compensates for too little VRAM.
2. Memory bandwidth (GB/s) — how fast the card can move data. Decode speed is almost entirely bandwidth-bound, so two cards with the same VRAM but different bandwidth will run the same model at very different speeds.
3. Compute — mostly irrelevant for using a model. It matters for fine-tuning, not for running one.

For running models, the ranking is: VRAM first, bandwidth second, everything else distant third.

The compatibility question

• NVIDIA is the path of least resistance. CUDA is the most-supported backend and nearly every tool assumes it.
• AMD works via ROCm or Vulkan, but support is patchier and you will spend more time making things run.
• Apple uses unified memory (next lesson) and Metal; the memory is shared, so a Mac with "only" 64 GB can run models that no discrete card can.

Why a 24 GB 3090 beats a 16 GB 5080 here

For inference, the 3090's 24 GB lets a whole class of models (27B at Q4, with room for context) fit where the 16 GB card must quantize harder or offload. Even though the 5080 is faster and newer, in this specific use the extra VRAM wins. That is the whole lesson: for running models, capacity usually beats raw speed.

Reading a card for inference, step by step

1. Find the VRAM. That is your model size budget.
2. Find the bandwidth. That is your realistic tokens-per-second.
3. Check the backend support (CUDA/ROCm/Metal) before you buy for local AI specifically.
4. Ignore the marketing numbers about ray tracing and clock speed. They are for games.

The catalog on vram.wiki lets you filter by exact GPU and see what people actually run on each card, with their real reported speed. That is the spec sheet that matters.