CPU/GPU offload

Offload is what you do when the model does not quite fit in VRAM: put some layers on the GPU, and let the CPU and system RAM carry the rest. It is the difference between "it does not run" and "it runs, slowly."

How it works

A model is a stack of layers. A runtime can keep the first N layers on the GPU and run the remaining layers on the CPU, using system RAM instead of VRAM. The model still works; the CPU just does part of the work each token.

What it costs

The cost is speed, and it is steep. Every token has to cross the slow CPU↔GPU boundary, and the CPU is far slower at this kind of math than a GPU. A model that runs fully on GPU at 40 tokens per second can drop to single digits once a large part of it offloads. The more you offload, the more it hurts.

When it is the right call

• Big model, small VRAM, lots of RAM. A 70B on a 24 GB card with 64 GB of system RAM. It will be slow, but it will run, and for batch or overnight work that is often fine.
• One-off experiments. You want to try a model once without buying hardware. Offload lets you see it before you commit.

When it is a trap

For interactive daily use, heavy offload makes a model unpleasant enough that you stop using it. A smaller model that runs fully on GPU is usually the better tool than a bigger model crawling through the CPU. Remember the trade-off triangle: offload buys VRAM and pays for it in speed.

The practical guidance

1. If your model fits in VRAM, keep it there. Do not offload for no reason.
2. If it almost fits, offload only the few layers needed, not half the model.
3. If it needs heavy offload for interactive work, step down a model size or quant instead.

The catalog records who offloads and who does not, with their real numbers. Comparing an offloaded setup to a fully-on-GPU one is the fastest way to see exactly what that speed cost buys you.