Memory and offload, again

You met offload in the fundamentals. This is the deeper version, for when you are actually squeezing a setup: where the bytes really go, and how to decide what belongs where.

The full memory picture

For one model at a given context, memory is the sum of:

Weights — the model file, the dominant cost.
KV cache — grows with context.
Compute buffers — activations and scratch space, smaller but real.
Overhead — the runtime itself, the OS, and (on unified memory) everything else running.

Budget all four, not just the weights. Most "it should fit" miscalculations forget the cache and the overhead.

Offload, precisely

Offload moves weights to the CPU in whole layers. The finer points:

Where the layers go matters. Keeping the most active layers on GPU and offloading the less active ones minimises the speed hit, but most runtimes offload in a simple, predictable order. Know which one you are using.
RAM speed is the ceiling for offloaded layers. Slow RAM means a slow offloaded model. If you plan to offload a lot, fast RAM is part of the budget.
Offload is stepwise, not smooth. Speed drops in jumps as whole layers move to CPU. "A little offload" can already cost a lot.

The decision method

List your four memory costs at your target context.
If they fit on GPU, do not offload.
If they do not, offload the minimum number of layers, then measure speed.
If the speed is unacceptable, step down a model size or quant rather than offloading more.

The honest rule

Offload is a crutch, not a strategy. It lets you try a big model, but a setup that depends on heavy offload for daily work is a setup that is one model-size too ambitious. The catalog has both kinds; the setups that feel good long-term are the ones that fit.

Measure where the bytes go, and the speed you get back, before you decide the setup is done.