Fitting Qwen 27B and a 96K context onto a single 3090

This person runs Qwen 3.8 27B with Cline in VS Code on a single RTX 3090 24GB, and reports it as the first local model that really impressed them. The trick that makes it fit is quantising the KV cache to q8_0, which they believe is 8-bit native, shrinking the cache enough to hold both the full model and a 96K context inside 24GB of VRAM without spilling into system RAM.

On a headless server they end up with about 1.5GB to spare. Anything larger starts offloading layers. The model is not fast because it thinks a lot, but once it finishes churning they see about 43 tokens per second.

Correct or update this entry

Is this your setup? Claim it to keep the snapshot current