Qwen 3.8 27B at ~6 tok/s on an 8GB RTX 4060 via RAM offload and MTP

On an i5-14400F gaming PC with an RTX 4060 8GB, 32GB DDR5 and Windows 11, the author runs Qwen 3.8 27B locally. The model is a 17.56GB Unsloth Dynamic V3 UD-Q4_K_XL GGUF served through llama.cpp on CUDA at a 32K context with MTP (speculative decoding) enabled. Because the full model cannot fit in 8GB of VRAM, llama.cpp splits it between the GPU and system RAM (--gpu-layers auto --fit on); roughly 6-7GB of VRAM is used, with the rest resident in memory. Reported generation is about 3.82 tok/s without MTP and about 5.99 tok/s with it; one MTP run accepted 99 of 126 draft tokens (~79%), roughly a 57% gain. A smaller UD-IQ4_XS quant (14.25GB) measured about 4.17 tok/s on the same setup, so the larger Q4_K_XL was actually faster and is the one kept. The machine is used from a laptop over Tailscale as a coding agent with a harness for repo reading, shell/tools and agent work.

The same machine was tested with Qwen3-VL 4B Q4_K_M for real invoice extraction. Feeding the image alone completed 5 of 6 invoices at 8-38 seconds each (~2.2K-2.6K prompt tokens); adding a large OCR transcript dropped it to 4 of 6 at 25-40 seconds and ~6.2K-8.3K tokens, so more context made the pipeline slower and less reliable. In a controlled comparison on the same invoice task, exact top-level extraction matched 22/24 for Qwen versus 5/24 for Gemma, with text-validation timings of 8.7 and 36.3 seconds for Qwen against 275.5 and 335.8 seconds for Gemma (about 7.8 generated tok/s but far more deliberation). The author's conclusion is that tok/s alone does not show which model is faster for useful work.