Qwen 3.8 27B at 100K context on a single RTX 4080 16GB (NInfer 4080)

On a single RTX 4080 16GB, the author built NInfer 4080, a purpose-built engine for 16GB-class cards, to run ISTA-DASLab's Qwen 3.8 27B GSQ at 100K+ context with DFlash2 speculative decoding. It follows the same idea as the other NInfer variants: trade the wide model support of llama.cpp or vLLM for a large speed gain on one card, and ship it as a Docker image.

Measured results

In real use the author sees 2,000+ tok/s prefill on long prompts and roughly 150-200 tok/s decode on coding and ~100 on prose. Accuracy held at MBPP 90-92% and HumanEval 95-96%.

Context and accuracy trade

Reaching 100K context needs significant KV compression, because a full 256K F16 cache would need exactly 16GB of VRAM on this model. The author uses a compressed rk4v4-e8 KV variant and reports no noticeable quality loss versus higher-bit KV; small degradations are within measurement noise and can be traded back by lowering context. The harness is Pi (with a local SearXNG search).