Use case
Agentic quality-bakeoff harness: 1 orchestrator plus up to 7 workers hitting one vLLM endpoint. Orchestrator prompts up to ~128K, workers up to ~16K each, batch cap sum(prompt+gen) <= 260K at concurrency <= 8. Workload is reasoning + needle recall (secret-code retrieval) plus vision smoke tests, scored with real stop tokens (temp 0, never ignore_eos).
Live config (2026-09-16)
- Model: unsloth_Qwen3.8-27B-NVFP4, dtype half (Turing has no bf16), quantization compressed-tensors NVFP4 W4A16, kv-cache-dtype fp8
- vLLM 0.27.1-tp2 SM75 fork, TP=2/PP=1, max-model-len 196608, max-num-seqs 8, max-num-batched-tokens 8192, gpu-memory-utilization 0.92
- Speculative: MTP K=3. Cudagraph PIECEWISE [4,8,16,32]. Prefix caching ON. Prefill backend flashqla_legacy. BLOCK_KV=4. mamba-cache-mode align.
- Tool parser qwen3_xml + auto (non-streamed only: streamed tool traffic hits a name=null lexer bug, so the harness uses non-streamed tool calls). Reasoning parser qwen3 with 2048 budget; workers can disable thinking for ~+25% decode.
- Image limit 8 per prompt (encoder budget constant). Env: NCCL over NVLink, tokenizers parallelism off, CPU thread caps to protect a 4-core host.
Measured performance
- Solo 65K: ~1227 tok/s prefill / ~29 tok/s decode. Solo 131K: ~79 tok/s decode, 100% needle recall.
- 8x16K workers: ~192 tok/s aggregate. Mixed 128K + 7x16K (~240K): ~140 tok/s aggregate.
- Prefix cache: 86K-131K TTFT ~8s warm vs ~50-90s cold (up to 91s observed cold, 183s at 144K cold). Second seat holds 70-75K vs 25-30K on the rejected tq4nc lane.
- MTP K=1 35.5 tok/s -> K=3 41.3 tok/s (+16.5%) in short bench; K=4 faster short-bench only, rejected for TTFT tax (132s vs 73s). BLOCK_KV 2->41.3, 4->49.4 (+20%), 8->43.7 regress (register spill on SM75). flashqla_legacy +29% 8K / +25% 64K prefill vs Triton/FLA.
- VRAM: ~20.9GB used per GPU, ~1GB free per GPU headroom. KV ceiling ~324K tokens total FP8 (~32KB/token total, 16KB/GPU).
What lost (do not retry without new evidence)
- W4A16/AWQ/FP8 weights: 72% EOS-fail + multilingual ramble, or 6/7 workers looping to cap under 8-way load.
- FP8 weights lane: no-start (KV cannot fit one 196K request under 29GB weights + no native SM75 MarlinFP8 + uncalibrated scale).
- ExLlamaV3: 3-6x slower (15.8 vs 46.5 solo, 30 vs 192 agg), no TP, fp16 OOM at 131K, flaky needle. Hobby lane only.
- PP=2: launch failure (needs 4 GPUs). MTP+PP crashes. MAMBA FULL-CUDAGRAPH reverted (no benefit). w8a16 MTP3 and tq4nc MTP4 fastest at 128-token short bench but -27-30% prefill/decode and 1/3 headroom at 65-131K serving.
Honest limits
- Cold 64-131K TTFT is 50-90s; plan for warm prefix reuse (same system prefix) or async workers.
- 2x192K concurrent does NOT fit the ~324K KV ceiling; the harness caps batches at ~260K and rope caps single requests at 262144 (recall-safe 192K).
- Telemetry stays OFF (disable-log-stats): enabling costs per-step CPU on a pinned 4-core host. Throughput measured via API usage + timing probes.
- Streamed tool calls are broken on this fork (name=null); use non-streamed tool traffic. Sampling must use temp 0 + real stops; ignore_eos / min_new=max causes repetitive-symbol spirals.
PROVENANCE
Submitted through the vram.wiki contributor dashboard · reviewed before publication.
Is this your setup? Claim it to keep the snapshot current