A small ISP (~7,600 customers) serves its agentic operations work on two identical machines, each with 4× RTX 3090 (250W-capped) and vLLM v0.30.0; llama-benchy measures ~5993 tok/s prefill and ~199 tok/s generation.
Start with your machine.
Capacity is the reported total for this setup. Multi-GPU and unified-memory layouts can behave differently.
A Framework 13 with a Ryzen 7 7840U and 64GB of unified memory runs privacy-first coding and homelab work through llama.cpp and hermes, where the larger model is too slow to use interactively.
Two single-GPU Ubuntu servers, one with a blower-modded Tesla V100 32GB and one with an RTX 3090, run Strata with Qwen 3.8 Flash Next at 262K context for local business management automation.
A Radeon 7900XTX with 24GB VRAM and 64GB DDR5 runs Qwen 3.8 Flash Next through Strata at about 60 tok/s up to 250k context while leaving VRAM for gaming.
A llama.cpp expert-streaming build runs the 177B Qwen 3.8 Flash Next MoE at roughly 11.5 tok/s on a 12GB RTX 5070 and 32GB DDR4-2400 by streaming experts from SSD.
TensorSharp runs the 176B Qwen 3.8 Flash Next MoE on a 16GB RTX 3080 Laptop with 32GB RAM, matching Strata on decode while finishing the benchmark process far faster.
Long-context decode benchmark of Qwen 3.8 Flash Next on an M1 Max 64 GB through an unofficial Splash/ds4 fork in real OpenCode agent sessions.
Four quants of a vision model measured on a 12GB RTX 5070 SFF with 64GB DDR5, through a purpose-built inference engine, at 37 to 55 GB combined RAM+VRAM.
Daily OCR of math- and table-heavy PDFs into LaTeX-correct Markdown and Word, using Qwen vision models on llama.cpp instead of a smaller model that misreads numbers.
With access to both Claude Fable and high-end local models, this developer chooses local because a less autonomous model keeps him involved in debugging and research work.
A self-built server with 1TB of ECC RAM and a mix of used 3090s and P100s produced internal tools whose economic impact is reported at ten times the hardware cost.
A laptop workflow combines Qwen Flash Next, llama.cpp and a DeepSeek harness, using system memory and SSD-backed loading to make a MoE model practical on 12GB of VRAM.