The client is a 50-person company across HR, marketing and finance. The rig runs local RAG over internal documents, lightweight LLM chat, localized diffusion for visual assets and agentic coding, with bursty peaks of 3-5 concurrent requests, not 50 simultaneous streams.
Hardware
- 4x RTX 5090 (128GB total)
- ASUS W790E-SAGE SE, Intel Xeon w5-3423, 192GB ECC DDR5, 2TB NVMe
- Each card on its own PCIe 5.0 x16 riser, cooled by 18 server fans
Models
DeepSeek V4 Flash, Qwen 3.8 and Gemma 4 32B.
Caveat on the numbers
The published throughput figures (24 concurrent at 52 tok/s, ~2.4s to first token in short context, 15.3 tok/s at 25k-50k context, 4.3 tok/s at 100k for 8 concurrent) were measured on their 2x RTX 5090 box with Qwen 3.6 27B, not on this 4-card machine, which was not benchmarked after its models were updated.
Reported anonymously by an r/LocalLLM contributor · 315 upvotes at capture
Leave a comment