Building a two-server local AI ecosystem across 17 GPUs

Use case. This contributor uses local AI when cloud tokens run out, when work must stay private, and for everyday knowledge lookup, ideation, and sensitive questions (often medical). Local models also help control a Kubernetes cluster and assist with app development, depending on how many cloud tokens are left.

Models. Qwen 3.8 27B at q6_k and Qwen 3.8 Flash Next at q3_k_l.

Hardware. Two Linux servers (Debian, Proxmox, or Ubuntu Server):

• An open-frame rig: ASRockRack ROMED8-2T, an Epyc with 64 threads, 256GB DDR4, 7× RTX 3090.
• An ASRockRack 3U10G-F rack server: dual Xeon (48 threads), 256GB DDR4, 2× RTX 3090, 2× RTX 3080 10GB, 2× AMD V620 32GB, 4× Nvidia P100 16GB.

GPU roles. 4× 3090 run Qwen Flash Next on llama.cpp at 256K context for code and general work; 2× 3090 run Qwen 3.8 27B on vLLM, sharing a 380K context across conversations; 2× 3090 handle TTS/STT and training experiments; 1× 3090 runs ComfyUI. The 3080s take smaller TTS and small llama.cpp jobs, the P100s do embeddings and reranking for RAG (possibly OCR or video transcoding), and the V620s are still being assigned.

Runtime and tools. llama.cpp and vLLM, driven through OpenCode, Open WebUI, and a custom Rust harness. The author is building a full personal local-AI ecosystem: model management and download, remote llama.cpp/vLLM control, a private OpenAI-compatible router with server-side MCP, a RAG and memory layer over documents, chats, rules, and code repositories, plus TTS/STT and image generation (ComfyUI).

Honest limits. Prefill and reasoning are the pain points — prefill can take minutes, compaction is gruesome, and cloud models are far faster to first token and more succinct to read. The author keeps llama.cpp to one task at a time to minimize prefill and runs vLLM for the 27B to soften it.