This contributor runs a three-layer agentic architecture on an AMD Radeon RX 6650 XT with 8GB of VRAM and 32GB of system RAM, using Linux on Zorin OS 17 and Ubuntu 22.04. Layer 1 contains Markdown SOPs, Layer 2 is an orchestrator, and Layer 3 consists of deterministic Python tools. Cloud frontier models handle system architecture and direct design, while a local AMD ROCm vLLM instance runs continuously for code syntax checks, AST-based codebase mapping, confidential document processing, and long-running agent loops. The local model stack includes quantized Qwen 2.5 7B and Qwen 2.5 Coder 7B models, with Qwen 2.5 3B also available for lighter work. The contributor reports approximately 25–35 tokens per second at 2,048–4,096 context, with AWQ quantization required to fit the card. They use ROCm-specific settings including HSA_OVERRIDE_GFX_VERSION=10.3.0 and --enforce-eager, and report that vLLM removed a llama.cpp prefill recomputation penalty during agent tool returns.
Reported anonymously by an r/LocalLLM contributor · score 0