vram.wiki
My space Add your setup
Back to directory
Hybrid orchestration · 2026-09-13

A three-layer AMD ROCm agent architecture for 24/7 local automation

A Radeon 6650 XT runs quantized Qwen models through vLLM and Ollama, while a custom three-layer harness handles private, deterministic automation around the clock.

This contributor runs a three-layer agentic architecture on an AMD Radeon RX 6650 XT (8GB VRAM) with 32GB of system RAM, on Linux (Zorin OS 17 / Ubuntu 22.04). It is documented in the author's open guide, "Optimizing LLMs for custom and consumer hardware".

How it is built

  • Layer 1: Markdown SOPs
  • Layer 2: an orchestrator
  • Layer 3: deterministic Python tools

Cloud frontier models handle system architecture and design, while a local AMD ROCm vLLM instance runs continuously for:

  • code syntax checks
  • AST-based codebase mapping
  • confidential document processing
  • long-running agent loops

Stack and settings

  • Models: quantized Qwen 2.5 7B and Qwen 2.5 Coder 7B, plus Qwen 2.5 3B for lighter work
  • Speed: AWQ quantization is required to fit the card; the guide's 8GB tier table lists 45-60 tok/s for a 3-8B model in Q4_K_M or AWQ once fully in VRAM
  • ROCm settings: HSA_OVERRIDE_GFX_VERSION=10.3.0 and --enforce-eager
  • The guide notes vLLM's --enable-prefix-caching reuses the KV-cache for repeated system prompts, cutting prompt prefill latency by over 80%.
VERIFIABLE SOURCE

Published from an open-source project repository · reviewed before publication

View repository

Correct or update this entry

SETUP HISTORY

Snapshots over time

3 versions
CURRENT2026-09-13A three-layer AMD ROCm agent architecture for 24/7 local automation

A Radeon 6650 XT runs quantized Qwen models through vLLM and Ollama, while a custom three-layer harness handles private, deterministic automation around the clock.

Hybrid orchestration1 machineQwen 2.5 7B, Qwen 2.5 Coder 7B, Qwen 2.5 3BvLLM (ROCm 6.x), Ollama fallback
ChangedDescription
Compare with revision 2
DescriptionPreviousThis contributor runs a **three-layer agentic architecture** on an AMD Radeon RX 6650 XT (8GB VRAM) with 32GB of system RAM, on Linux (Zorin OS 17 / Ubuntu 22.04). ## How it is built - **Layer 1**: Markdown SOPs - **Layer 2**: an orchestrator - **Layer 3**: deterministic Python tools Cloud frontier models handle system architecture and design, while a **local AMD ROCm vLLM** instance runs continuously for: - code syntax checks - AST-based codebase mapping - confidential document processing - long-running agent loops ## Stack and settings - Models: quantized **Qwen 2.5 7B** and **Qwen 2.5 Coder 7B**, plus **Qwen 2.5 3B** for lighter work - Reported speed: **~25-35 tok/s** at 2,048-4,096 context, with **AWQ** quantization required to fit the card - ROCm settings: `HSA_OVERRIDE_GFX_VERSION=10.3.0` and `--enforce-eager` - The contributor reports vLLM removed a llama.cpp prefill-recomputation penalty during agent tool returns.This snapshotThis contributor runs a **three-layer agentic architecture** on an AMD Radeon RX 6650 XT (8GB VRAM) with 32GB of system RAM, on Linux (Zorin OS 17 / Ubuntu 22.04). It is documented in the author's open guide, "Optimizing LLMs for custom and consumer hardware". ## How it is built - **Layer 1**: Markdown SOPs - **Layer 2**: an orchestrator - **Layer 3**: deterministic Python tools Cloud frontier models handle system architecture and design, while a **local AMD ROCm vLLM** instance runs continuously for: - code syntax checks - AST-based codebase mapping - confidential document processing - long-running agent loops ## Stack and settings - Models: quantized **Qwen 2.5 7B** and **Qwen 2.5 Coder 7B**, plus **Qwen 2.5 3B** for lighter work - Speed: **AWQ** quantization is required to fit the card; the guide's 8GB tier table lists **45-60 tok/s** for a 3-8B model in Q4_K_M or AWQ once fully in VRAM - ROCm settings: `HSA_OVERRIDE_GFX_VERSION=10.3.0` and `--enforce-eager` - The guide notes vLLM's `--enable-prefix-caching` reuses the KV-cache for repeated system prompts, cutting prompt prefill latency by over 80%.

This is the currently published snapshot.

REVISION 22026-09-13A three-layer AMD ROCm agent architecture for 24/7 local automation

A Radeon 6650 XT runs quantized Qwen models through vLLM and Ollama, while a custom three-layer harness handles private, deterministic automation around the clock.

Hybrid orchestration1 machineQwen 2.5 7B, Qwen 2.5 Coder 7B, Qwen 2.5 3BvLLM (ROCm 6.x), Ollama fallback
ChangedDescription
Compare with revision 1
DescriptionPreviousThis contributor runs a three-layer agentic architecture on an AMD Radeon RX 6650 XT with 8GB of VRAM and 32GB of system RAM, using Linux on Zorin OS 17 and Ubuntu 22.04. Layer 1 contains Markdown SOPs, Layer 2 is an orchestrator, and Layer 3 consists of deterministic Python tools. Cloud frontier models handle system architecture and direct design, while a local AMD ROCm vLLM instance runs continuously for code syntax checks, AST-based codebase mapping, confidential document processing, and long-running agent loops. The local model stack includes quantized Qwen 2.5 7B and Qwen 2.5 Coder 7B models, with Qwen 2.5 3B also available for lighter work. The contributor reports approximately 25–35 tokens per second at 2,048–4,096 context, with AWQ quantization required to fit the card. They use ROCm-specific settings including HSA_OVERRIDE_GFX_VERSION=10.3.0 and --enforce-eager, and report that vLLM removed a llama.cpp prefill recomputation penalty during agent tool returns.This snapshotThis contributor runs a **three-layer agentic architecture** on an AMD Radeon RX 6650 XT (8GB VRAM) with 32GB of system RAM, on Linux (Zorin OS 17 / Ubuntu 22.04). ## How it is built - **Layer 1**: Markdown SOPs - **Layer 2**: an orchestrator - **Layer 3**: deterministic Python tools Cloud frontier models handle system architecture and design, while a **local AMD ROCm vLLM** instance runs continuously for: - code syntax checks - AST-based codebase mapping - confidential document processing - long-running agent loops ## Stack and settings - Models: quantized **Qwen 2.5 7B** and **Qwen 2.5 Coder 7B**, plus **Qwen 2.5 3B** for lighter work - Reported speed: **~25-35 tok/s** at 2,048-4,096 context, with **AWQ** quantization required to fit the card - ROCm settings: `HSA_OVERRIDE_GFX_VERSION=10.3.0` and `--enforce-eager` - The contributor reports vLLM removed a llama.cpp prefill-recomputation penalty during agent tool returns.

Archived published snapshot — kept for context and comparison.

REVISION 12026-09-13A three-layer AMD ROCm agent architecture for 24/7 local automation

A Radeon 6650 XT runs quantized Qwen models through vLLM and Ollama, while a custom three-layer harness handles private, deterministic automation around the clock.

Hybrid orchestration1 machineQwen 2.5 7B, Qwen 2.5 Coder 7B, Qwen 2.5 3BvLLM (ROCm 6.x), Ollama fallback

Archived published snapshot — kept for context and comparison.

Read next

Hybrid orchestrationFrontier plans the tickets, Qwen writes the codeHybrid orchestrationLocal as a verified subagent — but not trusted to touch files aloneHybrid orchestrationA local model and Codex reviewing each other's work, with a hard stop on disagreement

Discussion

0 comments

No comments yet. Questions and corrections on this setup are welcome.

Leave a comment

Comments are reviewed before they appear. Keep it civil, cite your sources, and stay on topic.