Tested Qwen 3.8 27B (NVFP4/MLX, 128K configured context, medium thinking) through Ollama 0.34.4 on a MacBook Pro M4 Max with 36 GB unified memory, for agentic coding. The point of the test was not "40 tok/s" but how long you actually wait before generation at real coding contexts.
Settings
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0
OLLAMA_NUM_PARALLEL=1
OLLAMA_KEEP_ALIVE=5m
MLX vs GGUF
| Model format | Decode |
|---|---|
| Q4_K_M / GGUF | ~10.7 tok/s |
| NVFP4 / MLX | ~40-41 tok/s |
A ~3.8x difference in decode throughput; the author waited around 18 minutes for one non-MLX landing-page task.
Context scaling (active input)
| Active input | Time before generation | Prompt | Decode |
|---|---|---|---|
| ~4K | 23.6 s | 198 tok/s | 46.6 tok/s |
| ~8K | 46.8 s | 178 tok/s | 41.4 tok/s |
| ~16K | 1m 30s | 176 tok/s | 30.8 tok/s |
| ~30K | 3m 15s | 157 tok/s | 31.9 tok/s |
Stress test with retrieval markers
| Context | Integrity | Time before generation | Prompt | Decode |
|---|---|---|---|---|
| ~32K | 4/4 PASS | 3m 29s | 152 tok/s | 29 tok/s |
| ~64K | 4/4 PASS | 7m 55s | 134 tok/s | 24 tok/s |
| ~96K | 4/4 PASS | 14m 55s | 107 tok/s | 20 tok/s |
| ~115K | 4/4 PASS | 21m 01s | 87 tok/s | 14 tok/s |
All four retrieval markers still passed at ~115K, so the runtime really does handle the long context; it is just not pleasant to use. Memory pressure rose too: by 64K available memory fell from 89% to 27% with swap rising 529 MB to 2.5 GB, and at 96K to 24% available with 2.9 GB swap.
Takeaway
Decode speed alone is misleading: prompt processing, time to first token and decode are three separate numbers, and memory pressure matters too.
Reported anonymously by an r/LocalLLM contributor · 98 upvotes at capture
Leave a comment