Setup
- CPU / GPU / RAM: i9-13900K / RTX 4090 48 GB (memory-modded, driver 617.14) / 128 GB DDR5-6000
- OS / engine: Windows 11, Strata 0.1.38 (sm_89), IQ3_S
- Config: 262K context,
--kv int8 --kv-resident 32768,--expert-cache auto --prefill auto --spec 4, vision on - Experts: 20,880 of 24,576 (85%) resident in VRAM, decode hit rate 97-99%
Baseline on this box: decode 128-151 tok/s, prefill 4,249-5,154 tok/s at 78K-92K context.
Where the 28.8 GB n-gram table lives
Four arms were run; the only clean comparison (same mode, only the row-cache size) is A vs C. Keeping the whole table in RAM bought just +0.65% warm prefill and +1.2% decode for +28.4 GiB RAM, while the default ~90 MB row cache already absorbs 92% of reusable traffic. --ple-io mmap was 3.7x slower on the first long prompt and silently disables the row cache. The author retracts an earlier "+4.7%" as a wrong baseline, and reports an 11.5% session-to-session spread on unchanged settings, so small deltas here are noise.
Conversation parking: the real win (35x)
--conversation-cache-mib 8192
--conversation-cache-slots 4
Returning to a 91,836-token conversation dropped from 18.8 s to 537 ms for 3.1 GB of RAM.
Other measured settings
--prefill auto:32768: +9.7% prefill / +13.6% decode at a 78.7K prompt (nothing on short prompts)--calibrate: +3.2% by changing one value, pool workers 23 to 12expert_profile_save: verified end to end, no gain at 98-99% hit rates- A 256K ceiling does not tax short chats (a 17-token prompt still decodes at 128-133 tok/s)
- Vision costs ~641 expert slots; a synthetic image was read in 0.97 s
Operating notes
- Sharing the GPU with ComfyUI via
POST /unloadworks: it frees 47.9 GB, ~18-19 s to reload. - On Windows 11, minimizing the engine console drops throughput ~20% (EcoQoS / E-cores).
Not tested
Output quality, AMD hardware, slower storage, and contexts past 262K.
Reported anonymously by an r/LocalLLM contributor · 1 upvotes at capture
Leave a comment