Back to directory
Research & analysis · 2026-10-04

Benchmarking Strata n-gram table placement on a modded RTX 4090 48GB with 128GB DDR5

A measured benchmark on a modded 48GB RTX 4090 shows Strata's n-gram table belongs on SSD while conversation parking delivers the large speedup.

Setup

  • CPU / GPU / RAM: i9-13900K / RTX 4090 48 GB (memory-modded, driver 617.14) / 128 GB DDR5-6000
  • OS / engine: Windows 11, Strata 0.1.38 (sm_89), IQ3_S
  • Config: 262K context, --kv int8 --kv-resident 32768, --expert-cache auto --prefill auto --spec 4, vision on
  • Experts: 20,880 of 24,576 (85%) resident in VRAM, decode hit rate 97-99%

Baseline on this box: decode 128-151 tok/s, prefill 4,249-5,154 tok/s at 78K-92K context.

Where the 28.8 GB n-gram table lives

Four arms were run; the only clean comparison (same mode, only the row-cache size) is A vs C. Keeping the whole table in RAM bought just +0.65% warm prefill and +1.2% decode for +28.4 GiB RAM, while the default ~90 MB row cache already absorbs 92% of reusable traffic. --ple-io mmap was 3.7x slower on the first long prompt and silently disables the row cache. The author retracts an earlier "+4.7%" as a wrong baseline, and reports an 11.5% session-to-session spread on unchanged settings, so small deltas here are noise.

Conversation parking: the real win (35x)

--conversation-cache-mib 8192
--conversation-cache-slots 4

Returning to a 91,836-token conversation dropped from 18.8 s to 537 ms for 3.1 GB of RAM.

Other measured settings

  • --prefill auto:32768: +9.7% prefill / +13.6% decode at a 78.7K prompt (nothing on short prompts)
  • --calibrate: +3.2% by changing one value, pool workers 23 to 12
  • expert_profile_save: verified end to end, no gain at 98-99% hit rates
  • A 256K ceiling does not tax short chats (a 17-token prompt still decodes at 128-133 tok/s)
  • Vision costs ~641 expert slots; a synthetic image was read in 0.97 s

Operating notes

  • Sharing the GPU with ComfyUI via POST /unload works: it frees 47.9 GB, ~18-19 s to reload.
  • On Windows 11, minimizing the engine console drops throughput ~20% (EcoQoS / E-cores).

Not tested

Output quality, AMD hardware, slower storage, and contexts past 262K.

VERIFIABLE SOURCE

Reported anonymously by an r/LocalLLM contributor · 1 upvotes at capture

View source
SETUP HISTORY

Snapshots over time

3 versions
CURRENT2026-10-04Benchmarking Strata n-gram table placement on a modded RTX 4090 48GB with 128GB DDR5

A measured benchmark on a modded 48GB RTX 4090 shows Strata's n-gram table belongs on SSD while conversation parking delivers the large speedup.

Research & analysis1 machineModels unspecifiedStrata
ChangedDescription
Compare with revision 2
DescriptionPreviousOn a memory-modded RTX 4090 48GB with an i9-13900K, 128GB DDR5-6000 and Windows 11, the author benchmarked Strata 0.1.38 (262K context, IQ3_S, --kv int8 --kv-resident 32768, --spec 4, 20,880 of 24,576 experts resident in VRAM, decode hit rate 97-99%) across several settings. On placement of the 28.8GB n-gram table, four arms were run; the only clean comparison (same mode, only the row-cache size) showed that putting the whole table in RAM bought just +0.65% warm prefill and +1.2% decode for +28.4GiB RAM, while the default ~90MB row cache already absorbed 92% of reusable traffic. --ple-io mmap was 3.7x slower on the first long prompt and silently disabled the row cache. The author corrects an earlier '+4.7%' for the RAM-resident table as a wrong baseline, and reports an 11.5% session-to-session spread on unchanged settings, so small deltas here are noise. Conversation parking was the real win: with --conversation-cache-mib 8192 --conversation-cache-slots 4, returning to a 91,836-token conversation dropped from 18.8s to 537ms for 3.1GB of RAM, about 35x. --prefill auto:32768 gave +9.7% prefill and +13.6% decode at a 78.7K prompt (nothing on short prompts), and --calibrate found +3.2% by changing one value, pool workers 23 to 12. expert_profile_save was verified end to end but showed no gain at 98-99% hit rates, and a 256K ceiling does not tax short chats (a 17-token prompt still decodes at 128-133 tok/s). Vision works and is cheap (a synthetic image read in 0.97s, costing ~641 expert slots). Sharing the GPU with ComfyUI works via POST /unload (frees 47.9GB; ~18-19s to reload). Baseline on this box is decode 128-151 tok/s and prefill 4,249-5,154 tok/s at 78K-92K context. Not tested: output quality, AMD, other storage, and contexts past 262K.This snapshot## Setup - **CPU / GPU / RAM**: i9-13900K / RTX 4090 **48 GB** (memory-modded, driver 617.14) / 128 GB DDR5-6000 - **OS / engine**: Windows 11, Strata 0.1.38 (sm_89), IQ3_S - **Config**: 262K context, `--kv int8 --kv-resident 32768`, `--expert-cache auto --prefill auto --spec 4`, vision on - **Experts**: 20,880 of 24,576 (85%) resident in VRAM, decode hit rate 97-99% Baseline on this box: decode **128-151 tok/s**, prefill **4,249-5,154 tok/s** at 78K-92K context. ## Where the 28.8 GB n-gram table lives Four arms were run; the only clean comparison (same mode, only the row-cache size) is A vs C. Keeping the whole table in RAM bought just **+0.65% warm prefill** and **+1.2% decode** for **+28.4 GiB RAM**, while the default ~90 MB row cache already absorbs **92%** of reusable traffic. `--ple-io mmap` was **3.7x slower** on the first long prompt and silently disables the row cache. The author retracts an earlier "+4.7%" as a wrong baseline, and reports an **11.5%** session-to-session spread on unchanged settings, so small deltas here are noise. ## Conversation parking: the real win (35x) ```shell --conversation-cache-mib 8192 --conversation-cache-slots 4 ``` Returning to a 91,836-token conversation dropped from **18.8 s to 537 ms** for 3.1 GB of RAM. ## Other measured settings - `--prefill auto:32768`: **+9.7% prefill / +13.6% decode** at a 78.7K prompt (nothing on short prompts) - `--calibrate`: **+3.2%** by changing one value, pool workers 23 to 12 - `expert_profile_save`: verified end to end, **no gain** at 98-99% hit rates - A 256K ceiling does not tax short chats (a 17-token prompt still decodes at 128-133 tok/s) - Vision costs ~641 expert slots; a synthetic image was read in 0.97 s ## Operating notes - Sharing the GPU with ComfyUI via `POST /unload` works: it frees 47.9 GB, ~18-19 s to reload. - On Windows 11, minimizing the engine console drops throughput ~20% (EcoQoS / E-cores). ## Not tested Output quality, AMD hardware, slower storage, and contexts past 262K.

This is the currently published snapshot.

REVISION 22026-10-04Benchmarking Strata n-gram table placement on a modded RTX 4090 48GB with 128GB DDR5

A measured benchmark on a modded 48GB RTX 4090 shows Strata's n-gram table belongs on SSD while conversation parking delivers the large speedup.

Research & analysis1 machineModels unspecifiedStrata
ChangedDescription
Compare with revision 1
DescriptionPreviousOn a modded RTX 4090 with 48GB VRAM, 128GB DDR5-6000 and a 262K Strata config, holding the 28.8GB n-gram table entirely in RAM gained only +0.65% prefill and +1.2% decode for +28.4 GiB RAM, while the default 90MB row cache already absorbed 92% of reusable traffic. Conversation parking cut the return to a 91,836-token conversation from 18.8s to 537ms for 3.1GB of RAM, about 35x, and preview auto 32768 gave +9.7% prefill and +13.6% decode at a 78.7K prompt. The calibrate pass changed only pool workers from 23 to 12 for +3.2%. The author reports an 11.5% session-to-session spread and did not test output quality.This snapshotOn a memory-modded RTX 4090 48GB with an i9-13900K, 128GB DDR5-6000 and Windows 11, the author benchmarked Strata 0.1.38 (262K context, IQ3_S, --kv int8 --kv-resident 32768, --spec 4, 20,880 of 24,576 experts resident in VRAM, decode hit rate 97-99%) across several settings. On placement of the 28.8GB n-gram table, four arms were run; the only clean comparison (same mode, only the row-cache size) showed that putting the whole table in RAM bought just +0.65% warm prefill and +1.2% decode for +28.4GiB RAM, while the default ~90MB row cache already absorbed 92% of reusable traffic. --ple-io mmap was 3.7x slower on the first long prompt and silently disabled the row cache. The author corrects an earlier '+4.7%' for the RAM-resident table as a wrong baseline, and reports an 11.5% session-to-session spread on unchanged settings, so small deltas here are noise. Conversation parking was the real win: with --conversation-cache-mib 8192 --conversation-cache-slots 4, returning to a 91,836-token conversation dropped from 18.8s to 537ms for 3.1GB of RAM, about 35x. --prefill auto:32768 gave +9.7% prefill and +13.6% decode at a 78.7K prompt (nothing on short prompts), and --calibrate found +3.2% by changing one value, pool workers 23 to 12. expert_profile_save was verified end to end but showed no gain at 98-99% hit rates, and a 256K ceiling does not tax short chats (a 17-token prompt still decodes at 128-133 tok/s). Vision works and is cheap (a synthetic image read in 0.97s, costing ~641 expert slots). Sharing the GPU with ComfyUI works via POST /unload (frees 47.9GB; ~18-19s to reload). Baseline on this box is decode 128-151 tok/s and prefill 4,249-5,154 tok/s at 78K-92K context. Not tested: output quality, AMD, other storage, and contexts past 262K.

Archived published snapshot — kept for context and comparison.

REVISION 12026-10-04Benchmarking Strata n-gram table placement on a modded RTX 4090 48GB with 128GB DDR5

A measured benchmark on a modded 48GB RTX 4090 shows Strata's n-gram table belongs on SSD while conversation parking delivers the large speedup.

Research & analysis1 machineModels unspecifiedStrata

Archived published snapshot — kept for context and comparison.

Discussion

0 comments

No comments yet. Questions and corrections on this setup are welcome.

Leave a comment

Comments are reviewed before they appear. Keep it civil, cite your sources, and stay on topic.