What I built
An expert cache for llama.cpp, for batch 1 decode only. On RTX 5090 hosts it ran 2.0-4.0× stock llama.cpp and faster than FreeToken in 11 of 12 cells. gpt-oss-120b ran at 58-153 tokens/s with 11-40% of its experts on the GPU. At its smallest cache on 25 consumer machines, a scope I set after seeing the data, the cache reached 31-54% of a limit computed from my own bandwidth probe, not a physical one.
That model has 36 MoE layers of 128 experts at 13.25 MB each, about 61 GB, against 32 GB on the card. Each layer gets GPU slots. A miss goes to CPU helper threads through a mailbox the GPU writes, or a per-machine table copies it over PCIe into a slot. A decayed-frequency rule decides what stays, and an admitted miss is read twice: once by the CPU, once by a background copy.
The limit is the fewest host reads any cache of that size can make (Belady's MIN with bypass, on the routing trace), at the machine's best probed read rate. Oracles in the engine read future routing from a recorded trace, so I could change one thing at a time.
What I measured
First, one machine, teacher-forced so every row sees the same routing.
| State, gpt-oss-120b at 11% of experts on the GPU | ms per token |
|---|---|
| Deployed cache, no oracle | |
| Deployed set, each admission read once | |
| MIN's set, read twice (oracle) | |
| MIN's set, read once (oracle) | |
| MIN's set, read once, copies issued ahead (oracle) | |
| Computed limit |
Then the 12 cells: 2 RTX 5090 hosts, 30 AIME-25 problems, first 256 greedy tokens, equal GPU expert memory.
| Model | Experts on GPU | Stock (tok/s) | FreeToken (tok/s) | Cache (tok/s) | Cache ÷ FreeToken (95% CI) | Cache ÷ stock |
|---|---|---|---|---|---|---|
| Host B: Ryzen 9 9950X3D, card memory 17001 MHz | ||||||
| gpt-oss-120b | 11% | 34.9 | 54.0 | 1.29 [1.28, 1.31] | 2.00 | |
| gpt-oss-120b | 25% | 40.0 | 85.6 | 1.27 [1.25, 1.29] | 2.73 | |
| gpt-oss-120b | 40% | 47.7 | 132.2 | 1.15 [1.13, 1.17] | 3.20 | |
| Qwen3-30B-A3B | 12.5% | 19.5 | 38.7 | 1.03 [1.02, 1.04] | 2.05 | |
| Qwen3-30B-A3B | 25% | 22.1 | 54.7 | 1.15 [1.13, 1.17] | 2.86 | |
| Qwen3-30B-A3B | 43.75% | 28.4 | 103.1 | 1.05 [1.04, 1.06] | 3.81 | |
| Host S: Ryzen 9 9950X, stock clock, card memory 14001 MHz | ||||||
| gpt-oss-120b | 11% | 28.9 | 48.0 | 1.21 [1.19, 1.22] | 2.00 | |
| gpt-oss-120b | 25% | 33.6 | 78.7 | 1.20 [1.16, 1.22] | 2.80 | |
| gpt-oss-120b | 40% | 40.1 | 122.2 | 1.09 [1.08, 1.11] | 3.34 | |
| Qwen3-30B-A3B | 12.5% | 16.1 | 32.7 | 1.03 [1.02, 1.04] | 2.08 | |
| Qwen3-30B-A3B | 25% | 18.5 | 51.4 | 1.05 [1.04, 1.06] | 2.91 | |
| Qwen3-30B-A3B | 43.75% | 23.8 | 98.3 | 0.97 [0.96, 0.99] | 4.02 | |
Beside those ratios:
- The 1.29× is host B, the faster-memory host. The same cell at stock clock is 1.21×.
- FreeToken was faster at one cell, 0.97× at Qwen3-30B-A3B 43.75% on host S. I didn't separate why.
- The difference from FreeToken is mostly admission policy. On host S, LRU in place of my rule ran 6-21% slower, and slower than FreeToken at 4 of 6 cells.
- Host S missed a band I registered before renting it: 5 of 6 ratios fell outside ±0.06 of host B's.
- With FreeToken tuned on a separate rental, the ratios are 0.98-1.17×.
- Stock llama.cpp is a weak baseline.
What limited it
The gap is the deployed cache's time above the limit. I split it with the oracles.
| Mean share of the gap, % (95% interval) | Copy table off (5 machines) | Table on, exploratory panel (15 machines) | Table on, registered test (5 machines) |
|---|---|---|---|
| Measured | |||
| What is cached: MIN's set, read twice | 19 [13, 25] | 0 [-2, 3] | 3 [-1, 8] |
| How it is read: the deployed set, read once | 8 [2, 14] | 0 [-2, 2] | 0 [-2, 1] |
| Both: MIN's set, read once | 36 [13, 58] | 33 [21, 44] | 35 [22, 48] |
| Then the read-ahead oracle | 18 [10, 25] | 15 [11, 19] | 19 [14, 24] |
| Left after all three | 43 [31, 56] | 52 [43, 62] | 46 [38, 55] |
| The copy table itself | 3 [-5, 12] | n/a | n/a |
| Attributed, of what is left | |||
| The GPU's non-expert time | 25 [18, 31] | 27 [25, 30] | 28 [20, 36] |
| The oracle's reads beyond MIN's | 18 [14, 23] | 21 [20, 23] | 18 [15, 20] |
| Residual | 0 [-11, 12] | 4 [-6, 13] | 1 [-3, 4] |
At 11%, with the table on, it puts misses the cache didn't choose into its slots, so MIN's set read twice closes about none of the gap. With it off, that set closes 19%. Either way, 43-52% stays, and 42-48% at 25%.
Policies without foresight recovered at most 6% of an oracle's gain on 5 fast-link machines, against a loose registered bound.
I read this as: closing the rest would take reads issued before routing, at MIN's count.
What it doesn't show
- Anything above batch 1. At 25% on host S, prefill ran 79-83% of stock and decode at 2-4 sequences ran 77-79%, one run per configuration.
- A production part. A GPU wait on a stuck helper aborts the process, after a default of 2 s.
- A physical limit. It's relative to my probe and assumes exact routing, whole experts, per-layer slots and one token per step. A reader faster than my probe could go below it. None of 302 timed runs did, the closest at 1.33× the limit, but a closed-form estimate puts the implied read rate more than 5% above the probe on 12 of 84 launch and cache-size pairs.
- A registered range. The 31-54% is post hoc and covers gpt-oss's smallest cache only, median 47%. At 25%, consumer machines reach 23-36%. My machine list drops 5 unstable rentals, all server processors, so the drop can only touch the server range. The 3 server machines left reach 12-46%, and I haven't established why.
- Other setups. Rented hosts as a convenience sample, clocks unlocked, one engine, two models, AIME prompts also used in development.
- Task accuracy. I didn't measure it.
- A clean prediction record. 142 of 553 scored prediction clauses failed, all in the scorecard.
- Review. This is a public manuscript, not peer reviewed. The patch is not upstream.