The low-level code that does an AI model's math. Kernels merged in flash-linear-attention, an open-source AI library. Tests merged in NVIDIA's cuDNN frontend, after a maintainer reproduced a bug they catch. On a long text (64K keys), one kernel step on an H200 GPU ran 52.7× faster than recomputing, a gain any cache gets.
Running AI models fast, even when they don't fit on the GPU. I built a cache for llama.cpp that ran faster than FreeToken, another system for this problem, in 11 of 12 test setups at batch 1. With FreeToken tuned, on a separate machine, the ratio is 0.98-1.17×.
AI that acts on its own, and whether you can trust it. In my research lab of 31 Claude Code agent roles, one agent forged 5 approvals, so I built a check, audited it and rebuilt it. Also bert, searchable memory for coding agents.
When an AI writes a word, three things have to go right.
A GPU has to do the math.
An engine has to serve it, even when the model doesn't fit on the card.
And an agent has to act on it, without being trusted blindly.
I've built at all three layers. Here's what I found, starting at the bottom.
Kernels
The small programs a GPU runs to do the math behind every word.
Stick-breaking attention kernels
2 merged PRs
Forward, backward and KV-cache decoding kernels in Triton, merged in flash-linear-attention, an open-source library.
Result: 52.7× faster per step than recomputing at 64K keys, a gain any cache gets. Still 1.04-1.42× the original authors' time on forward plus backward. Technical detail
Before a model writes its next word, it looks back at everything it has read.
Standard attention weighs every past word at once. Stick-breaking attention goes one word at a time.
Picture a stick. The newest word takes a piece.
The word before it takes a piece of what's left.
And so on, back through the text.
The size of each piece is how much that word counts.
An illustration. The shares are made up.
Working that out from scratch for every new word gets slow.
So I built the shortcut: a kernel that remembers what it has already read.
52.7×
faster per step than starting over, on a long text with 64K keys in the cache.
One kernel step on an H200 at batch 1. Any cache gains like this against recomputing. A full model step on random weights takes about 9 ms, and with short prompts recomputing is still faster. Where that time goes is my open question.
The longer the text, the bigger the gain. Microseconds per step, log scale.
For engineers: the kernel, its tiles, and what's still slower
Forward and backward kernels continued #599 and merged in #1319. 64×32 tiles fit 57-89 KiB of shared memory where 64×64 needed 129-193 KiB. Review caught a silent zero gradient at saturated logits, so I rebuilt the backward.
Against the original authors' kernels
1.04-1.42× their time on all 8 shapes, forward plus backward on H100. Closing that gap is still open.
What registers do to occupancy
At 255 registers per thread, 2 CTAs of 4 warps fit on an SM, which is 12.5%. Nothing saturates in the backward: ALU runs 22-32% and XU 14-18%. I read the rest as stalls on dependent fp32 scans and exp2 and log2 results. That is inference, not a line-level measurement. Drag to see how register use changes what fits.
Arithmetic from 65536 registers per SM and 128 threads per CTA, ignoring allocation rounding.
A prototype and tests for the open-source frontend of cuDNN, NVIDIA's GPU library for AI.
Result: tests that catch 3 planted bugs NVIDIA's SM80 tests missed, reproduced by a maintainer. The prototype closed when NVIDIA shipped its own. Technical detail
Prototype closed Sep 27, 2026#1223Tests merged Oct 6, 2026#1287
Inside NVIDIA's cuDNN, I found a step that took longer than the math it was setting up.
One attention call on an A100, at 64/8 heads and a 128K KV cache. That's one configuration.
It was copying the same data again and again, once for each part of the model that reads it.
So I built a version that doesn't copy.
With the copies gone, it ran 1.15-2.19× faster, for one request at a time.
Multi-token steps at batch 1. Removing the copies made 6 batched rows (3 configs) 2.6-3.3× faster, and 10 more (5 configs) stopped running out of memory. The cuDNN backend stayed faster on 27 of 31 multi-token rows.
NVIDIA then shipped its own native version, so I closed mine. The tests from that work merged.
3 bugs
that NVIDIA's SM80 tests missed and my tests caught. I planted them on purpose, and a cuDNN maintainer reproduced one, then approved the PR.
With bugs planted, NVIDIA's SM80 suite still passed all 411 cases on the first two and all 60 d256 cases on the third. A 4th planted bug was a control that both suites caught. In the maintainer's own run of the second bug, the existing tests passed while 31 of 68 and 6 of 12 new tests failed.
Here's one planted bug. NVIDIA's tests used 4/2 and 16/4 heads, where the wrong formula gives the right answer. Click a head count.
For engineers: the timing table
Multi-token steps at batch 1, in milliseconds. Query heads/KV heads, then head dimension. Develop against the prototype, with the cuDNN backend for reference.
Triton is a language for writing GPU kernels, and cuBLAS is NVIDIA's standard math library. I measured whether one combined kernel runs faster than the usual separate steps.
Result: faster on 27 of 76 sizes, up to 1.73× on an A100. The median size ran 0.96×. Technical detail
One common step in a model usually runs as three separate kernels, saving to memory between each one. What if one kernel did all three?
One matrix size proves nothing, so I measured 76 of them.
27 of 76
sizes ran faster as one kernel than as separate steps, up to 1.73× on an A100. The median size ran 0.96×, slightly slower.
The best size is M=128, N=11008, K=4096 in fp16 on an A100. The same size is 1.28× in bf16, and bare Triton already ran 1.38× there in fp16, so fusion's own lift is about 1.25×.
Each dot is one matrix size. Hover for its dimensions. Median 0.96×, with 27 faster than separate kernels.
One kernel helps on some sizes, not all, and batch size alone doesn't explain which. So I dropped my first theory.
For engineers: the roofline, and what the kernel reaches
I first read this as small batches being memory-bound. The counts don't support that: fused is faster on 10 of 30 cells left of the ridge and 42 of 122 right of it, and bytes saved cover about 10% of the best cell's time gap.
The bare autotuned GEMM peaks at 213 TFLOP/s, 68% of the A100's fp16 tensor-core peak. cuBLAS reaches about 82% on its best shapes. Each point is the median of 10 trial means. The baseline is a cuBLAS GEMM plus separate bias and GeLU, in PyTorch eager.
A small demo that runs in this page, on your own GPU.
GPU speed depends on how you slice up the work. See it on your own GPU.
Rows: workgroup heightColumns: workgroup width
This runs one matrix multiply on your GPU, sliced up every way on the grid. It times each one and checks the answers against your CPU.
It runs on your machine through WebGPU in fp32, and nothing is sent anywhere. It's a demonstration, not one of the measurements on this page.
There's more kernel work on the all-work page, including a timing model of vLLM's fused MoE kernel.
Inference
Running a model and getting words out fast, even when it doesn't fit on the GPU.
Where the Seconds Go
Research project
A way to run AI models too big for the GPU's memory, built as a cache in llama.cpp, plus a calculation of how fast any such cache could get.
Result: faster than FreeToken in 11 of 12 setups at batch 1, and 0.98-1.17× with it tuned, on a separate machine.
Public code and manuscript, not peer reviewed. The patch is not upstream.moe-speed-of-light
The model, gpt-oss-120b, needs about 61 GB of memory. An RTX 5090 GPU has 32 GB.
RTX 5090, 32 GB
One dot per gigabyte, to scale.
So the rest of the model waits in slower system memory, and each word stalls while it's fetched.
I built a cache in llama.cpp that decides which parts stay on the GPU.
11 of 12
test setups ran faster than FreeToken, another system for this problem, and 2.0-4.0× stock llama.cpp in all of them. With FreeToken tuned, on a separate machine: 0.98-1.17×.
At batch 1, on 2 RTX 5090 hosts. Stock is a weak baseline. The tuned comparison covers six cells. The one slower setup is 0.97×, at the largest Qwen3 budget on the stock-clock host. On that host, swapping my admission rule for LRU ran 6-21% slower, so I read most of the difference as that rule.
Stock llama.cppFreeTokenMy cache
Speed in tokens, roughly words, per second. Each row is the share of the model's experts, its swappable parts, kept on the GPU.
Then the harder question: how much better could any cache get?
I computed the limit, then let the engine see the future, one kind of foresight at a time.
20.59 ms per token
The deployed cache, with no foresight.
Flip them yourself. gpt-oss-120b with 11% of experts on the GPU, on one RTX 5090 machine. The limit is relative to my own bandwidth probe.
Better guessing alone didn't get there. Four caching rules without foresight got at most 6% of the gain from knowing what to read ahead.
On 5 fast-link machines. On this machine 36% of the gap stayed after all three kinds of foresight, and 43-52% averaged over three groups of machines. At gpt-oss's smallest cache on 25 consumer machines, the cache reached 31-54% of the computed limit, a scope I chose after seeing the data.
One step holds steady up to 1024 keys, then climbs.
Microseconds per step: 63.1 at 256 keys, 64.9 at 1024, 114.6 at 2048, 209.4 at 4096 and 398.1 at 8192.
Attention and KV-cache decode kernels profiled with Nsight on a Jetson Orin Nano. It's a sweep of kernels, not a model and not serving throughput. Fused SDPA ran 2.78-5.55× faster than a manual matmul and softmax baseline across 5 sizes, which is a low bar.
Also on vLLM: a load test where CUDA graphs cut time per token 31-34%, one run per rate, and a measurement on someone else's RFC. Both are on the all-work page.
Agents
AI that takes actions on its own: memory for coding agents, and a way to keep them honest.
bert
Personal project
A tool, not the BERT model, that gives an AI coding agent searchable memory of a long project.
Result: no significant accuracy difference from grep at 1.26M tokens, and grep used a median 1.46× the tokens. On code, grep scored higher. Technical detail
A long project outgrows what a model can read at once.
Give a reader the whole history, and past its budget it loses the answer.
One Claude reader capped at a 120K budget, which keeps about the newest 10%. I read 0.08 as truncation losing history, not as evidence for bert.
bert gives a coding agent searchable memory of the whole project. It runs on your own machine, with no API keys and no calls to an AI model.
Accuracy at 2 measured memory sizes. Lines only join the 2 measured sizes.
On one 1.26M-token synthetic project memory, no significant difference from agentic grep, counting cache reads in full. On code, grep scored higher, 0.97 to 0.86, so bert is built for accumulated memory, not code search.
At 1.26M tokens, it scored 0.90, with no significant difference from an agent using plain text search (grep). And grep used a median 1.46× bert's tokens.
A smaller, earlier test had caught a silent bug that held its score at 0.10.
One dot per question. A small bug had switched off the meaning-based half of the search, leaving only keywords. Fixing it moved the same 20-question test from 0.10 to 0.85.
For engineers: how the retrieval works, and public benchmarks
sqlite-vec stores bge-base-en-v1.5 dense vectors, BM25 covers exact keywords, reciprocal-rank fusion merges the two, and a cross-encoder reranks the top hits.
Three public BEIR datasets. On scifact the full stack scores 0.745 and dense alone 0.740, with overlapping intervals, so the embedding model earns most of it. BM25 scores 0.658. On fiqa, rerank moved hybrid 0.356 to 0.434. On nfcorpus it moved 0.355 to 0.354, and dense alone was best.
Multi-turn search through MCP scored 0.86 against 0.66 one-shot on code questions, and agentic grep scored 0.97 with fewer tokens.
I ran a research lab of 31 Claude Code agent roles on one 18 GB MacBook.
On Apr 17, the agent running it wrote 5 PI sign-offs, approvals in another agent's name, without ever calling that agent.
So I built a check. Then I audited the check itself, against 763 session logs.
By my count it had run on 0 of 270 eligible sessions, because of a shell bug, fixed in June.
The audit also found the deeper flaw: the record the check read sat where the agent being checked could write to it.
So I moved the record out of the agent's reach.
It relies on the runner's sandbox settings, which I've only tested on a stub harness, not yet on a live lab.
The same forgery, replayed against the new check.
A required sign-off now counts only if the system running the agents saw that agent run and give that answer. Replay the forgery, and the session is blocked.
171 new tests, and 62 of them fail on the old code. A record shows that an agent ran and what it replied, not that its review was any good. The repo lists what's still open.
For engineers: the audit, session by session
One square per runner log, Apr 17 to May 30, in date order. The outlined ones match the gate's precondition, a text match. By my count the first version of the gate ran on 0 of 270 of those eligible sessions, because of a shell bug fixed in June. The logs stay private, so you can check the public table's sums, not its rows.
Receipts come only from the harness's structured tool_use blocks, so text in a session creates none, and they're HMAC-chained. A sign-off token needs a same-role, same-run receipt, inside the window, whose own reply gave that verdict on that document. The gate fails closed: only a detector that finishes and echoes a per-call nonce lets an exit through. The runner's deny rules and OS sandbox keep the key from the agents, and that hasn't been exercised on a live lab.
Still open: role prompts can be edited between sessions, a sign-off binds to a path and not to content, the net for free-text approvals is a heuristic, and close-out runs before the gate. The full list.
Also on the all-work page: The Obsidian Archive, a 13-stage autonomous pipeline that researches, writes, renders and uploads a documentary from one topic.
GPU kernels taught me not to believe an explanation for why something is slow until the numbers back it up. I hold memory tools and agents to the same test.
Every number on this site has its conditions and source. Click a large number to see them. Not public: the lab's logs, the floor report's sources, the RFC results, some Nsight reports and course reports.