In September I set out to make a harness I work on cache-friendly. The big change was freezing the system prompt for the life of a session, so nothing in it could shift mid-conversation.
Then I ran a full agent session against the OpenAI API and logged every request. 91% of input tokens came from cache. By the usual measure, that’s healthy.
Kamran Tahir’s survey is the best overview I’ve found. It covers how prefix caching works down to the KV cache, and what each major harness documents about prefix layout, invalidation, TTLs and compaction.
This post is one measured case.
A hit rate hides a lot, because a request that appends a big tool result is mostly new tokens anyway. So I counted something narrower: tokens the API had already seen, billed again as new.
Most requests re-billed nothing. Four re-billed tens of thousands of tokens each, more than half of everything the session paid full price for.
The first big one was a mode switch. I’d made sure the instructions stayed the same when the agent switched modes. They did. The cache still missed completely.
The tools are part of the prefix
The common advice is to keep the prefix identical across turns. I thought I had. But I’d been treating “the prefix” as the system prompt. To the provider, it’s everything before the conversation, and tool definitions come first.
here's a prompt to improve your agent harness based on what we've learned at cursor. enjoy
— Eric Zakariasson (@ericzakariasson) September 23, 2026
The switch changed two things: it trimmed the tool list from 47 to 42, and it added a one-line note announcing the new mode.
On the next run I dropped the note. Everything else was byte-identical across the switch. Still a full miss. The tool list alone was enough.
Hiding tools per mode felt like the obvious way to scope an agent. It cost the whole cache on every switch.
The fix was to stop hiding them. Every mode now sends the same tool list, and a tool the mode doesn’t allow just refuses when called. The switch went from 0% cached to 91%.
opencode landed on the same rule. Its plan mode keeps the edit tool listed and rejects the edits it doesn’t allow. Its docs put it well: “Tool filtering is catalog visibility, not execution authorization.”
Don’t edit what you’ve already sent
The small spike was the agent switching back. The other two had nothing to do with modes. The tools and prompt were untouched. The history wasn’t.
In one, a tool output the model had seen at 59,651 characters came back on the next turn as 828. The cache served everything before it and nothing after.
The other was the same bug in reverse: a 1,608-character output came back as 51,814, because the stored copy held more than the model was sent. Together they re-billed twice what the mode switch did.
Neither was a caching decision. Both came from code that tidied up history, or filled it in, after the fact. Each version was reasonable. They just weren’t the same version.
This is the whole problem in one picture. The provider’s cache is an append-only log of what you sent. Append, and you pay for the new part. Edit anything already in it, however small or far back, and you pay again for everything after.
The fix is to render each tool output once and replay exactly that.
Codex truncates output once, when it’s recorded; its source notes the truncation “applies only to live history.” opencode dropped its option to prune old outputs and now caps them once, when produced.
Other ways to edit the past
Once I knew the shape, I found more. None showed up in that session. All of them showed up in production.
Tool order that depends on timing. MCP servers connect in parallel, and the harness listed their tools in whatever order the connections finished. In production the order changed 14 times in 3 days, sometimes mid-session.
Same tools, different order, full miss. The fix is to sort. Codex connects in parallel too, then sorts. opencode sorts and waits for the first connections before sending anything.
Replay that tells the story differently. The agent called a tool, read the output, then called another. On replay, the harness merged those into one step: two calls, then two outputs.
Same content, different shape, and every new turn or resume diverged there. The fix was to record which step each part belonged to and replay the steps as they happened.
Instructions that refresh themselves. The harness re-read AGENTS.md and git status every turn, so a commit or a doc edit rewrote the system prompt mid-session.
Freezing them fixed the cache and created a new bug: a guide written after the freeze never reached the model.
Codex and opencode both keep the original and append the change as a new message. Codex’s begins: “These AGENTS.md instructions replace all previously provided AGENTS.md instructions.”
Finding your own
Usage numbers tell you that a request missed, not why. For that you need the request bodies, so the first thing I do in any harness I work on is log them in full.
For every request that cached less than it should, I compare it with the one before. I flatten each body into prefix order (tools, instructions, then each conversation item), hash each part, and find the first that differs.
import { createHash } from "node:crypto";
const h = (v: unknown) =>
createHash("sha1").update(JSON.stringify(v)).digest("hex").slice(0, 8);
// A Responses API request, in prefix order.
function parts(body: any): string[] {
return [
`tools ${h(body.tools ?? [])}`,
`instructions ${h(body.instructions ?? "")}`,
...(body.input ?? []).map(
(item: any, i: number) =>
`input[${i}] ${item.role ?? item.type} ${h(item)}`
),
];
}
function firstDivergence(prev: any, next: any): string {
const a = parts(prev);
const b = parts(next);
const i = a.findIndex((p, k) => p !== b[k]);
return i === -1
? "prefix intact, only appended"
: `${a[i]} -> ${b[i] ?? "missing"}`;
}The answer lands in one of three buckets. If tools or instructions differ, the harness changed its prefix. If an input[i] item differs, usually an old tool output, it rewrote history.
If the prefix is intact and the request still missed, it’s the provider, or the cache expired.
Most of the misses in this post were found with a version of that function. It takes minutes to write. Guessing took me much longer.
Then make it a test. Codex has one that changes approval and sandbox settings mid-session and asserts the next request is the previous one plus appended items. If your harness can’t pass that, your invoice will tell you.
It’s usually you
To check this wasn’t just my harness, I ran the same analysis on several weeks of my own Claude Code and Codex logs, about 59,000 requests. One person’s usage, mostly agent-driven runs, but a consistent shape.
Claude Code cached 97% of input tokens; Codex cached 93%. In both, 3 to 4% of requests carried more than half of what was avoidable. The averages looked fine. The cost sat in a handful of transitions.
The provider does miss. In Codex, about 0.6% of steps within a turn missed with nothing changed and no time passed, and the next request usually hit again. Real, but about one step in 170.
Most of the rest came from the harness’s own choices. In Codex, misses clustered around compaction. In Claude Code, it was time: requests after the cache expired, like a subagent waiting on a slow tool or a session idle over lunch.
OpenAI has since made 12-hour caching available on its API. That helps with time. It does nothing for a harness that rewrites its own prefix: twelve hours of cache can’t match a prefix you no longer send.
You can now choose 12-hour prompt caching in the API.
— OpenAI Developers (@OpenAIDevs) October 20, 2026
The cache doesn’t miss at random. It misses where your harness changed its mind about the past.