Prompt caching as a design constraint

Byte-stable prefixes, a date-free system prompt, why tool order matters, and the alarm that fires when a cache write shows up where a read should be.

In short. A provider caches the opening bytes of a request only when they are identical to a recent request’s. Keeping them identical shaped the app: the prompt is frozen when a conversation starts, the date rides on a user message, and the tools array is treated as a contract. When the cache misses anyway, a log line says which part of the prefix moved.

Every reply Token Stanza produces starts by sending the model the whole conversation again. The system prompt, the tool definitions, every message so far, and then the new one. For a long conversation that is tens of thousands of tokens of input per turn, most of it identical to the turn before. That is how these models work. There is no session on the other end. Each request stands alone.

Prompt caching is the provider’s answer. If the opening part of a request is byte-for-byte identical to the opening part of a recent one, the provider reads it from a cache instead of processing it again. It charges a fraction of the input price for the part it read. The word that matters in that sentence is byte-for-byte. One changed character anywhere in the prefix, and everything from that point on is processed and billed in full.

That turns caching from a billing feature into a design constraint. This post is about what it did to the shape of the app.

What the prefix is

The cached prefix is three things in a fixed order: the system text, the tools array, and the message history through the newest user message. One code path produces it, for the chat turn and for every other request that needs the same prefix. Each provider’s renderer turns that neutral prompt into its own request format. Bedrock takes the Converse API’s shape, and the Anthropic API and Vertex take the Messages API’s. None of the renderers adds or reorders anything.

The Prompt panel listing the system prompt's sections (built-in, custom prompt, memory snapshot, conversation tools) with a token estimate each, and a line saying it matches what the latest turn sent
The View prompt panel shows the prefix in the order it is sent: the system sections, then the tools. The "matches" line compares it against a fingerprint of what the last turn really sent.

There is a cache point after each system block and one after the newest user message, each asking for the one-hour TTL. In steady state, every turn reads the previous turn’s prefix and writes only what the last exchange added. A read also refreshes the TTL. That is the fact the keep-warm pings in the next post rest on, and it is what the countdown in the composer is counting.

The composer bar with its controls, a context indicator reading 35% and a cache indicator reading cached 56m
The cache countdown under the composer. It restarts at an hour every time a turn or a ping reads the prefix.

Conversations started on the current prompt layout send the system prompt as two blocks. The first is shared by every conversation on a provider: the built-in instructions and a line about web citations. The second is specific to the conversation: the custom prompt, the memory index, the recent conversations, and the file-library sentence. The split means a brand-new conversation’s first turn can read the tools and the shared block from an entry some other conversation wrote. A turn that loses its history entry, because a message was edited or a reply retried and the history past that point is new, rereads both blocks instead of rewriting them.

I did not take the split on faith. I needed to check three assumptions about how Bedrock’s caching behaves, so a small test program sends a fixed sequence of requests to live Bedrock and reports pass or fail on each.

The three facts the probe confirmed
  • The 1,024-token minimum for a cache entry applies to everything before a cache point, not only to the segment since the previous one. A short conversation block still caches.
  • A request can read from both system entries, or from the shared one alone, while they are warm.
  • A read at the message cache point also refreshes the system entries behind it.

The documentation leaves room to doubt each of these, so I tested rather than inferred. A run costs a few cents and removes a lot of guesswork.

What keeps it byte-stable

The hard part is not the first turn. It is making sure that nothing between turns changes the bytes. Here is what had to change to keep the prefix stable, and what each change cost.

The conversation-start snapshot. The system prompt includes things that change: the user’s memory index, their five most recent conversations, their custom system prompt, and whether they have files in their library. If the prompt read those live, a memory write in another tab would invalidate this conversation’s cache. So on the first turn the app gathers all of it, stores it on the first message, and the system prompt embeds that text verbatim for the conversation’s life. The cost is that a conversation sees memory and settings as they were when it started. The memory tool reads live, so the model can always look at the current state. The prompt just does not restate it.

A date-free system prompt. The obvious place to tell the model today’s date is the system prompt. A dated system prompt invalidates every cached prefix at midnight, which among other things defeats an overnight snooze. Instead the date rides in a small block on a user message, added only when the date differs from the previous user message’s. It is computed from the stored send time and the time zone the conversation snapshotted, so every replay reproduces it exactly. The cost is a date on a user message instead of in the system prompt, which is an odd place for it.

Tool registration order. The tools array is part of the prefix, and its byte order is the dependency-injection registration order in one file. Which tools are offered depends only on facts that are fixed per deployment or frozen at the conversation’s start. Gating a tool on something that moves (“the user has other conversations”, say) would flip the prefix at a boundary nobody can see. There is a longer post coming on the tools themselves. The short version is that the tools array is treated as a contract. The cost is tools offered when they are not strictly needed. A user with one conversation is still offered the conversation-lookup tools.

Stable attachment blocks. Attachments precede the text and are rebuilt identically every turn. A document’s name carries a sequence-and-index suffix rather than anything freshly generated. The cost is that the model sees report.pdf 12-0 rather than report.pdf.

Thinking is part of the key. Changing the extended-thinking level between turns invalidates the message-level cache by construction, so the app does not pretend otherwise. No replay ever sends a thinking configuration. The cost is that changing the thinking level mid-conversation buys a full write on the history, and the app lets it.

The thing I would underline for anyone building something similar: none of these is a trick. Each one is the removal of a source of change from the prefix, and each one has a cost that had to be accepted.

Changes that invalidate everyone’s cache

Some changes rewrite the prefix of every open conversation at once, and every user pays a full write on their next turn:

  • editing the built-in system prompt or any fixed instruction sentence
  • editing a tool’s name, description or schema, or reordering the registrations
  • setting or clearing the web-search key, which adds or removes two tools and a system sentence

None of these is forbidden. Each has to be deliberate, and the repository’s conventions file says so in as many words.

Some changes are necessary despite the cost. If a tool’s description is confusing the model, or an instruction needs to be clearer, leaving it alone is not an option. The way out is a new prompt layout version. Conversations carry the version they started on, the old wording is kept as a second constant, and only new conversations see the new text. That is how a round of tool rewording shipped without touching any existing conversation’s prefix. The layout versions came after the per-request fingerprints described under “Diagnosing a miss”, and in that order for a reason. It was only once a log line could say which part of the prefix had moved that it became obvious how often a harmless-looking edit to a tool description was the part.

The replays

Once the prefix is stable, it becomes a resource. Four other requests reuse it. The conversation title after the first exchange, capped at 64 output tokens. The lookup summary after 45 quiet minutes, a short summary the model’s conversation-lookup tools read. The keep-warm ping, capped at 8. And the handoff summary a user asks for when a conversation is getting long, a structured summary meant to be pasted into a new conversation. Each reproduces the finished turn’s prefix and appends one instruction after the cache point. So it costs only a cache read plus a few hundred tokens instead of full-price input. At list prices, a title on a 20,000-token conversation reads the prefix for well under a cent. Sending that history at full price would be about six cents.

That only works if the replay’s prefix really is the turn’s prefix. A test pins it. The replay reproduces the chat request block for block. Thinking is present on the turn and absent on the replay. The guardrail wrapper, which is on the turn, never rides a replay. That last point is subtle enough to get a post of its own later in this series.

Diagnosing a miss

When a cache read comes back smaller than expected, the provider tells you only that the prefix moved, never where. I learned that the expensive way. In the first weeks after launch the alarm that fired was not anything in the app. It was an AWS budget alert, because the daily spend was running well above what the token counts said it should. The provider’s usage numbers showed cache writes where reads should have been, on conversations nobody had touched in between. Something in the prefix was moving between turns, and nothing I had would say what.

So the assembler now hashes the system text, the tools (names, order and full definitions) and the messages separately. It logs the three per request and stores them on the reply.

What the fingerprint hashes
Prefix fingerprint
  system hash    built-in instructions, custom prompt, memory snapshot, file sentence
  tools hash     names, order and full definitions
  history hash   every message through the newest user message

On the next turn a reporter compares them. That was the tool built to answer the budget alarm. It went in on September 6, and so did the fix it pointed at: the prompt snapshot from earlier in this post, and the summary suppression described below. The ping’s write-dominated check was already there. The system prompt had been reading the memory index and the file library live. Every edit to either, in any tab, had been silently rewriting the prefix of every open conversation. The system hash was the one that moved, and the snapshot is why it no longer does.

A changed system, tools or history hash logs as a warning: this side provably sent a different prefix, which is the bug the line exists to find. A thinking-level change or a conversation from before snapshots existed logs at information, because those misses are expected. All three hashes matching across a miss means the assembly was stable and the request was served by a region that did not hold the entry. The cache is per region, and Bedrock’s cross-region inference profiles can send consecutive requests from the same conversation to different regions (the retirement post has why Sonnet 4.5 is only reachable that way). That case is not a bug in the app, and the log line says so.

The write-dominated alarm

The ping and the summary are the clearest signal, because they are supposed to be pure reads. If one of them comes back with more cache-write tokens than cache-read tokens, something is wrong. Either the prefix changed under it, or the request landed in a region that had never seen it. The app calls that write-dominated. It logs a warning with both token counts and how long it had been since the prefix was last warmed, and then does something about it.

A write-dominated ping cancels the snooze. The whole point of the ping was to avoid a full-price write. If it just paid one, paying another every fifty minutes until the snooze expires would be the worst outcome available. Auto-snooze also skips one re-arm, so the user’s next message does not immediately buy another.

A write-dominated summary suppresses summaries for that conversation for a day. Without that, the cost would recur forever: the summary marker is re-armed on every turn, so a conversation whose prefix had gone unstable would pay a full write once per quiet period indefinitely. A day is long enough to outlive the gap between two armings and short enough that a false positive costs little.

For scale, here is what the Usage page says for the thirty days ending October 25. Cache reads were 99% of all input tokens, and they outnumbered cache writes by well over a hundred to one. The estimated bill for the month was about $170. Had caching not existed and the same tokens been sent at the uncached price, the input alone would have been over $1,100. Even at a tenth of the price, the cache reads were still the largest line on the bill, more than output, cache writes and uncached input put together. That is the honest shape of a chat app: the history is most of every request, and a tenth of the price on most of every request is where the money is. For an app that charges only for what it uses, that ratio is not an optimisation. It is the premise.

The Usage page with its summary tiles, including Cache read and Cache write token counts, and the per-day token table
The Usage page. The Cache read and Cache write tiles are the ratio to watch, and the table has them per day and per kind of request.

What it cost

I will not pretend this was free. The snapshot machinery, the layout versions, the fingerprints, the probe, and the tests that pin all of it are a meaningful fraction of the chat path. A test fails if a single byte of the system prompt changes, and that test is there on purpose.

What it bought is that a long conversation costs about what its newest message costs. An idle conversation can be kept warm overnight for pennies. When the cache does miss, the log says whether it was my fault. For an app whose whole premise is paying for exactly what you use, that seemed worth the complexity.