# Prompt caching

Cache the stable prompt prefix on every provider, and know when it hits.

Every iteration of the agent loop resends the system prompt, the tool list, and the conversation so far. Vendors cache that prefix in incompatible ways, so HarnessX splits the work. The engine decides what is stable and where the reusable prefix ends. Each provider translates that into its vendor's mechanism. Caching is on by default and is an optimization, never a correctness dependency.

## Configure it

`AgentConfig.prompt_cache` holds a `PromptCachePolicy`. The default policy is already on; pass `None` to disable caching for every provider.

~~~python
from harnessx import Agent, AgentConfig, PromptCachePolicy

# Default: on, vendor default lifetime, history breakpoint after the last turn.
agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))

# Tuned: a one-hour lifetime and per-tenant separation of identical prompts.
agent = Agent(config=AgentConfig(
    provider="gemini",
    model="gemini-3.6-flash",
    prompt_cache=PromptCachePolicy(ttl_seconds=3600, key_salt="tenant-a"),
))

# Off.
agent = Agent(config=AgentConfig(prompt_cache=None))
~~~

| Field | Default | Meaning |
| --- | --- | --- |
| ttl_seconds | None | Requested cache lifetime; None means the vendor's default |
| cache_history | True | Also mark the last complete turn, so each iteration reuses the conversation before it |
| key_salt | "" | Separates caches for otherwise identical prompts, for example per tenant |

The policy is part of the agent configuration, so a session snapshot or a resumed durable run caches the same way as the original.

## What the engine sends

Before each model call the engine builds a `PromptCacheHint` from the request it is about to send:

- `prefix_key`, a SHA-256 of the model, the system prompt, the tool list, and the salt. Two requests share a key only when that prefix is byte-identical.
- `breakpoints`, the positions where a reusable prefix ends: `system`, `tools`, and `message:<index>` for the last message when `cache_history` is on.
- `ttl_seconds` from the policy.

The hint is persisted with the request in durable execution state, so a resumed run sends the same hint. Providers written before the hint existed are called without it.

## What each provider does with it

| Provider | Translation |
| --- | --- |
| anthropic | Places `cache_control` markers after the system prompt, the last tool, and the last message. `ttl_seconds` of 3600 or more selects the one-hour cache; otherwise the five-minute default applies. Anthropic only caches above a minimum prefix size, and a request it rejects is retried without markers |
| openai | Relies on automatic prefix caching and sets `prompt_cache_key` to the prefix key, so identical prefixes route to the same cache |
| azure, openrouter | Automatic prefix caching only. No routing key is sent, because support for it varies by API version and upstream |
| gemini | Implicit caching by default. When `ttl_seconds` is set, or the `GEMINI_PROMPT_CACHE_TTL` variable or `prompt_cache_ttl` constructor argument is used, the system prompt and tools are uploaded once as an explicit cache and referenced per call. A prefix below Gemini's minimum cacheable size runs uncached, and that prefix is not retried for ten minutes |

A custom `LLMProvider` receives the hint as an optional `cache` keyword on `create` and `stream`. Ignoring it is fine. Honoring it means failing open when the vendor rejects the request.

## See whether it works

Cache activity appears in the usage fields every provider already fills: `RunResult.usage.cache_read_input_tokens` counts prefix tokens served from cache, and `cache_creation_input_tokens` counts tokens written to it on vendors that report writes. Both are included in `input_tokens`, which is the full prompt size on every provider. A cached prompt therefore shows the same `input_tokens` as an uncached one. The cache counters say how much of that prompt came from cache.

Every `LLM_REQUEST` hook carries `data["prefix_key"]`. Consecutive iterations of one run should share it. A key that changes between iterations means the prefix drifted, and no vendor will hit its cache.

~~~python
from harnessx import Agent, HookContext, HookEvent

agent = Agent()
seen = []

def watch(ctx: HookContext) -> None:
    seen.append(ctx.data["prefix_key"])

agent.hooks.on(HookEvent.LLM_REQUEST, watch)
# After a run: assert len(set(seen)) == 1
~~~

## Keep the prefix stable

The harness guarantees the parts it owns: tools are serialized in registration order and the system prompt is assembled the same way on every call. The application owns the rest.

- Do not put timestamps, request ids, or per-call context in the system prompt. Put them in the user message.
- Register tools once, in a fixed order. A tool added mid-run changes the key for the rest of that run.
- Prompt providers that compose the system prompt dynamically must be deterministic for the life of a run.
- Use `key_salt` rather than editing the prompt when identical prompts must not share a cache.

## Costs and limits

- Anthropic bills cache writes above the plain input rate and reads well below it. A single request with no reuse pays slightly more; an agent loop pays less from the second iteration on.
- Gemini explicit caches are server-side resources that expire by TTL. The provider keeps its registry per instance and per process, so two workers create two caches for the same prefix.
- Vendors apply minimum sizes. A short prompt is simply not cached, with no error.
- Nothing here changes what the model sees. Disabling the policy restores the exact requests sent before caching existed.