Every iteration of the agent loop resends the system prompt, the tool list, and the conversation so far. Vendors cache that prefix in incompatible ways, so HarnessX splits the work. The engine decides what is stable and where the reusable prefix ends. Each provider translates that into its vendor's mechanism. Caching is on by default and is an optimization, never a correctness dependency.

Configure it

AgentConfig.prompt_cache holds a PromptCachePolicy. The default policy is already on; pass None to disable caching for every provider.

from harnessx import Agent, AgentConfig, PromptCachePolicy

# Default: on, vendor default lifetime, history breakpoint after the last turn.
agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))

# Tuned: a one-hour lifetime and per-tenant separation of identical prompts.
agent = Agent(config=AgentConfig(
    provider="gemini",
    model="gemini-3.6-flash",
    prompt_cache=PromptCachePolicy(ttl_seconds=3600, key_salt="tenant-a"),
))

# Off.
agent = Agent(config=AgentConfig(prompt_cache=None))
Field Default Meaning
ttl_seconds None Requested cache lifetime; None means the vendor's default
cache_history True Also mark the last complete turn, so each iteration reuses the conversation before it
key_salt "" Separates caches for otherwise identical prompts, for example per tenant

The policy is part of the agent configuration, so a session snapshot or a resumed durable run caches the same way as the original.

What the engine sends

Before each model call the engine builds a PromptCacheHint from the request it is about to send:

  • prefix_key, a SHA-256 of the model, the system prompt, the tool list, and the salt. Two requests share a key only when that prefix is byte-identical.
  • breakpoints, the positions where a reusable prefix ends: system, tools, and message:<index> for the last message when cache_history is on.
  • ttl_seconds from the policy.

The hint is persisted with the request in durable execution state, so a resumed run sends the same hint. Providers written before the hint existed are called without it.

What each provider does with it

Provider Translation
anthropic Places cache_control markers after the system prompt, the last tool, and the last message. ttl_seconds of 3600 or more selects the one-hour cache; otherwise the five-minute default applies. Anthropic only caches above a minimum prefix size, and a request it rejects is retried without markers
openai Relies on automatic prefix caching and sets prompt_cache_key to the prefix key, so identical prefixes route to the same cache
azure, openrouter Automatic prefix caching only. No routing key is sent, because support for it varies by API version and upstream
gemini Implicit caching by default. When ttl_seconds is set, or the GEMINI_PROMPT_CACHE_TTL variable or prompt_cache_ttl constructor argument is used, the system prompt and tools are uploaded once as an explicit cache and referenced per call. A prefix below Gemini's minimum cacheable size runs uncached, and that prefix is not retried for ten minutes

A custom LLMProvider receives the hint as an optional cache keyword on create and stream. Ignoring it is fine. Honoring it means failing open when the vendor rejects the request.

See whether it works

Cache activity appears in the usage fields every provider already fills: RunResult.usage.cache_read_input_tokens counts prefix tokens served from cache, and cache_creation_input_tokens counts tokens written to it on vendors that report writes. Both are included in input_tokens, which is the full prompt size on every provider. A cached prompt therefore shows the same input_tokens as an uncached one. The cache counters say how much of that prompt came from cache.

Every LLM_REQUEST hook carries data["prefix_key"]. Consecutive iterations of one run should share it. A key that changes between iterations means the prefix drifted, and no vendor will hit its cache.

from harnessx import Agent, HookContext, HookEvent

agent = Agent()
seen = []

def watch(ctx: HookContext) -> None:
    seen.append(ctx.data["prefix_key"])

agent.hooks.on(HookEvent.LLM_REQUEST, watch)
# After a run: assert len(set(seen)) == 1

Keep the prefix stable

The harness guarantees the parts it owns: tools are serialized in registration order and the system prompt is assembled the same way on every call. The application owns the rest.

  • Do not put timestamps, request ids, or per-call context in the system prompt. Put them in the user message.
  • Register tools once, in a fixed order. A tool added mid-run changes the key for the rest of that run.
  • Prompt providers that compose the system prompt dynamically must be deterministic for the life of a run.
  • Use key_salt rather than editing the prompt when identical prompts must not share a cache.

Costs and limits

  • Anthropic bills cache writes above the plain input rate and reads well below it. A single request with no reuse pays slightly more; an agent loop pays less from the second iteration on.
  • Gemini explicit caches are server-side resources that expire by TTL. The provider keeps its registry per instance and per process, so two workers create two caches for the same prefix.
  • Vendors apply minimum sizes. A short prompt is simply not cached, with no error.
  • Nothing here changes what the model sees. Disabling the policy restores the exact requests sent before caching existed.