Retries & fallback models

Configure RetryPolicy, add fallback models with AgentConfig.fallbacks, and observe model and tool retries.

Configure how an agent handles temporary model failures, when it tries another provider, and how your application observes retries.

Failover is configuration: name the primary in AgentConfig.provider and the providers to try after it in AgentConfig.fallbacks. A Fallback can select a different model for one entry in the chain; there is no FallbackModel class in this API. FallbackProvider covers the case configuration cannot express, covered in When a provider must be a live object.

Add a fallback model or provider

AgentConfig.fallbacks lists the providers to try after the primary, in order. Each Fallback names a provider and may override model and max_tokens for that entry alone. An omitted override reuses the request's value; a different vendor does not automatically get a compatible model ID.

The example below uses Anthropic first and OpenAI as the backup. Install the OpenAI extra with pip install "harnessx[openai]". Export ANTHROPIC_API_KEY, OPENAI_API_KEY, PRIMARY_MODEL, and FALLBACK_MODEL; use model IDs enabled for those accounts.

import asyncio
import os

from harnessx import Agent, AgentConfig, Fallback, RetryPolicy


async def main():
    config = AgentConfig(
        model=os.environ["PRIMARY_MODEL"],
        provider="anthropic",
        max_tokens=4096,
        fallbacks=[Fallback("openai", model=os.environ["FALLBACK_MODEL"], max_tokens=4096)],
        retry=RetryPolicy(attempts=2, max_backoff_seconds=10, cooldown_seconds=60),
    )
    async with Agent(config=config) as agent:
        result = await agent.run("Explain database indexes in three sentences.")
        result.raise_for_status()
        print(result.output)


if __name__ == "__main__":
    asyncio.run(main())

The agent builds the chain and closes it, so there is nothing extra to clean up.

For another model on the same provider, name that provider again with a different Fallback(..., model=...). For a different vendor, supply that vendor's model ID and dependency/credentials.

Setting	Default	Behavior
`Fallback.provider`	required	Built-in or registered provider name. Configuration accepts names only.
`Fallback.model`	`None`	Model ID override for this entry.
`Fallback.max_tokens`	`None`	Positive reply-budget override for this entry.
`RetryPolicy.switch_after`	`1`	Transient failures one provider accumulates before the next is tried within the same call.
`RetryPolicy.cooldown_seconds`	`0`	How long a provider that was passed over is skipped.

Strikes persist across calls and reset when that provider succeeds. Every call starts with the first eligible provider. If all are cooling down, the chain tries them all rather than returning without a request.

Because the chain is configuration, it serializes with the rest of AgentConfig. Durable runs and Temporal workers carry failover like any other setting, and a restored session rebuilds the same chain. In-memory strikes and cooldown timers are runtime state and are not persisted.

Do not also pass a provider object: Agent(config=AgentConfig(fallbacks=[...]), provider=...) raises ConfigurationError. See When a provider must be a live object.

Model limits per member

Each member has its own model limits. A cap learned from one member's refusal applies to that provider and model only, and a member's cap and any repaired budget survive snapshots. A Fallback.max_tokens cannot raise a member's budget above a cap that member has reported.

A conversation that changes vendor

A fallback can move a conversation from one vendor to another mid-session. When it reaches Anthropic, thinking from other vendors carries no Anthropic signature and is left out of the request, as are private keys such as Gemini's thought signature. Native signed thinking, redacted thinking and server tool blocks keep their order and metadata in snapshots and recorder payloads.

A pending native continuation, such as a paused Anthropic turn, stays on the member that started it. Transcript repair and task reminders do not alter it, and request middleware must not change it.

Configure model retries

AgentConfig.retry controls the engine's attempts for one model step. The default classification retries rate limits, server errors, timeouts, and connection failures, including the vendor SDKs' own APIConnectionError. A connection dropped part way through a reply (RemoteProtocolError, ReadError and the other httpx transport errors) is retried too. Other HTTP 4xx errors, including bad requests and authentication errors, stop immediately. Quota and billing errors stop too, even when they arrive as a 429. A forward proxy that refuses the tunnel with a 4xx also stops immediately. An egress proxy denying a destination on policy grounds has made a decision, not hit a temporary fault. Retrying it only spends the budget to be refused again.

from harnessx import AgentConfig, RetryPolicy

config = AgentConfig(retry=RetryPolicy(
    attempts=3,
    backoff_seconds=0.5,
    max_backoff_seconds=10,
    call_timeout_seconds=60,
))

Field	Default	Meaning
`attempts`	`4`	Total engine attempts for one model step, including the initial call. `1` disables engine retries.
`rate_limit_attempts`	`8`	How many tries a rate limit (429) or an overload (529) gets in total. It can't be lower than `attempts`.
`rate_limit_budget_seconds`	`120.0`	The most time, in seconds, a step spends waiting on those extra tries. `None` means no limit.
`backoff_seconds`	`0.5`	Initial retry delay, doubled after each failed attempt.
`max_backoff_seconds`	`60.0`	Maximum delay for one wait, including a server's `Retry-After`. Must be positive.
`call_timeout_seconds`	`None`	Timeout for one engine model attempt. When unset, derived from the reply budget with a minimum of 300 seconds. An explicit positive value overrides it.
`stream_idle_timeout_seconds`	`180.0`	Longest silence allowed between two chunks of a streamed reply. A stream that stalls past it is abandoned and the call retried. `None` turns the check off.
`stream_first_event_timeout_seconds`	`20.0`	Longest wait for the first event of a reply, on providers that send one as soon as the model starts (Anthropic). Raised by 10 seconds per 100K tokens of request, and never longer than `stream_idle_timeout_seconds`. Other providers wait the idle limit for the first event too. `None` turns the check off.
`stream_first_event_growth`	`2.0`	How much longer each retry waits for the first event. By default that's 20, 40, 80, then 160 seconds, up to the idle limit. Use `1.0` to wait the same time on every try.
`switch_after`	`1`	Transient failures one provider accumulates before the chain moves to the next, within the same call. Must be a positive integer.
`cooldown_seconds`	`0.0`	How long a provider that was passed over is skipped. Zero makes it eligible again on the next call.
`jitter`	`0.25`	How much of the backoff is randomly shaved off, so agents that failed together don't all retry at the same moment. `0` makes waits exact.

After a failed attempt, the engine waits an exponential backoff, minus a random slice of up to jitter. If the server sent Retry-After, the wait is at least that long: ask for three seconds and you get at least three. No single wait goes past max_backoff_seconds. Retry-After can be a number of seconds or an HTTP date, and a retry_after field in the error body works too.

from harnessx import RetryPolicy

policy = RetryPolicy(backoff_seconds=1, max_backoff_seconds=5, jitter=0)
assert policy.wait_for(1) == 1
assert policy.wait_for(3) == 4
assert policy.wait_for(1, retry_after=60) == 5

Rate limits

Rate limits usually clear up within a minute, so they get more tries than other errors. By default a 429 or 529 is tried up to 8 times, as long as the waiting adds up to two minutes or less. Server errors and dropped connections still stop after attempts.

A 429 about quota or billing won't go away by waiting, so it fails straight away. If you write your own provider, override LLMProvider.is_rate_limited(exc) to tell the engine which of your errors are rate limits.

While the agent sits out a long Retry-After, the run stream sends WAITING events with {"on": "retry", "seconds": ...}. A UI can show that it's waiting, so it doesn't look stuck. See Streaming.

To give rate limits the same budget as everything else:

from harnessx import AgentConfig, RetryPolicy

# Four attempts for every failure, rate limits included.
config = AgentConfig(retry=RetryPolicy(rate_limit_attempts=4))

The attempt timeout applies to one call through a fallback chain as a whole. It is not a total run deadline. Iteration and cost budgets remain in Limits.

How retry and failover interact

With switch_after=1 and two providers:

The engine calls the chain.
A transient primary failure makes the chain try the backup immediately.
If the backup also fails transiently, the error returns to the engine.
The engine waits and starts another attempt if RetryPolicy.attempts allows it.

Try it with no API key: uv run python examples/05-control/retries_and_fallback.py.

attempts counts calls to the chain, not individual vendor requests. Two engine attempts across two providers can make up to four vendor requests. attempts=1 still permits failover within that single chain call. With switch_after=2, the first primary failure returns to the engine; the primary needs a second transient failure before the backup becomes eligible through that threshold.

A deterministic failure stops the chain immediately under the default classifier. Failed member calls are not separate RETRY hook events: that hook describes the engine scheduling another attempt.

Streaming behavior

The fallback chain can switch providers only before its first chunk reaches the caller. Once a provider has emitted a chunk, a later exception returns to the engine. If the engine retries, it emits ATTEMPT_RESET; discard the provisional text from that attempt.

Use stream_text(..., on_reset=...) for a text UI, or handle RunEventType.ATTEMPT_RESET in the event stream. See Streaming. The final RunResult remains authoritative.

Streams that stall or end early

What arrives from a broken stream can look like a whole reply: an answer missing its end, or a tool call missing arguments. So the harness never uses it.

A stream that never starts. Anthropic sends its first event (message_start) within a second or two, even when the model is about to think for a while. If nothing arrives within RetryPolicy.stream_first_event_timeout_seconds (20 seconds, plus 10 for every 100K tokens of request), the stream is dropped and retried. Each retry waits twice as long, so a server that takes 25 seconds gets through on the second try. From then on the agent knows the server is slow and gives later calls enough time up front. OpenAI's and Gemini's reasoning models stay silent until they've finished thinking, so on those providers the first event gets the idle limit instead.
A stall. No chunk for RetryPolicy.stream_idle_timeout_seconds (180 by default) abandons the stream. Streamed tool arguments count as activity. Anthropic's provider asks for them as they are written (eager_input_streaming), so a long write_file keeps the stream alive. Gemini and some OpenAI-compatible models send a whole tool call at once, so writing a large file there can be quiet for tens of seconds; the idle limit allows for that.
An early end. An Anthropic stream with no stop reason, or an OpenAI or Gemini stream with no finish reason, raises IncompleteStreamError. This includes the stream Anthropic's provider opens itself for a long non-streaming call.
Broken tool arguments. If a tool call's arguments arrive garbled, or stop halfway, the tool doesn't run. The model gets an error saying the JSON was incomplete and sends the call again. That doesn't use up a retry. When a reply stops because it ran out of tokens, truncation recovery handles it instead.

These deadlines bound agent.run() as well as run_stream(): on Anthropic, run() uses the streaming call internally and returns the same result. Other providers keep their non-streaming call for run(), bounded by call_timeout_seconds.

A stall and an early end are retried like a dropped connection, under the same attempts. IncompleteStreamError is importable from harnessx.errors and is a ConnectionError.

Some OpenAI-compatible servers never send a finish reason for plain text. Allow that explicitly on the provider:

from harnessx.providers import OpenAIProvider

provider = OpenAIProvider(allow_missing_finish_reason_for_text=True)

OpenRouterProvider and AzureOpenAIProvider accept the same option. It applies to non-empty text only. A tool-call stream always needs a finish reason.

Anthropic's prompt-cache fallback never restarts a stream that has already delivered text.

Observe retries

HookEvent.RETRY carries RetryData from harnessx.hooks:

Field	Meaning
`kind`	`"model"` or `"tool"`
`name`	Configured model name or tool name
`attempt` / `next_attempt`	Failed and scheduled attempt numbers, counted from 1
`wait_seconds`	Computed delay before trying again
`error`	Failure message
`provider`	Provider diagnostic label for model retries; `None` for tool retries

LLM_RESPONSE also includes provider. For a successful fallback call it identifies the provider that answered, as does agent.provider.last_served. The retry hook's diagnostic label can be the chain's last serving provider or the wrapper name; it is not a reliable identifier of the failing one. See Hooks for an observer example.

The LangSmith extension reads the same value, so a run that failed over is labelled with the vendor that answered rather than the configured primary. Traces and cost attribution follow the call.

When a provider must be a live object

Configuration names providers, so it cannot carry a provider holding a pre-configured SDK client. For that case, build the chain yourself with FallbackProvider and pass it as Agent(provider=...). Members can be live providers, provider names, or Fallback entries.

OpenAIProvider lives in harnessx.providers and needs harnessx[openai]; each vendor provider is gated on its own SDK being installed.

from harnessx import Agent, AgentConfig, Fallback, FallbackProvider
from harnessx.providers import OpenAIProvider

chain = FallbackProvider(
    OpenAIProvider(client=my_client),
    fallbacks=[Fallback("anthropic", model="claude-sonnet-4-6")],
    switch_after=2,
)
async with chain:
    async with Agent(config=AgentConfig(model="gpt-5", provider="openai"), provider=chain) as agent:
        ...

Setting both AgentConfig.fallbacks and provider= raises ConfigurationError: they describe the same thing twice. Passing a live provider inside a Fallback in configuration raises as well, and the message points here.

Ownership

A chain built from AgentConfig.fallbacks is owned by the agent, which closes it along with everything it created from a name. A chain you construct and inject is yours, as any injected provider is; close it with async with or aclose(). Provider instances placed inside a chain stay owned by whoever created them.

"fallback" is the chain's diagnostic name, not a built-in provider factory name.

Built-in Anthropic, OpenAI, Azure, and OpenRouter clients constructed by HarnessX disable the vendor SDK's own retries, so the engine owns the attempt count. An injected vendor client keeps its configured retry behavior. Custom providers can override LLMProvider.is_transient(exc) and retry_after(exc); middleware transforms calls and results but does not own a retry loop.

Tool and MCP retries

Tool attempts use ToolRetry, independently of AgentConfig.retry. Set it per registration or as ToolPolicy(retry=...). Automatic repetition requires a safe or idempotent replay policy. A TransientToolError declares a temporary failure that did not take effect; unknown outcomes for manual tools still require recovery.

See Retry a tool for retry_if_result and error-result handling, and MCP retry policies for server-level configuration.