# Retries & fallback models Configure RetryPolicy, add fallback models with AgentConfig.fallbacks, and observe model and tool retries. Configure how an agent handles temporary model failures, when it tries another provider, and how your application observes retries. Failover is configuration: name the primary in `AgentConfig.provider` and the providers to try after it in `AgentConfig.fallbacks`. A `Fallback` can select a different model for one entry in the chain; there is no `FallbackModel` class in this API. `FallbackProvider` covers the case configuration cannot express, covered in [When a provider must be a live object](#when-a-provider-must-be-a-live-object). ## Add a fallback model or provider `AgentConfig.fallbacks` lists the providers to try after the primary, in order. Each `Fallback` names a provider and may override `model` and `max_tokens` for that entry alone. An omitted override reuses the request's value; a different vendor does not automatically get a compatible model ID. The example below uses Anthropic first and OpenAI as the backup. Install the OpenAI extra with `pip install "harnessx[openai]"`. Export `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `PRIMARY_MODEL`, and `FALLBACK_MODEL`; use model IDs enabled for those accounts. ~~~python import asyncio import os from harnessx import Agent, AgentConfig, Fallback, RetryPolicy async def main(): config = AgentConfig( model=os.environ["PRIMARY_MODEL"], provider="anthropic", max_tokens=4096, fallbacks=[Fallback("openai", model=os.environ["FALLBACK_MODEL"], max_tokens=4096)], retry=RetryPolicy(attempts=2, max_backoff_seconds=10, cooldown_seconds=60), ) async with Agent(config=config) as agent: result = await agent.run("Explain database indexes in three sentences.") result.raise_for_status() print(result.output) if __name__ == "__main__": asyncio.run(main()) ~~~ The agent builds the chain and closes it, so there is nothing extra to clean up. For another model on the same provider, name that provider again with a different `Fallback(..., model=...)`. For a different vendor, supply that vendor's model ID and dependency/credentials. | Setting | Default | Behavior | | --- | --- | --- | | `Fallback.provider` | required | Built-in or registered provider name. Configuration accepts names only. | | `Fallback.model` | `None` | Model ID override for this entry. | | `Fallback.max_tokens` | `None` | Positive reply-budget override for this entry. | | `RetryPolicy.switch_after` | `1` | Transient failures one provider accumulates before the next is tried within the same call. | | `RetryPolicy.cooldown_seconds` | `0` | How long a provider that was passed over is skipped. | Strikes persist across calls and reset when that provider succeeds. Every call starts with the first eligible provider. If all are cooling down, the chain tries them all rather than returning without a request. Because the chain is configuration, it serializes with the rest of `AgentConfig`. Durable runs and Temporal workers carry failover like any other setting, and a restored session rebuilds the same chain. In-memory strikes and cooldown timers are runtime state and are not persisted. Do not also pass a provider object: `Agent(config=AgentConfig(fallbacks=[...]), provider=...)` raises `ConfigurationError`. See [When a provider must be a live object](#when-a-provider-must-be-a-live-object). ### Model limits per member Each member has its own model limits. A cap learned from one member's refusal applies to that provider and model only, and a member's cap and any repaired budget survive snapshots. A `Fallback.max_tokens` cannot raise a member's budget above a cap that member has reported. ### A conversation that changes vendor A fallback can move a conversation from one vendor to another mid-session. When it reaches Anthropic, thinking from other vendors carries no Anthropic signature and is left out of the request, as are private keys such as Gemini's thought signature. Native signed thinking, redacted thinking and server tool blocks keep their order and metadata in snapshots and recorder payloads. A pending native continuation, such as a paused Anthropic turn, stays on the member that started it. Transcript repair and task reminders do not alter it, and request middleware must not change it. ## Configure model retries `AgentConfig.retry` controls the engine's attempts for one model step. The default classification retries rate limits, server errors, timeouts, and connection failures, including the vendor SDKs' own `APIConnectionError`. A connection dropped part way through a reply (`RemoteProtocolError`, `ReadError` and the other httpx transport errors) is retried too. Other HTTP 4xx errors, including bad requests and authentication errors, stop immediately. Quota and billing errors stop too, even when they arrive as a 429. A forward proxy that refuses the tunnel with a 4xx also stops immediately. An egress proxy denying a destination on policy grounds has made a decision, not hit a temporary fault. Retrying it only spends the budget to be refused again. ~~~python from harnessx import AgentConfig, RetryPolicy config = AgentConfig(retry=RetryPolicy( attempts=3, backoff_seconds=0.5, max_backoff_seconds=10, call_timeout_seconds=60, )) ~~~ | Field | Default | Meaning | | --- | --- | --- | | `attempts` | `4` | Total engine attempts for one model step, including the initial call. `1` disables engine retries. | | `rate_limit_attempts` | `8` | How many tries a rate limit (429) or an overload (529) gets in total. It can't be lower than `attempts`. | | `rate_limit_budget_seconds` | `120.0` | The most time, in seconds, a step spends waiting on those extra tries. `None` means no limit. | | `backoff_seconds` | `0.5` | Initial retry delay, doubled after each failed attempt. | | `max_backoff_seconds` | `60.0` | Maximum delay for one wait, including a server's `Retry-After`. Must be positive. | | `call_timeout_seconds` | `None` | Timeout for one engine model attempt. When unset, derived from the reply budget with a minimum of 300 seconds. An explicit positive value overrides it. | | `stream_idle_timeout_seconds` | `180.0` | Longest silence allowed between two chunks of a streamed reply. A stream that stalls past it is abandoned and the call retried. `None` turns the check off. | | `stream_first_event_timeout_seconds` | `20.0` | Longest wait for the first event of a reply, on providers that send one as soon as the model starts (Anthropic). Raised by 10 seconds per 100K tokens of request, and never longer than `stream_idle_timeout_seconds`. Other providers wait the idle limit for the first event too. `None` turns the check off. | | `stream_first_event_growth` | `2.0` | How much longer each retry waits for the first event. By default that's 20, 40, 80, then 160 seconds, up to the idle limit. Use `1.0` to wait the same time on every try. | | `switch_after` | `1` | Transient failures one provider accumulates before the chain moves to the next, within the same call. Must be a positive integer. | | `cooldown_seconds` | `0.0` | How long a provider that was passed over is skipped. Zero makes it eligible again on the next call. | | `jitter` | `0.25` | How much of the backoff is randomly shaved off, so agents that failed together don't all retry at the same moment. `0` makes waits exact. | After a failed attempt, the engine waits an exponential backoff, minus a random slice of up to `jitter`. If the server sent `Retry-After`, the wait is at least that long: ask for three seconds and you get at least three. No single wait goes past `max_backoff_seconds`. `Retry-After` can be a number of seconds or an HTTP date, and a `retry_after` field in the error body works too. ~~~python from harnessx import RetryPolicy policy = RetryPolicy(backoff_seconds=1, max_backoff_seconds=5, jitter=0) assert policy.wait_for(1) == 1 assert policy.wait_for(3) == 4 assert policy.wait_for(1, retry_after=60) == 5 ~~~ ### Rate limits Rate limits usually clear up within a minute, so they get more tries than other errors. By default a 429 or 529 is tried up to 8 times, as long as the waiting adds up to two minutes or less. Server errors and dropped connections still stop after `attempts`. A 429 about quota or billing won't go away by waiting, so it fails straight away. If you write your own provider, override `LLMProvider.is_rate_limited(exc)` to tell the engine which of your errors are rate limits. While the agent sits out a long `Retry-After`, the run stream sends `WAITING` events with `{"on": "retry", "seconds": ...}`. A UI can show that it's waiting, so it doesn't look stuck. See [Streaming](../streaming/). To give rate limits the same budget as everything else: ~~~python from harnessx import AgentConfig, RetryPolicy # Four attempts for every failure, rate limits included. config = AgentConfig(retry=RetryPolicy(rate_limit_attempts=4)) ~~~ The attempt timeout applies to one call through a fallback chain as a whole. It is not a total run deadline. Iteration and cost budgets remain in [Limits](../agents/). ## How retry and failover interact With `switch_after=1` and two providers: 1. The engine calls the chain. 2. A transient primary failure makes the chain try the backup immediately. 3. If the backup also fails transiently, the error returns to the engine. 4. The engine waits and starts another attempt if `RetryPolicy.attempts` allows it. Try it with no API key: `uv run python examples/05-control/retries_and_fallback.py`. `attempts` counts calls to the chain, not individual vendor requests. Two engine attempts across two providers can make up to four vendor requests. `attempts=1` still permits failover within that single chain call. With `switch_after=2`, the first primary failure returns to the engine; the primary needs a second transient failure before the backup becomes eligible through that threshold. A deterministic failure stops the chain immediately under the default classifier. Failed member calls are not separate `RETRY` hook events: that hook describes the engine scheduling another attempt. ## Streaming behavior The fallback chain can switch providers only **before its first chunk** reaches the caller. Once a provider has emitted a chunk, a later exception returns to the engine. If the engine retries, it emits `ATTEMPT_RESET`; discard the provisional text from that attempt. Use `stream_text(..., on_reset=...)` for a text UI, or handle `RunEventType.ATTEMPT_RESET` in the event stream. See [Streaming](../streaming/). The final `RunResult` remains authoritative. ## Streams that stall or end early What arrives from a broken stream can look like a whole reply: an answer missing its end, or a tool call missing arguments. So the harness never uses it. - **A stream that never starts.** Anthropic sends its first event (`message_start`) within a second or two, even when the model is about to think for a while. If nothing arrives within `RetryPolicy.stream_first_event_timeout_seconds` (20 seconds, plus 10 for every 100K tokens of request), the stream is dropped and retried. Each retry waits twice as long, so a server that takes 25 seconds gets through on the second try. From then on the agent knows the server is slow and gives later calls enough time up front. OpenAI's and Gemini's reasoning models stay silent until they've finished thinking, so on those providers the first event gets the idle limit instead. - **A stall.** No chunk for `RetryPolicy.stream_idle_timeout_seconds` (180 by default) abandons the stream. Streamed tool arguments count as activity. Anthropic's provider asks for them as they are written (`eager_input_streaming`), so a long `write_file` keeps the stream alive. Gemini and some OpenAI-compatible models send a whole tool call at once, so writing a large file there can be quiet for tens of seconds; the idle limit allows for that. - **An early end.** An Anthropic stream with no stop reason, or an OpenAI or Gemini stream with no finish reason, raises `IncompleteStreamError`. This includes the stream Anthropic's provider opens itself for a long non-streaming call. - **Broken tool arguments.** If a tool call's arguments arrive garbled, or stop halfway, the tool doesn't run. The model gets an error saying the JSON was incomplete and sends the call again. That doesn't use up a retry. When a reply stops because it ran out of tokens, [truncation recovery](../loop-reliability/) handles it instead. These deadlines bound `agent.run()` as well as `run_stream()`: on Anthropic, `run()` uses the streaming call internally and returns the same result. Other providers keep their non-streaming call for `run()`, bounded by `call_timeout_seconds`. A stall and an early end are retried like a dropped connection, under the same `attempts`. `IncompleteStreamError` is importable from `harnessx.errors` and is a `ConnectionError`. Some OpenAI-compatible servers never send a finish reason for plain text. Allow that explicitly on the provider: ~~~python from harnessx.providers import OpenAIProvider provider = OpenAIProvider(allow_missing_finish_reason_for_text=True) ~~~ `OpenRouterProvider` and `AzureOpenAIProvider` accept the same option. It applies to non-empty text only. A tool-call stream always needs a finish reason. Anthropic's prompt-cache fallback never restarts a stream that has already delivered text. ## Observe retries `HookEvent.RETRY` carries `RetryData` from `harnessx.hooks`: | Field | Meaning | | --- | --- | | `kind` | `"model"` or `"tool"` | | `name` | Configured model name or tool name | | `attempt` / `next_attempt` | Failed and scheduled attempt numbers, counted from 1 | | `wait_seconds` | Computed delay before trying again | | `error` | Failure message | | `provider` | Provider diagnostic label for model retries; `None` for tool retries | `LLM_RESPONSE` also includes `provider`. For a successful fallback call it identifies the provider that answered, as does `agent.provider.last_served`. The retry hook's diagnostic label can be the chain's last serving provider or the wrapper name; it is not a reliable identifier of the failing one. See [Hooks](../hooks/#observe-retries-and-serving-providers) for an observer example. The LangSmith extension reads the same value, so a run that failed over is labelled with the vendor that answered rather than the configured primary. Traces and cost attribution follow the call. ## When a provider must be a live object Configuration names providers, so it cannot carry a provider holding a pre-configured SDK client. For that case, build the chain yourself with `FallbackProvider` and pass it as `Agent(provider=...)`. Members can be live providers, provider names, or `Fallback` entries. `OpenAIProvider` lives in `harnessx.providers` and needs `harnessx[openai]`; each vendor provider is gated on its own SDK being installed. ~~~python from harnessx import Agent, AgentConfig, Fallback, FallbackProvider from harnessx.providers import OpenAIProvider chain = FallbackProvider( OpenAIProvider(client=my_client), fallbacks=[Fallback("anthropic", model="claude-sonnet-4-6")], switch_after=2, ) async with chain: async with Agent(config=AgentConfig(model="gpt-5", provider="openai"), provider=chain) as agent: ... ~~~ Setting both `AgentConfig.fallbacks` and `provider=` raises `ConfigurationError`: they describe the same thing twice. Passing a live provider inside a `Fallback` in configuration raises as well, and the message points here. ## Ownership A chain built from `AgentConfig.fallbacks` is owned by the agent, which closes it along with everything it created from a name. A chain you construct and inject is yours, as any injected provider is; close it with `async with` or `aclose()`. Provider instances placed inside a chain stay owned by whoever created them. `"fallback"` is the chain's diagnostic name, not a built-in provider factory name. Built-in Anthropic, OpenAI, Azure, and OpenRouter clients constructed by HarnessX disable the vendor SDK's own retries, so the engine owns the attempt count. An injected vendor client keeps its configured retry behavior. Custom providers can override `LLMProvider.is_transient(exc)` and `retry_after(exc)`; middleware transforms calls and results but does not own a retry loop. ## Tool and MCP retries Tool attempts use `ToolRetry`, independently of `AgentConfig.retry`. Set it per registration or as `ToolPolicy(retry=...)`. Automatic repetition requires a `safe` or `idempotent` replay policy. A `TransientToolError` declares a temporary failure that did not take effect; unknown outcomes for manual tools still require recovery. See [Retry a tool](../tools/#retry-a-tool) for `retry_if_result` and error-result handling, and [MCP retry policies](../mcp/#retry-policies-for-mcp-tools) for server-level configuration.