# Model providers

Choose model providers, configure retries, and connect a fallback chain.

Agent() uses Anthropic by default. Select a provider and an explicit model ID through AgentConfig; the tool and result APIs remain the same.

## Provider configuration

| Provider | Dependency | Environment variable |
| --- | --- | --- |
| anthropic | Included | ANTHROPIC_API_KEY |
| openai | harnessx[openai] | OPENAI_API_KEY |
| gemini | harnessx[gemini] | GEMINI_API_KEY |
| openrouter | harnessx[openrouter] | OPENROUTER_API_KEY |
| azure | harnessx[azure] | AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, OPENAI_API_VERSION |

Choose a model available to your account. The SDK does not maintain a live provider model catalog.

~~~python
import os
from harnessx import Agent, AgentConfig

agent = Agent(config=AgentConfig(
    provider="openrouter",
    model=os.environ["AGENT_MODEL"],
    system_prompt="You are a concise data-analysis assistant.",
))
~~~

The default config uses anthropic and claude-sonnet-4-6. Changing the provider does not automatically choose a compatible model; set both.

On Azure the `model` is the deployment name, and an endpoint plus API version are required. `AzureOpenAIProvider` also accepts `azure_endpoint`, `api_key`, `api_version`, and `azure_deployment` for explicit construction and injection. `"azure"` is the only Azure provider name.

## Images, documents and audio

A user turn can carry a file as well as text. Pass `agent.run()` or `agent.run_stream()` a list of content blocks instead of a string: a `text` block for the question, and an `image`, `document` or `audio` block for the file. A media block holds base64 data and its media type. `agent.stream_text()` takes a string only.

~~~python
import base64
from pathlib import Path
from harnessx import Agent, AgentConfig

async def describe(path: str) -> str:
    data = base64.b64encode(Path(path).read_bytes()).decode()
    async with Agent(config=AgentConfig(provider="gemini", model="gemini-3.6-flash")) as agent:
        result = await agent.run([
            {"type": "text", "text": "What is on this receipt?"},
            {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}},
        ])
        return result.raise_for_status().output
~~~

| Block | Media types |
| --- | --- |
| `image` | `image/jpeg`, `image/png`, `image/gif`, `image/webp`, `image/heic`, `image/heif` |
| `document` | `application/pdf` |
| `audio` | `audio/wav`, `audio/mp3`, `audio/mpeg`, `audio/aiff`, `audio/aac`, `audio/ogg`, `audio/flac`, `audio/mp4`, `audio/x-m4a`, `audio/webm` |

Media blocks belong to user messages. The media type is checked when the message is built: `Message(role="user", content=[...])` raises `ValueError` for a type outside this table, such as `image/jpg`, and the same block passed to `agent.run()` fails the run before any model call.

Each provider translates the blocks into its vendor's format:

| Provider | Images | PDFs | Audio |
| --- | --- | --- | --- |
| gemini | Yes | Yes | Yes |
| openai, azure, openrouter | Yes | Yes | WAV and MP3 only. Another audio type fails the run with `ValueError` |
| anthropic | Yes | Yes | No. The run fails with `ValueError` instead of sending the prompt without the clip |

Whether a particular model accepts a given file is up to the vendor. `examples/01-basics/media_input.py` sends an image, PDF or audio clip to Gemini:

~~~bash
uv run python examples/01-basics/media_input.py path/to/photo.jpg
~~~

With no path it sends a 1x1 PNG held in the script. It needs the `gemini` extra and `GEMINI_API_KEY`, and `HARNESSX_MODEL` selects the model.

## Retries, fallback, and caching

The engine retries transient model failures under `AgentConfig.retry`, and
`AgentConfig.fallbacks` names the providers to try after the primary. See
[Retries & fallback models](../retries-and-fallbacks/).

Every provider also receives a prompt-cache hint for the stable prefix and
translates it into its vendor's mechanism. See [Prompt caching](../prompt-caching/).

## Reply budgets

Set `AgentConfig.max_tokens` to choose your own reply budget.

### Model limits

With `max_tokens` unset, the reply budget comes from the model's own limits: its
output limit, capped at 32K and fitted into what's left of the context window.
Anthropic's provider gets those limits from the Models API and remembers them
for the rest of the process. If the built-in table already knows the model, the
first call goes ahead with the table's numbers while the API is asked; a model
the table doesn't know waits for the API.
`AnthropicProvider.clear_model_limits_cache()` clears what it remembered. The
other providers use the built-in table, matched by longest prefix. If the table
doesn't know your model, or has it wrong, register its limits:

~~~python
from harnessx import register_model_limits

register_model_limits("my-finetune", context_window=200_000, max_output=16_000)
~~~

`register_model_limits(prefix, *, context_window, max_output)` applies to every
model whose normalized id starts with `prefix`, and beats every other source.
`ModelLimits(context_window=..., max_output=...)` is the record it stores. A
custom provider that can look limits up overrides `model_limits(model)` to
return a `ModelLimits`, or `None` to fall back to the table; it may be a
coroutine. A `default_max_tokens()` override sets a custom provider's default
budget. See [Long-run
reliability](../loop-reliability/#reply-budgets-from-the-model) for the lookup
order and what happens when a reply is cut off.

## Stop reasons and usage

Every provider maps its vendor's stop or finish reason onto `StopReason`:

| `StopReason` | Anthropic | OpenAI-compatible | Gemini |
| --- | --- | --- | --- |
| `END_TURN` | `end_turn` | `stop` | `STOP` |
| `TOOL_USE` | `tool_use` | `tool_calls` | a reply with function calls |
| `MAX_TOKENS` | `max_tokens` | `length` | `MAX_TOKENS` |
| `STOP_SEQUENCE` | `stop_sequence` | | |
| `SAFETY` | | `content_filter` | `SAFETY`, `BLOCKLIST`, `PROHIBITED_CONTENT` |
| `OTHER` | anything else | anything else | `MALFORMED_FUNCTION_CALL` and anything else |

`TokenUsage.thinking_tokens` reports reasoning tokens where the provider counts
them separately.

### Refusals, paused turns and thinking tokens

- `StopReason.REFUSAL`: Anthropic's `refusal`, or an OpenAI-compatible
  reply carrying `message.refusal`.
- `StopReason.PAUSE_TURN`: Anthropic's `pause_turn`, from long
  server-tool turns. The loop sends the turn back to resume it.
- Anthropic's `model_context_window_exceeded` maps to `MAX_TOKENS`.
- Gemini's `SPII`, `RECITATION` and `IMAGE_SAFETY` map to `SAFETY`, and
  `MALFORMED_FUNCTION_CALL` to `OTHER`. A `MAX_TOKENS` reply that carries
  function calls stays `MAX_TOKENS`.

A run that ends in `REFUSAL` or `SAFETY` completes but is not `ok`:
`result.refused` is true and `raise_for_status()` raises `RunRefused`. Tool calls
in such a reply are not run.

`TokenUsage.output_tokens` includes thinking tokens on every built-in provider,
so cost estimates and `max_cost_dollars` count them; `thinking_tokens` is the
breakdown.

### Streams without a finish reason

A streamed reply must end with the provider's stop or finish reason. One that
stalls or ends early is retried. For an OpenAI-compatible server that omits the
finish reason on plain text, pass `allow_missing_finish_reason_for_text=True` to
`OpenAIProvider`, `OpenRouterProvider`, or `AzureOpenAIProvider`:

~~~python
from harnessx.providers import OpenAIProvider

provider = OpenAIProvider(allow_missing_finish_reason_for_text=True)
~~~

It never lets an incomplete tool call through. See [Streams that stall or end
early](../retries-and-fallbacks/#streams-that-stall-or-end-early).

### Requiring a tool call

By default the model chooses between calling a tool and answering with text. An
agent whose every turn should be a tool call can say so with `tool_choice`, so the
model cannot answer with prose that describes the call it meant to make:

~~~python
from harnessx import AnthropicProvider, ToolChoice

provider = AnthropicProvider(tool_choice=ToolChoice.ANY)
~~~

| Value | Meaning |
| --- | --- |
| `"auto"` | The model may call a tool or answer with text |
| `"any"` | The model must call one of the tools |
| `"none"` | The model may not call tools |

Every built-in provider accepts it, and sends it in its own vendor's form: OpenAI
spells `"any"` as `"required"`, and Gemini sends it as its function-calling mode,
alongside the cached prompt when prompt caching is on. Each provider also reads an
environment variable: `ANTHROPIC_TOOL_CHOICE`, `OPENAI_TOOL_CHOICE`,
`GEMINI_TOOL_CHOICE`, `OPENROUTER_TOOL_CHOICE`, `AZURE_OPENAI_TOOL_CHOICE`. Left
unset, nothing is added to the request.

### Long tool arguments on Anthropic

Anthropic's provider asks for tool arguments to stream as the model writes them
(`eager_input_streaming` on each tool). Without it, the API holds a tool call back
until its arguments are complete, so a model writing a whole report with
`write_file` sends nothing for minutes and looks stalled. Server tools such as web
search, and tools that set the field themselves, are left alone. An
Anthropic-compatible endpoint that rejects the field gets the request again
without it. To turn it off:

~~~python
provider = AnthropicProvider(eager_tool_streaming=False)
~~~

The Anthropic SDK is forgiving about streamed arguments, so JSON that was
garbled or cut off could still reach a tool as a partial input. To stop that,
the provider keeps each call's argument text as it streams and checks it
properly once the reply is done. If it doesn't parse, the tool doesn't run; the
model gets an error and sends the call again. See
[Long-run reliability](../loop-reliability/#recovering-from-a-limit).

### Tool calls as they are written

Every built-in provider reports a tool call while the model is still writing it,
as `RunEventType.TOOL_CALL_DELTA` on [the run stream](../streaming/). The first
delta for a call has the tool's name and no arguments yet.

| Provider | What the deltas carry |
| --- | --- |
| Anthropic | The arguments in fragments, as the model writes them |
| OpenAI, Azure OpenAI, OpenRouter | The arguments in fragments. A call that arrives without an id gets one at its first fragment and keeps it when it runs |
| Gemini | The whole arguments in one delta, since Gemini sends each call complete |

## Run the provider example

~~~bash
uv run python examples/01-basics/switching_providers.py \
  --provider openrouter \
  --model YOUR_MODEL_ID \
  --prompt "What is 40 * 10?"
~~~

Add --no-stream to use ordinary runs. Omit --prompt for a multi-turn conversation. The example includes a real calculator tool.

## Inject a provider

Agent accepts a provider implementing `LLMProvider` for custom adapters or scripted tests. Only two *different built-in* names contradict each other: `Agent(provider=OpenAIProvider(), config=AgentConfig(provider="anthropic"))` is rejected. A custom provider with any `name` is accepted with any config label. Non-Anthropic providers require explicit configuration and a model.

To use your own SDK client, wrap it in a provider: `Agent(provider=AnthropicProvider(client=client))`.

A custom provider implements `create()`, optionally `stream()`, and `count_tokens(*, model, messages, system: str | None, tools)`; accept `system=None`, matching `create()`. Override `default_max_tokens(model)` to choose its reply budget when `max_tokens` is unset. The prompt-cache hint is only offered to a provider whose `create()` declares a `cache` parameter; a `**kwargs` signature does not count.

## Register a provider name

`register_provider(name, factory, *, replace=False)` adds a name that `AgentConfig(provider=name)` accepts and `make_provider(name)` builds by calling `factory(**kwargs)`. Built-in names cannot be replaced; registering an existing custom name again requires `replace=True`. `registered_providers()` lists the custom names and `unregister_provider(name)` removes one.

~~~python
from harnessx import Agent, AgentConfig, register_provider
from myapp.providers import ScriptedProvider  # your LLMProvider implementation

register_provider("scripted", ScriptedProvider)  # factory(**kwargs) -> LLMProvider

agent = Agent(config=AgentConfig(provider="scripted", model="fixture"))
~~~

Registration is process-local. A process that resumes a persisted session must register the same names before loading it, exactly as it must bind the same tools.

## Resource ownership

The agent closes a provider it constructs, including a fallback chain built from `AgentConfig.fallbacks`. An injected provider stays caller-owned. Every `LLMProvider` is an async context manager, so `async with` closes the clients it built:

~~~python
from harnessx import Agent, AgentConfig, AnthropicProvider

async def answer(prompt):
    async with AnthropicProvider() as provider:
        async with Agent(config=AgentConfig(), provider=provider) as agent:
            result = await agent.run(prompt)
            return result.output
~~~

`await provider.aclose()` does the same outside a `with` block. Use `async with agent` to release the agent's own resources reliably.

## Diagnose failures

A key being present does not establish that a service is reachable or a model is enabled for that account. Inspect RunResult.error and provider responses for authentication, rate, model availability, or unsupported parameter failures. Keep temperature unset unless the selected model accepts it.

## Decisions are not a provider

[Decisions with Jev](../decisions/) has its own `DecisionProvider` contract.
`jev` is not an `AgentConfig.provider` value, and its usage is accounted
separately.