Model providers · HarnessX

Agent() uses Anthropic by default. Select a provider and an explicit model ID through AgentConfig; the tool and result APIs remain the same.

Provider configuration

Provider	Dependency	Environment variable
anthropic	Included	ANTHROPIC_API_KEY
openai	harnessx[openai]	OPENAI_API_KEY
gemini	harnessx[gemini]	GEMINI_API_KEY
openrouter	harnessx[openrouter]	OPENROUTER_API_KEY
azure	harnessx[azure]	AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, OPENAI_API_VERSION

Choose a model available to your account. The SDK does not maintain a live provider model catalog.

import os
from harnessx import Agent, AgentConfig

agent = Agent(config=AgentConfig(
    provider="openrouter",
    model=os.environ["AGENT_MODEL"],
    system_prompt="You are a concise data-analysis assistant.",
))

The default config uses anthropic and claude-sonnet-4-6. Changing the provider does not automatically choose a compatible model; set both.

On Azure the model is the deployment name, and an endpoint plus API version are required. AzureOpenAIProvider also accepts azure_endpoint, api_key, api_version, and azure_deployment for explicit construction and injection. "azure" is the only Azure provider name.

Images, documents and audio

A user turn can carry a file as well as text. Pass agent.run() or agent.run_stream() a list of content blocks instead of a string: a text block for the question, and an image, document or audio block for the file. A media block holds base64 data and its media type. agent.stream_text() takes a string only.

import base64
from pathlib import Path
from harnessx import Agent, AgentConfig

async def describe(path: str) -> str:
    data = base64.b64encode(Path(path).read_bytes()).decode()
    async with Agent(config=AgentConfig(provider="gemini", model="gemini-3.6-flash")) as agent:
        result = await agent.run([
            {"type": "text", "text": "What is on this receipt?"},
            {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}},
        ])
        return result.raise_for_status().output

Block	Media types
`image`	`image/jpeg`, `image/png`, `image/gif`, `image/webp`, `image/heic`, `image/heif`
`document`	`application/pdf`
`audio`	`audio/wav`, `audio/mp3`, `audio/mpeg`, `audio/aiff`, `audio/aac`, `audio/ogg`, `audio/flac`, `audio/mp4`, `audio/x-m4a`, `audio/webm`

Media blocks belong to user messages. The media type is checked when the message is built: Message(role="user", content=[...]) raises ValueError for a type outside this table, such as image/jpg, and the same block passed to agent.run() fails the run before any model call.

Each provider translates the blocks into its vendor's format:

Provider	Images	PDFs	Audio
gemini	Yes	Yes	Yes
openai, azure, openrouter	Yes	Yes	WAV and MP3 only. Another audio type fails the run with `ValueError`
anthropic	Yes	Yes	No. The run fails with `ValueError` instead of sending the prompt without the clip

Whether a particular model accepts a given file is up to the vendor. examples/01-basics/media_input.py sends an image, PDF or audio clip to Gemini:

uv run python examples/01-basics/media_input.py path/to/photo.jpg

With no path it sends a 1x1 PNG held in the script. It needs the gemini extra and GEMINI_API_KEY, and HARNESSX_MODEL selects the model.

Retries, fallback, and caching

The engine retries transient model failures under AgentConfig.retry, and AgentConfig.fallbacks names the providers to try after the primary. See Retries & fallback models.

Every provider also receives a prompt-cache hint for the stable prefix and translates it into its vendor's mechanism. See Prompt caching.

Reply budgets

Set AgentConfig.max_tokens to choose your own reply budget.

Model limits

With max_tokens unset, the reply budget comes from the model's own limits: its output limit, capped at 32K and fitted into what's left of the context window. Anthropic's provider gets those limits from the Models API and remembers them for the rest of the process. If the built-in table already knows the model, the first call goes ahead with the table's numbers while the API is asked; a model the table doesn't know waits for the API. AnthropicProvider.clear_model_limits_cache() clears what it remembered. The other providers use the built-in table, matched by longest prefix. If the table doesn't know your model, or has it wrong, register its limits:

from harnessx import register_model_limits

register_model_limits("my-finetune", context_window=200_000, max_output=16_000)

register_model_limits(prefix, *, context_window, max_output) applies to every model whose normalized id starts with prefix, and beats every other source. ModelLimits(context_window=..., max_output=...) is the record it stores. A custom provider that can look limits up overrides model_limits(model) to return a ModelLimits, or None to fall back to the table; it may be a coroutine. A default_max_tokens() override sets a custom provider's default budget. See Long-run reliability for the lookup order and what happens when a reply is cut off.

Stop reasons and usage

Every provider maps its vendor's stop or finish reason onto StopReason:

`StopReason`	Anthropic	OpenAI-compatible	Gemini
`END_TURN`	`end_turn`	`stop`	`STOP`
`TOOL_USE`	`tool_use`	`tool_calls`	a reply with function calls
`MAX_TOKENS`	`max_tokens`	`length`	`MAX_TOKENS`
`STOP_SEQUENCE`	`stop_sequence`
`SAFETY`		`content_filter`	`SAFETY`, `BLOCKLIST`, `PROHIBITED_CONTENT`
`OTHER`	anything else	anything else	`MALFORMED_FUNCTION_CALL` and anything else

TokenUsage.thinking_tokens reports reasoning tokens where the provider counts them separately.

Refusals, paused turns and thinking tokens

StopReason.REFUSAL: Anthropic's refusal, or an OpenAI-compatible reply carrying message.refusal.
StopReason.PAUSE_TURN: Anthropic's pause_turn, from long server-tool turns. The loop sends the turn back to resume it.
Anthropic's model_context_window_exceeded maps to MAX_TOKENS.
Gemini's SPII, RECITATION and IMAGE_SAFETY map to SAFETY, and MALFORMED_FUNCTION_CALL to OTHER. A MAX_TOKENS reply that carries function calls stays MAX_TOKENS.

A run that ends in REFUSAL or SAFETY completes but is not ok: result.refused is true and raise_for_status() raises RunRefused. Tool calls in such a reply are not run.

TokenUsage.output_tokens includes thinking tokens on every built-in provider, so cost estimates and max_cost_dollars count them; thinking_tokens is the breakdown.

Streams without a finish reason

A streamed reply must end with the provider's stop or finish reason. One that stalls or ends early is retried. For an OpenAI-compatible server that omits the finish reason on plain text, pass allow_missing_finish_reason_for_text=True to OpenAIProvider, OpenRouterProvider, or AzureOpenAIProvider:

from harnessx.providers import OpenAIProvider

provider = OpenAIProvider(allow_missing_finish_reason_for_text=True)

It never lets an incomplete tool call through. See Streams that stall or end early.

Requiring a tool call

By default the model chooses between calling a tool and answering with text. An agent whose every turn should be a tool call can say so with tool_choice, so the model cannot answer with prose that describes the call it meant to make:

from harnessx import AnthropicProvider, ToolChoice

provider = AnthropicProvider(tool_choice=ToolChoice.ANY)

Value	Meaning
`"auto"`	The model may call a tool or answer with text
`"any"`	The model must call one of the tools
`"none"`	The model may not call tools

Every built-in provider accepts it, and sends it in its own vendor's form: OpenAI spells "any" as "required", and Gemini sends it as its function-calling mode, alongside the cached prompt when prompt caching is on. Each provider also reads an environment variable: ANTHROPIC_TOOL_CHOICE, OPENAI_TOOL_CHOICE, GEMINI_TOOL_CHOICE, OPENROUTER_TOOL_CHOICE, AZURE_OPENAI_TOOL_CHOICE. Left unset, nothing is added to the request.

Long tool arguments on Anthropic

Anthropic's provider asks for tool arguments to stream as the model writes them (eager_input_streaming on each tool). Without it, the API holds a tool call back until its arguments are complete, so a model writing a whole report with write_file sends nothing for minutes and looks stalled. Server tools such as web search, and tools that set the field themselves, are left alone. An Anthropic-compatible endpoint that rejects the field gets the request again without it. To turn it off:

provider = AnthropicProvider(eager_tool_streaming=False)

The Anthropic SDK is forgiving about streamed arguments, so JSON that was garbled or cut off could still reach a tool as a partial input. To stop that, the provider keeps each call's argument text as it streams and checks it properly once the reply is done. If it doesn't parse, the tool doesn't run; the model gets an error and sends the call again. See Long-run reliability.

Tool calls as they are written

Every built-in provider reports a tool call while the model is still writing it, as RunEventType.TOOL_CALL_DELTA on the run stream. The first delta for a call has the tool's name and no arguments yet.

Provider	What the deltas carry
Anthropic	The arguments in fragments, as the model writes them
OpenAI, Azure OpenAI, OpenRouter	The arguments in fragments. A call that arrives without an id gets one at its first fragment and keeps it when it runs
Gemini	The whole arguments in one delta, since Gemini sends each call complete

Run the provider example

uv run python examples/01-basics/switching_providers.py \
  --provider openrouter \
  --model YOUR_MODEL_ID \
  --prompt "What is 40 * 10?"

Add --no-stream to use ordinary runs. Omit --prompt for a multi-turn conversation. The example includes a real calculator tool.

Inject a provider

Agent accepts a provider implementing LLMProvider for custom adapters or scripted tests. Only two different built-in names contradict each other: Agent(provider=OpenAIProvider(), config=AgentConfig(provider="anthropic")) is rejected. A custom provider with any name is accepted with any config label. Non-Anthropic providers require explicit configuration and a model.

To use your own SDK client, wrap it in a provider: Agent(provider=AnthropicProvider(client=client)).

A custom provider implements create(), optionally stream(), and count_tokens(*, model, messages, system: str | None, tools); accept system=None, matching create(). Override default_max_tokens(model) to choose its reply budget when max_tokens is unset. The prompt-cache hint is only offered to a provider whose create() declares a cache parameter; a **kwargs signature does not count.

Register a provider name

register_provider(name, factory, *, replace=False) adds a name that AgentConfig(provider=name) accepts and make_provider(name) builds by calling factory(**kwargs). Built-in names cannot be replaced; registering an existing custom name again requires replace=True. registered_providers() lists the custom names and unregister_provider(name) removes one.

from harnessx import Agent, AgentConfig, register_provider
from myapp.providers import ScriptedProvider  # your LLMProvider implementation

register_provider("scripted", ScriptedProvider)  # factory(**kwargs) -> LLMProvider

agent = Agent(config=AgentConfig(provider="scripted", model="fixture"))

Registration is process-local. A process that resumes a persisted session must register the same names before loading it, exactly as it must bind the same tools.

Resource ownership

The agent closes a provider it constructs, including a fallback chain built from AgentConfig.fallbacks. An injected provider stays caller-owned. Every LLMProvider is an async context manager, so async with closes the clients it built:

from harnessx import Agent, AgentConfig, AnthropicProvider

async def answer(prompt):
    async with AnthropicProvider() as provider:
        async with Agent(config=AgentConfig(), provider=provider) as agent:
            result = await agent.run(prompt)
            return result.output

await provider.aclose() does the same outside a with block. Use async with agent to release the agent's own resources reliably.

Diagnose failures

A key being present does not establish that a service is reachable or a model is enabled for that account. Inspect RunResult.error and provider responses for authentication, rate, model availability, or unsupported parameter failures. Keep temperature unset unless the selected model accepts it.

Decisions are not a provider

Decisions with Jev has its own DecisionProvider contract. jev is not an AgentConfig.provider value, and its usage is accounted separately.