Durable agent runs

Persist execution, identify sessions and runs, resume work, and use run handles for results and progress.

This is an advanced topic. Start with Agents, Tools, and Streaming. Add durable execution when an application needs to recover unfinished work or persist approval waits across process restarts.

For example, a sales agent writes a report, saves the tool result, and then its Python process crashes before delivering the answer. A replacement process can restore the run, reuse that committed result, and finish the answer. The PostgreSQL walkthrough demonstrates this with a report file and a tool invocation count that stays at one.

When to use durable execution

Need	API	What is retained
Run an agent in your application process	`Agent.run()` or `Agent.run_stream()`	Conversation state while that instance lives
Explicitly save and reload a conversation	`Agent.save_session()` / `Agent.load_session()`	A conversation snapshot, configuration, and supported extension state
Recover interrupted execution or a persisted approval	`AgentRuntime`	Execution boundaries, messages, tool outcomes, approvals, events, and artifact references

An ordinary agent does not need a runtime or registry. A conversation snapshot does not resume an in-flight tool call. Choose AgentRuntime when those execution boundaries matter to your workflow.

What each object does

Object	Responsibility
`Agent`	Defines the model, prompt, tools, permissions, middleware, and extensions
`PostgresBackend`	Stores durable state in your PostgreSQL database
`AgentRuntime`	Drives the agent, persists progress, and exposes start, resume, approval, and control operations
`RunHandle`	Gives your application access to one submitted or resumed run's progress and result
`RunResult`	Describes the run's output, status, usage, errors, and pending operations

For PostgreSQL, pass the configured agent directly: AgentRuntime(agent, backend=backend). No AgentRegistry or AgentRef is required for this pattern.

On a single host, SQLiteBackend is the default store. Every backend is an async context manager that initialises on entry and closes on exit. When a context manager does not fit, await SQLiteBackend.connect(path) constructs and initialises one in a single call:

from harnessx import Agent, AgentRuntime, SQLiteBackend


async def local_run(prompt: str):
    async with SQLiteBackend("runtime.sqlite3") as backend:
        async with AgentRuntime(Agent(), backend=backend) as runtime:
            return await runtime.run(prompt)

The runtime offers the same three verbs as Agent:

Verb	What it does
`run()`	Waits for the `RunResult`
`run_stream()`	Yields typed events
`stream_text()`	Yields the answer's text, and raises `RunError` if the run does not complete

Start a durable run

from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend, RunResult


async def start_run(dsn: str, prompt: str) -> tuple[str, RunResult]:
    async with PostgresBackend(dsn) as backend:          # opens the pool, migrates
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent, AgentRuntime(agent, backend=backend) as runtime:
            handle = await runtime.submit(prompt, request_id="first-request")
            return handle.session_id, await handle.result()

Entering the runtime's context manager calls start(). submit() begins driving a run and returns a handle you can await. Store the returned session_id beside your application's own chat or workflow record: it is the only way back to this conversation after a restart.

runtime.run(prompt) combines submit and wait. run_stream() and stream_text() are the streaming forms.

Sessions and run IDs

Identifier	Meaning	Application use
`session_id`	One durable conversation or workflow session	Save when starting; pass to `resume()` after a restart
`run_id`	One submitted request within that session	Associate output and progress with a particular execution
`request_id`	Application-supplied submission deduplication key	Reuse for the same request in the same session
Event `cursor`	A position in one run's stored event stream	Reconnect after the last processed event

A session can contain multiple sequential runs. Resuming an unfinished run preserves its run ID; submitting another message creates another run. resume() takes a session ID, loads that session's latest persisted run, and resumes it if eligible. It does not select an arbitrary session or look up a user automatically.

Request deduplication is scoped to a session. It does not make an external tool effect idempotent. Your application must also retain the session ID; repeating start() creates a new session.

Resume after a restart

Create a fresh agent with matching configuration and compatible tools and extensions. Connect to the same PostgreSQL database and schema. Read the saved session ID and explicitly resume it:

import json
from pathlib import Path

from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend, RunResult


async def resume_run(dsn: str, session_file: Path) -> RunResult | None:
    saved = json.loads(session_file.read_text(encoding="utf-8"))
    async with PostgresBackend(dsn) as backend:
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent:
            runtime = AgentRuntime(agent, backend=backend)
            try:
                handle = await runtime.resume(saved["session_id"])
                if handle is None:
                    print("No run was resumed.")
                    return None
                print("Resuming run:", handle.run_id)
                return await handle.result()
            finally:
                await runtime.aclose()  # alias of stop()

Call await resume_run(dsn, Path(".agent_sessions/my-chat.json")), where that file is the {"session_id": ...} record your application saved when it started the run. If the earlier function finished normally, there is no unfinished run to drive. Use the crash demonstration below to exercise recovery.

Do not enter the runtime context manager before calling resume(). That context manager starts a new session. The outer async with agent only manages the agent's resources and is safe for this recovery pattern.

resume() returns a handle for a resumed run, or None when the session has no run or its latest run is completed/cancelled. It can raise for a missing session, incompatible definition/configuration, or an active worker lease. None does not mean the session could not be found.

On a successfully bound runtime, await runtime.status() returns a dictionary with session_id, state, and run, the latest RunResult or None for a session without a run. This lets an application read a completed result even when resume() returns no handle.

Use a RunHandle

A handle provides access to an execution. It is not the agent's answer, and it is not a running agent to serialize into your database. Store the relevant IDs instead.

Handle API	What it provides
`handle.session_id`	The session this execution belongs to
`handle.run_id`	This execution's ID
`await handle.result()`	Wait for a `RunResult` from the current execution
`handle.events(after=cursor)`	Read stored events and follow new ones
`handle.stream(after=cursor)`	An async-context-managed event stream with its own `result()` method

Both await runtime.submit(...) and a successful await runtime.resume(...) return a handle. run() returns a RunResult directly. Use the new handle returned when resuming after an approval or pause; an earlier local handle can retain its earlier execution result.

Waiting for a result does not guarantee completed. A run can stop at an approval wait, pause, cancellation, or failure. Inspect its status before treating the output as a successful answer. Infrastructure failures can also raise exceptions; normal application exception handling still applies.

Understand a RunResult

def present_result(result) -> None:
    if result.ok:
        print("Answer:", result.output)
    elif result.failed:
        print("Error:", result.error["message"])
    for pending in result.pending:
        print(pending.status, pending.execution_key, pending.call.name)

ok, failed and needs_input are the three branches most applications take. raise_for_status() returns the result or raises RunFailed, RunAwaitingInput, RunTruncated or RunCancelled, each carrying the result. A completed run is not always ok: truncated marks one whose reply was cut off at the token budget. Inspect the result covers the predicates and exceptions.

Of the statuses, running is what you see while a run is active. It is also the last thing persisted before a crash, so it does not prove a worker is alive. awaiting_input needs an approval or a reconciliation decision. A cancelled run is never resumed automatically and can still have uncertain effects worth inspecting.

Refused and limited runs

refused and limited also mark a completed run that is not ok, and raise_for_status() raises RunRefused or RunLimitReached for them. See Inspect the result.

Stream progress and reconnect

For a UI, consume the handle's events. This helper takes two async callbacks supplied by your application: send(event) renders or forwards a typed event, and save_cursor(run_id, cursor) remembers the last processed position.

from harnessx import RunHandle, RunResult


async def forward_progress(
    handle: RunHandle, send, save_cursor, *, after: str = "0"
) -> RunResult:
    async for event in handle.events(after=after):
        await send(event)
        if event.cursor:
            await save_cursor(handle.run_id, event.cursor)
    return await handle.result()

On reconnect, provide the saved cursor for that same run. With no cursor, SQL handles start from "0" and include previously stored events. A replayed tool event describes past execution; receiving it does not invoke the tool again.

Events include text_delta, text_complete, tool_call_start, tool_result, approval_required, recovery_required, and run_result. Render text deltas as provisional. Handle attempt_reset by discarding the interrupted attempt's provisional text, and use processed/final output to settle the display. See Streaming for response-middleware behavior.

Delivery and saving your UI cursor are separate operations. Reconnection can repeat an event or result summary; deduplicate stored events by event ID and render the run's final state idempotently. Some synthesized terminal summaries have no cursor, which is why the helper checks before saving one.

Disconnecting an event subscriber does not cancel a durable run. Keep the worker and runtime alive for execution to continue. A browser reconnect should normally reattach to the existing run's observation stream, not ask another worker to resume an actively leased session.

Approvals and uncertain tools

A durable ASK permission persists: the run returns needs_input and result.pending lists PendingTool objects with status == "approval". Record the decision with runtime.approve(...) or runtime.decline(...), passing resume=True to continue in the same call. Several pending tools take one decision each, then a single await runtime.resume(session_id).

An uncertain pending operation is a different thing: the tool may already have had an external effect. Investigate that system, then pick exactly one resolution.

Resolution	API
Record a verified result	`runtime.resolve_tool(pending, result=ToolResult(...))`
Accept another invocation	`runtime.resolve_tool(pending, retry=True)`
Abort the run	`runtime.resolve_tool(pending, abort=True)`

Aborting does not undo an effect that already happened. Approvals & recovery covers execution keys and the replay policies that decide which calls can repeat on their own.

Pause, cancel, and cleanup

Controls belong to AgentRuntime, while the handle observes a particular run:

Runtime API	Behavior
`await runtime.pause()`	Stops new steps and lets active tools settle
`await runtime.cancel()`	Requests cooperative cancellation; threads or remote effects may outlive it
`await runtime.stop()` or `await runtime.aclose()`	Pauses active local work, closes the agent the runtime executes, and closes a backend the runtime created
`await runtime.status()`	Returns `session_id`, `state`, and `run`, the latest `RunResult` or `None`

Leaving the runtime context manager calls stop(). Returning a handle from inside that context does not leave an independent background worker running after the context exits. Keep the runtime open for as long as its driver is needed.

The runtime closes its agent. It closes the backend only if it created that backend; close an explicitly supplied backend yourself. The outer agent contexts also cover failures before an agent is bound to the runtime. Repeated close calls are safe. Create a fresh agent after its resources have been closed.

PostgreSQL stores state and coordinates leases; it does not restart Python processes or schedule recovery workers. Your application or supervisor restarts the worker and calls resume(). A hard crash leaves a lease that must expire before another worker claims the session. The default lease lasts 30 seconds with renewal approximately every 10 seconds.

Binding an agent to a runtime

For SQLite and PostgreSQL, pass the configured Agent straight to AgentRuntime. Temporal runs the agent in a worker process, so it takes an AgentRef such as assistant@v1 and builds the agent from a factory registered in that worker's AgentRegistry.

Binding	How construction works
SQLite or PostgreSQL with `Agent`	Pass the instance to `AgentRuntime`
SQLite or PostgreSQL with `AgentRef`	Supply `registry=`, or use the process-local default
Temporal	Submit an `AgentRef`; configure its factory on the worker

A version label records intent. It does not prove the code still matches, so keep agent definitions under your normal deployment controls.

What survives a failure

The engine records dispatch intent before invoking a tool and a raw result before result middleware. Recovery depends on what actually committed:

Last saved state	Meaning on recovery
Prepared call, before dispatch is recorded	The runtime can dispatch the call
Dispatch recorded, no saved result	The external outcome can be unknown; a `manual` tool waits for reconciliation
Raw result committed	Reuse it and finish processing without invoking the tool again
Run completed	Read the saved result; there is no run to resume

SQL execution-boundary updates and their events commit transactionally. Every persisted phase boundary also emits HookEvent.CHECKPOINT on the agent's hooks with session_id, run_id, status, and phase. PostgreSQL fencing rejects state writes from workers that lose their leases. It cannot stop an already-running remote request.

External effects and runtime storage are not one transaction. The SDK does not provide universal rollback or exactly-once external effects. An idempotent tool requires its external adapter/service to enforce the execution key. A safe tool must actually tolerate repetition. Result middleware may run again and should tolerate replay.

Interrupted live-model requests may be reissued, return different content, and incur additional charges. Arbitrary local files are not automatically copied into PostgreSQL. Recovery on another machine requires compatible code and shared access to the workflow's resources.

Resume semantics

Symptom	What it means
`resume` returned `None`	The session has no run, or its latest run already finished. Inspect status on the bound runtime
Status is still `running` after a crash	It is the last persisted status, not proof a worker is alive
The session is busy	A worker holds the lease, or a crashed worker's lease has not expired yet

Environment and connection problems are covered under Execution backends.