# Durable agent runs

Persist execution, identify sessions and runs, resume work, and use run handles for results and progress.

This is an **advanced topic**. Start with [Agents](../agents/), [Tools](../tools/), and [Streaming](../streaming/). Add durable execution when an application needs to recover unfinished work or persist approval waits across process restarts.

For example, a sales agent writes a report, saves the tool result, and then its Python process crashes before delivering the answer. A replacement process can restore the run, reuse that committed result, and finish the answer. The [PostgreSQL walkthrough](../backends/#crash-and-resume) demonstrates this with a report file and a tool invocation count that stays at one.

## When to use durable execution

| Need | API | What is retained |
| --- | --- | --- |
| Run an agent in your application process | `Agent.run()` or `Agent.run_stream()` | Conversation state while that instance lives |
| Explicitly save and reload a conversation | `Agent.save_session()` / `Agent.load_session()` | A conversation snapshot, configuration, and supported extension state |
| Recover interrupted execution or a persisted approval | `AgentRuntime` | Execution boundaries, messages, tool outcomes, approvals, events, and artifact references |

An ordinary agent does not need a runtime or registry. A conversation snapshot does not resume an in-flight tool call. Choose `AgentRuntime` when those execution boundaries matter to your workflow.

## What each object does

| Object | Responsibility |
| --- | --- |
| `Agent` | Defines the model, prompt, tools, permissions, middleware, and extensions |
| `PostgresBackend` | Stores durable state in your PostgreSQL database |
| `AgentRuntime` | Drives the agent, persists progress, and exposes start, resume, approval, and control operations |
| `RunHandle` | Gives your application access to one submitted or resumed run's progress and result |
| `RunResult` | Describes the run's output, status, usage, errors, and pending operations |

For PostgreSQL, pass the configured agent directly: `AgentRuntime(agent, backend=backend)`. **No `AgentRegistry` or `AgentRef` is required for this pattern.**

On a single host, `SQLiteBackend` is the default store. Every backend is an async context manager that initialises on entry and closes on exit. When a context manager does not fit, `await SQLiteBackend.connect(path)` constructs and initialises one in a single call:

```python
from harnessx import Agent, AgentRuntime, SQLiteBackend


async def local_run(prompt: str):
    async with SQLiteBackend("runtime.sqlite3") as backend:
        async with AgentRuntime(Agent(), backend=backend) as runtime:
            return await runtime.run(prompt)
```

The runtime offers the same three verbs as `Agent`:

| Verb | What it does |
| --- | --- |
| `run()` | Waits for the `RunResult` |
| `run_stream()` | Yields typed events |
| `stream_text()` | Yields the answer's text, and raises `RunError` if the run does not complete |

## Start a durable run

```python
from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend, RunResult


async def start_run(dsn: str, prompt: str) -> tuple[str, RunResult]:
    async with PostgresBackend(dsn) as backend:          # opens the pool, migrates
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent, AgentRuntime(agent, backend=backend) as runtime:
            handle = await runtime.submit(prompt, request_id="first-request")
            return handle.session_id, await handle.result()
```

Entering the runtime's context manager calls `start()`. `submit()` begins
driving a run and returns a handle you can await. Store the returned
`session_id` beside your application's own chat or workflow record: it is the
only way back to this conversation after a restart.

`runtime.run(prompt)` combines submit and wait. `run_stream()` and
`stream_text()` are the streaming forms.

## Sessions and run IDs

| Identifier | Meaning | Application use |
| --- | --- | --- |
| `session_id` | One durable conversation or workflow session | Save when starting; pass to `resume()` after a restart |
| `run_id` | One submitted request within that session | Associate output and progress with a particular execution |
| `request_id` | Application-supplied submission deduplication key | Reuse for the same request in the same session |
| Event `cursor` | A position in one run's stored event stream | Reconnect after the last processed event |

A session can contain multiple sequential runs. Resuming an unfinished run preserves its run ID; submitting another message creates another run. `resume()` takes a **session ID**, loads that session's latest persisted run, and resumes it if eligible. It does not select an arbitrary session or look up a user automatically.

Request deduplication is scoped to a session. It does not make an external tool effect idempotent. Your application must also retain the session ID; repeating `start()` creates a new session.

## Resume after a restart

Create a fresh agent with matching configuration and compatible tools and extensions. Connect to the same PostgreSQL database and schema. Read the saved session ID and explicitly resume it:

```python
import json
from pathlib import Path

from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend, RunResult


async def resume_run(dsn: str, session_file: Path) -> RunResult | None:
    saved = json.loads(session_file.read_text(encoding="utf-8"))
    async with PostgresBackend(dsn) as backend:
        agent = Agent(config=AgentConfig(system_prompt="You are helpful."))
        async with agent:
            runtime = AgentRuntime(agent, backend=backend)
            try:
                handle = await runtime.resume(saved["session_id"])
                if handle is None:
                    print("No run was resumed.")
                    return None
                print("Resuming run:", handle.run_id)
                return await handle.result()
            finally:
                await runtime.aclose()  # alias of stop()
```

Call `await resume_run(dsn, Path(".agent_sessions/my-chat.json"))`, where that file is the `{"session_id": ...}` record your application saved when it started the run. If the earlier function finished normally, there is no unfinished run to drive. Use the crash demonstration below to exercise recovery.

**Do not enter the runtime context manager before calling `resume()`.** That context manager starts a new session. The outer `async with agent` only manages the agent's resources and is safe for this recovery pattern.

`resume()` returns a handle for a resumed run, or `None` when the session has no run or its latest run is completed/cancelled. It can raise for a missing session, incompatible definition/configuration, or an active worker lease. `None` does not mean the session could not be found.

On a successfully bound runtime, `await runtime.status()` returns a dictionary with `session_id`, `state`, and `run`, the latest `RunResult` or `None` for a session without a run. This lets an application read a completed result even when `resume()` returns no handle.

## Use a RunHandle

A handle provides access to an execution. It is not the agent's answer, and it is not a running agent to serialize into your database. Store the relevant IDs instead.

| Handle API | What it provides |
| --- | --- |
| `handle.session_id` | The session this execution belongs to |
| `handle.run_id` | This execution's ID |
| `await handle.result()` | Wait for a `RunResult` from the current execution |
| `handle.events(after=cursor)` | Read stored events and follow new ones |
| `handle.stream(after=cursor)` | An async-context-managed event stream with its own `result()` method |

Both `await runtime.submit(...)` and a successful `await runtime.resume(...)` return a handle. `run()` returns a `RunResult` directly. Use the new handle returned when resuming after an approval or pause; an earlier local handle can retain its earlier execution result.

Waiting for a result does not guarantee `completed`. A run can stop at an approval wait, pause, cancellation, or failure. Inspect its status before treating the output as a successful answer. Infrastructure failures can also raise exceptions; normal application exception handling still applies.

## Understand a RunResult

```python
def present_result(result) -> None:
    if result.ok:
        print("Answer:", result.output)
    elif result.failed:
        print("Error:", result.error["message"])
    for pending in result.pending:
        print(pending.status, pending.execution_key, pending.call.name)
```

`ok`, `failed` and `needs_input` are the three branches most applications take.
`raise_for_status()` returns the result or raises `RunFailed`,
`RunAwaitingInput`, `RunTruncated` or `RunCancelled`, each carrying the result.
A completed run is not always `ok`: `truncated` marks one whose reply was cut
off at the token budget. [Inspect the result](../agents/#inspect-the-result)
covers the predicates and exceptions.

Of the statuses, `running` is what you see while a run is active. It is also
the last thing persisted before a crash, so it does not prove a worker is
alive.
`awaiting_input` needs an approval or a reconciliation decision. A `cancelled`
run is never resumed automatically and can still have uncertain effects worth
inspecting.

### Refused and limited runs

`refused` and `limited` also mark a completed run that is not `ok`, and
`raise_for_status()` raises `RunRefused` or `RunLimitReached` for them. See
[Inspect the result](../agents/#inspect-the-result).

## Stream progress and reconnect

For a UI, consume the handle's events. This helper takes two async callbacks supplied by your application: `send(event)` renders or forwards a typed event, and `save_cursor(run_id, cursor)` remembers the last processed position.

```python
from harnessx import RunHandle, RunResult


async def forward_progress(
    handle: RunHandle, send, save_cursor, *, after: str = "0"
) -> RunResult:
    async for event in handle.events(after=after):
        await send(event)
        if event.cursor:
            await save_cursor(handle.run_id, event.cursor)
    return await handle.result()
```

On reconnect, provide the saved cursor for that same run. With no cursor, SQL handles start from `"0"` and include previously stored events. A replayed tool event describes past execution; receiving it does not invoke the tool again.

Events include `text_delta`, `text_complete`, `tool_call_start`, `tool_result`, `approval_required`, `recovery_required`, and `run_result`. Render text deltas as provisional. Handle `attempt_reset` by discarding the interrupted attempt's provisional text, and use processed/final output to settle the display. See [Streaming](../streaming/) for response-middleware behavior.

Delivery and saving your UI cursor are separate operations. Reconnection can repeat an event or result summary; deduplicate stored events by event ID and render the run's final state idempotently. Some synthesized terminal summaries have no cursor, which is why the helper checks before saving one.

Disconnecting an event subscriber does not cancel a durable run. Keep the worker and runtime alive for execution to continue. A browser reconnect should normally reattach to the existing run's observation stream, not ask another worker to resume an actively leased session.

## Approvals and uncertain tools

A durable `ASK` permission persists: the run returns `needs_input` and
`result.pending` lists `PendingTool` objects with `status == "approval"`. Record
the decision with `runtime.approve(...)` or `runtime.decline(...)`, passing
`resume=True` to continue in the same call. Several pending tools take one
decision each, then a single `await runtime.resume(session_id)`.

An `uncertain` pending operation is a different thing: the tool may already have
had an external effect. Investigate that system, then pick exactly one
resolution.

| Resolution | API |
| --- | --- |
| Record a verified result | `runtime.resolve_tool(pending, result=ToolResult(...))` |
| Accept another invocation | `runtime.resolve_tool(pending, retry=True)` |
| Abort the run | `runtime.resolve_tool(pending, abort=True)` |

Aborting does not undo an effect that already happened. [Approvals &
recovery](../recovery/) covers execution keys and the replay policies that
decide which calls can repeat on their own.

## Pause, cancel, and cleanup

Controls belong to `AgentRuntime`, while the handle observes a particular run:

| Runtime API | Behavior |
| --- | --- |
| `await runtime.pause()` | Stops new steps and lets active tools settle |
| `await runtime.cancel()` | Requests cooperative cancellation; threads or remote effects may outlive it |
| `await runtime.stop()` or `await runtime.aclose()` | Pauses active local work, closes the agent the runtime executes, and closes a backend the runtime created |
| `await runtime.status()` | Returns `session_id`, `state`, and `run`, the latest `RunResult` or `None` |

Leaving the runtime context manager calls `stop()`. Returning a handle from inside that context does not leave an independent background worker running after the context exits. Keep the runtime open for as long as its driver is needed.

The runtime closes its agent. It closes the backend only if it created that backend; close an explicitly supplied backend yourself. The outer agent contexts also cover failures before an agent is bound to the runtime. Repeated close calls are safe. Create a fresh agent after its resources have been closed.

PostgreSQL stores state and coordinates leases; it does not restart Python processes or schedule recovery workers. Your application or supervisor restarts the worker and calls `resume()`. A hard crash leaves a lease that must expire before another worker claims the session. The default lease lasts 30 seconds with renewal approximately every 10 seconds.

## Binding an agent to a runtime

For SQLite and PostgreSQL, pass the configured `Agent` straight to
`AgentRuntime`. Temporal runs the agent in a worker process, so it takes an
`AgentRef` such as `assistant@v1` and builds the agent from a factory registered
in that worker's `AgentRegistry`.

| Binding | How construction works |
| --- | --- |
| SQLite or PostgreSQL with `Agent` | Pass the instance to `AgentRuntime` |
| SQLite or PostgreSQL with `AgentRef` | Supply `registry=`, or use the process-local default |
| Temporal | Submit an `AgentRef`; configure its factory on the worker |

A version label records intent. It does not prove the code still matches, so
keep agent definitions under your normal deployment controls.

## What survives a failure

The engine records dispatch intent before invoking a tool and a raw result before result middleware. Recovery depends on what actually committed:

| Last saved state | Meaning on recovery |
| --- | --- |
| Prepared call, before dispatch is recorded | The runtime can dispatch the call |
| Dispatch recorded, no saved result | The external outcome can be unknown; a `manual` tool waits for reconciliation |
| Raw result committed | Reuse it and finish processing without invoking the tool again |
| Run completed | Read the saved result; there is no run to resume |

SQL execution-boundary updates and their events commit transactionally. Every persisted phase boundary also emits `HookEvent.CHECKPOINT` on the agent's hooks with `session_id`, `run_id`, `status`, and `phase`. PostgreSQL fencing rejects state writes from workers that lose their leases. It cannot stop an already-running remote request.

External effects and runtime storage are not one transaction. The SDK does not provide universal rollback or exactly-once external effects. An `idempotent` tool requires its external adapter/service to enforce the execution key. A `safe` tool must actually tolerate repetition. Result middleware may run again and should tolerate replay.

Interrupted live-model requests may be reissued, return different content, and incur additional charges. Arbitrary local files are not automatically copied into PostgreSQL. Recovery on another machine requires compatible code and shared access to the workflow's resources.

## Resume semantics

| Symptom | What it means |
| --- | --- |
| `resume` returned `None` | The session has no run, or its latest run already finished. Inspect status on the bound runtime |
| Status is still `running` after a crash | It is the last persisted status, not proof a worker is alive |
| The session is busy | A worker holds the lease, or a crashed worker's lease has not expired yet |

Environment and connection problems are covered under
[Execution backends](../backends/#postgresql).