Part of Advanced topics. Start with Durable agent runs for the runtime API, session identity, run handles, and lifecycle.
AgentRuntime uses the same execution semantics with different storage and scheduling backends. Choose based on deployment needs, then qualify the actual services and adapters.
Compare the implementations
| Backend | Stores / drives | Operational requirements |
|---|---|---|
| SQLite | Local execution state, events, journal, artifacts | Writable database and single-host session locking |
| PostgreSQL | Shared SQL state, events, journal, artifacts | Existing database, schema privileges, worker resume logic |
| Temporal | Workflow recovery; activities execute model/tool calls | Temporal, Redis events, shared artifact store, compatible workers |
SQLite is the default.
SQLite
from harnessx import SQLiteBackend
backend = await SQLiteBackend.connect("runtime.sqlite3") # construct and initialise
async with SQLiteBackend("runtime.sqlite3") as scoped: # initialise on entry, close on exit
...
await SQLiteBackend.connect(path) constructs and initialises in one call. Every backend is also an async context manager: async with backend: initialises on entry and closes on exit. A backend constructed directly is initialised by the runtime's start() or resume(), and initialisation is idempotent. SQL backends save transitions and their events transactionally. SQLite uses an OS session lock. The runtime does not schedule work after your process exits; an application must reopen and resume it.
PostgreSQL
Install harnessx[postgres]. The backend provisions its schema, tables,
constraints and migration history inside an existing database; it does not
create the server or its users.
import os
from harnessx import Agent, AgentConfig, AgentRuntime, PostgresBackend
backend = await PostgresBackend.connect(
os.environ["DATABASE_URL"], schema="harness_x", auto_migrate=True,
)
runtime = AgentRuntime(Agent(config=AgentConfig()), backend=backend)
connect(dsn, **kwargs) opens the pool and migrates in one call; async with PostgresBackend(dsn) as backend: does the same for a block. Pass the configured
Agent directly, with no registry. Call await runtime.start() for a new session.
In a replacement process build a compatible Agent and call await runtime.resume(session_id) before entering the runtime's context manager,
which would otherwise start a new session.
Migrations are transactionally locked and checksummed. With auto_migrate=False
initialization validates the schema instead of provisioning it. Leases are
renewed and stale writers fenced, but a replacement worker must still resume.
Crash and resume
examples/06-durability/durable_crash_recovery.py writes a report, dies with exit code 42, and lets
a second process finish the same run. The report is written once. Set
DATABASE_URL, or copy postgres.config.example.json to postgres.config.json
and pass --config with its path. The check command creates or validates the
schema first.
uv run python examples/06-durability/durable_crash_recovery.py crash # exits 42
uv run python examples/06-durability/durable_crash_recovery.py status # Tool invocations: 1
uv run python examples/06-durability/durable_crash_recovery.py resume # finishes the same run
- Process 1 asks for the report; PostgreSQL saves the run and the tool intent.
write_reportruns and writesreport.txt.- PostgreSQL saves the tool result, committed before anything else happens.
- Process 1 crashes with exit code 42.
- Process 2 resumes the same session; the run and its saved result come back.
- The result is reused, the answer is finished, and PostgreSQL marks the same run completed.
report.txtis unchanged.
What recovery does depends on where the crash landed.
| Crash boundary | What is known | Recovery behavior |
|---|---|---|
| Before dispatch, with intent saved | The tool has not been started by the runtime | Execute the prepared call |
| After dispatch is recorded, before the raw result commits | The tool may or may not have produced an effect | A manual call becomes uncertain; the run waits for reconciliation |
| After the raw result commits, as in this demo | The returned tool result is saved | Reuse it and retry unfinished result processing; do not invoke the tool again |
| After the run completes | The final answer is saved | resume returns None; inspect the saved result |
The hard case is a tool that sends an email or charges a card and then crashes before the result is saved. The external system and PostgreSQL are not one transaction, so the runtime cannot infer whether the effect happened. Check the external system, then resolve the pending call, retry it, or abort the run. See Approvals & recovery.
PostgreSQL here is storage, not a worker scheduler. Nothing restarts a crashed process for you.
Temporal
Install harnessx[temporal]. Bind versioned agent factories in an AgentRegistry on workers; submitting clients use AgentRef identifiers. Configure await TemporalBackend.connect(target, events=RedisEvents(...), artifact_store=S3ArtifactStore(...), task_queue=...). TemporalBackend.aclose() closes the Temporal client when connect() created it and leaves an injected client open.
Model and tool calls run in activities. Redis carries provisional events; it is not the source of execution truth. Expired event cursors or service outages can produce a gap. Query final results through the runtime.
Tools and files tied to a local host do not become portable because Temporal is used. recording=True needs a SQLite or PostgreSQL backend, so the Temporal path does not support it.
Activities classify failures with the provider's own rules, use the configured RetryPolicy.attempts, size their deadlines to the effective reply budget, and fall back when a history summary fails, as a local run does. This workflow scheduling is guarded by the harness-reliability-v2 patch marker, so a history recorded before it replays with its original command sequence.
Runtime configuration
RuntimeConfig carries storage_dir and max_session_duration_seconds. The runtime checkpoints at every phase boundary. Each checkpoint emits HookEvent.CHECKPOINT with session_id, run_id, status, and phase.
The runtime, backends, recorder, and their errors are also importable together from harnessx.durable.
Artifact and schema operations
PostgreSQL can store content-addressed artifact bytes or use a configured S3ArtifactStore. Artifact storage and external tool effects are not part of one global transaction.
Keep migration checksums stable. Test schema upgrades, recovery, worker compatibility, event reconnection, and actual adapters against your release configuration.