Flight recorder · HarnessX

The recorder preserves model and tool attempts in the durable runtime, in both their raw and processed form. Export a run as a portable .hx archive, verify it, and inspect the ordered timeline. Playback executes no models, handlers, bundled code, or services.

SDK usage

from harnessx import AgentRuntime, SQLiteBackend, ExportPolicy, IncidentRecorder

async with SQLiteBackend("runtime.db") as backend:
    async with AgentRuntime(agent, backend=backend, recording=True) as runtime:
        result = await runtime.run("Analyze overdue invoices")
        bundle = await runtime.export_incident(
            result.run_id,
            destination="incident.hx",  # Existing files are never overwritten.
            policy=ExportPolicy(include_payloads=True, include_artifacts=True),
        )

recorder = IncidentRecorder()
report = await recorder.verify(bundle)
playback = await recorder.playback(bundle)
for record in playback.records:
    print(record["seq"], record["kind"], record["attempt_id"])

Recording is off for new sessions unless you enable it, and the setting is saved with the session. On resume(), an omitted setting inherits that saved choice, and an incompatible one is rejected. Recording cannot be turned on retroactively for a run that was not recorded. Ordinary Agent.run() has no recorder option.

To export after a crash without resuming execution, open the store directly:

from harnessx import SQLiteBackend, export_incident, ExportPolicy

async with SQLiteBackend("runtime.db") as store:  # initialises on entry, closes on exit
    bundle = await export_incident(
        store, session_id, run_id, "crashed-run.hx",
        policy=ExportPolicy(include_payloads=True),
    )

Authorize access to the session in your application before exposing either export operation. A run is checked against its bound session, but knowing those IDs is not authentication. Opening an older store with automatic migration enabled upgrades its schema. Export never starts agent execution.

Recorded boundaries and recovery

Each private journal entry has a UUID, a contiguous per-run sequence, a format version, session/run/step/attempt IDs, a timestamp, an optional execution key, and a JSON payload. The journal is separate from the public RunEvent stream, so pre-middleware responses never reach it. It also keeps copies of the public events, including stream deltas.

Boundary	Meaning
`run.started`, `run.resumed`	Initial request/configuration and explicit resume history.
`model.started`	Provider-neutral request committed before provider invocation.
`model.response`, `model.processed`	Normalized response before and after middleware. Provider SDK transport objects excluded.
`model.completed`, `model.failed`, `model.interrupted`	Engine transition, captured failure, or interruption. Retries have distinct attempt IDs.
`tool.started`, `tool.dispatched`	Selected attempt, then admission after validation and permission checks, before the handler runs.
`tool.returned`, `tool.processed`	Raw handler result and post-middleware result.
`tool.failed`, `tool.interrupted`, `tool.resolved`	Exception/interruption or an explicit approval/recovery decision.
`checkpoint`, `event`, `command.failed`	Control state, extension/artifact bindings, public events, command failures.

Recording runs in the execution path, not in observation hooks. A failed recording commit blocks advancement at that boundary. Pending entries keep their IDs when storage recovers, so a boundary is never inserted twice. If storage stays unavailable, the run can raise RecordingError. The last durable boundary stays the recovery authority.

Raw tool results committed before failing middleware are reused on resume. An interrupted model call may be invoked again, with a new attempt ID. For what a crash between dispatch and return means for your replay policy, see durable recovery.

Disclosure and retention

The durable journal holds raw payloads. These can include sensitive prompts, tool results, provider reasoning fields, and data that middleware later removed. Enable recording only where storage access is controlled, and set your own retention: the journal has no automatic garbage collection.

Export defaults to metadata only: payloads are marked omitted and artifact bytes are excluded. That still reveals identifiers, timing, execution boundaries, and run status. Payloads and artifacts each require their own explicit flag. Credential-shaped dictionary keys, including those inside JSON strings, are redacted automatically, and the redaction markers are kept.

Application-specific redaction runs on a copy:

def redact(kind, payload):
    if kind == "run.started":
        return None  # Omit this payload.
    return payload

policy = ExportPolicy(include_payloads=True, redact=redact)

Key-based redaction does not find secrets in free text or binary artifacts, so review those under your own policy.

Export discovers artifact references after redaction and records each one as included, omitted, missing, or corrupt. Spill extension snapshots map their result IDs to artifact URIs. Only registered or captured artifacts are portable, so a filesystem path inside an application string is left behind. Files are published atomically, with private permissions, and without replacing an existing destination.

Verification and playback

The version-1 archive contains manifest.json, records.jsonl, and optional artifacts/<sha256> byte members. The manifest includes SDK and Python versions, the registered AgentRef, the final state at export, unresolved operations, sizes, and checksums. It records no dependency lockfiles or adapter code versions.

verify() returns a VerificationReport; malformed bundles return valid=False. playback() returns the verified manifest, ordered records, report, and included artifact bytes. It raises IncidentError for a malformed bundle. It never extracts archive members or imports bundled application code.

Valid means the archive structure, checksums, sequence, identities, event serialization, artifact catalog, and recorded final-state consistency all pass.

Complete additionally requires:

the initial run record,
unredacted payloads,
all referenced artifacts,
a terminal run state,
no outstanding model or tool attempts.

Complete describes captured runtime boundaries, not activity inside a handler. A fully recorded failed run can be complete. A metadata-only, redacted, live, interrupted, or unresolved recording stays valid while incomplete; read issues for the missing coverage.

BundleLimits defaults to 64 MiB for archive and expanded bytes, 32 MiB per member, 100,000 records, and 10,000 members. Oversized exports fail rather than truncate. Imports reject unknown member paths, symlinks, duplicate names or JSON keys, encrypted members, unsupported versions, mixed run IDs, and sequence gaps. Export builds its snapshot and archive in memory, so use it for bounded incidents, not large histories.

Run the example

The synthetic example needs no API key and no Datagol connector:

uv run python examples/07-quality/flight_recorder.py --output /tmp/invoice-incident.hx

It injects a model connection failure, computes a synthetic invoice total, applies output middleware, and destroys the temporary runtime database. It then verifies and plays back the bundle with no further model or tool calls.