# Flight recorder

Export a portable incident and inspect it offline.

The recorder preserves model and tool attempts in the durable runtime, in both their
raw and processed form. Export a run as a portable `.hx` archive, verify it, and
inspect the ordered timeline. Playback executes no models, handlers, bundled code, or
services.

## SDK usage

```python
from harnessx import AgentRuntime, SQLiteBackend, ExportPolicy, IncidentRecorder

async with SQLiteBackend("runtime.db") as backend:
    async with AgentRuntime(agent, backend=backend, recording=True) as runtime:
        result = await runtime.run("Analyze overdue invoices")
        bundle = await runtime.export_incident(
            result.run_id,
            destination="incident.hx",  # Existing files are never overwritten.
            policy=ExportPolicy(include_payloads=True, include_artifacts=True),
        )

recorder = IncidentRecorder()
report = await recorder.verify(bundle)
playback = await recorder.playback(bundle)
for record in playback.records:
    print(record["seq"], record["kind"], record["attempt_id"])
```

Recording is off for new sessions unless you enable it, and the setting is saved with
the session. On `resume()`, an omitted setting inherits that saved choice, and an
incompatible one is rejected. Recording cannot be turned on retroactively for a run
that was not recorded. Ordinary `Agent.run()` has no recorder option.

To export after a crash **without resuming execution**, open the store directly:

```python
from harnessx import SQLiteBackend, export_incident, ExportPolicy

async with SQLiteBackend("runtime.db") as store:  # initialises on entry, closes on exit
    bundle = await export_incident(
        store, session_id, run_id, "crashed-run.hx",
        policy=ExportPolicy(include_payloads=True),
    )
```

Authorize access to the session in your application before exposing either export
operation. A run is checked against its bound session, but knowing those IDs is not
authentication. Opening an older store with automatic migration enabled upgrades its
schema. Export never starts agent execution.

## Recorded boundaries and recovery

Each private journal entry has a UUID, a contiguous per-run sequence, a format
version, session/run/step/attempt IDs, a timestamp, an optional execution key, and a
JSON payload. The journal is separate from the public `RunEvent` stream, so
pre-middleware responses never reach it. It also keeps copies of the public events,
including stream deltas.

| Boundary | Meaning |
|---|---|
| `run.started`, `run.resumed` | Initial request/configuration and explicit resume history. |
| `model.started` | Provider-neutral request committed before provider invocation. |
| `model.response`, `model.processed` | Normalized response before and after middleware. Provider SDK transport objects excluded. |
| `model.completed`, `model.failed`, `model.interrupted` | Engine transition, captured failure, or interruption. Retries have distinct attempt IDs. |
| `tool.started`, `tool.dispatched` | Selected attempt, then admission after validation and permission checks, before the handler runs. |
| `tool.returned`, `tool.processed` | Raw handler result and post-middleware result. |
| `tool.failed`, `tool.interrupted`, `tool.resolved` | Exception/interruption or an explicit approval/recovery decision. |
| `checkpoint`, `event`, `command.failed` | Control state, extension/artifact bindings, public events, command failures. |

Recording runs in the execution path, not in observation hooks. A failed recording
commit blocks advancement at that boundary. Pending entries keep their IDs when storage
recovers, so a boundary is never inserted twice. If storage stays unavailable, the run
can raise `RecordingError`. The last durable boundary stays the recovery authority.

Raw tool results committed before failing middleware are reused on resume. An
interrupted model call may be invoked again, with a new attempt ID. For what a crash
between dispatch and return means for your replay policy, see
[durable recovery](../recovery/).

## Disclosure and retention

The durable journal holds raw payloads. These can include sensitive prompts, tool
results, provider reasoning fields, and data that middleware later removed. Enable
recording only where storage access is controlled, and set your own retention: the
journal has no automatic garbage collection.

Export defaults to **metadata only**: payloads are marked omitted and artifact bytes
are excluded. That still reveals identifiers, timing, execution boundaries, and run
status. Payloads and artifacts each require their own explicit flag. Credential-shaped
dictionary keys, including those inside JSON strings, are redacted automatically, and
the redaction markers are kept.

Application-specific redaction runs on a copy:

```python
def redact(kind, payload):
    if kind == "run.started":
        return None  # Omit this payload.
    return payload

policy = ExportPolicy(include_payloads=True, redact=redact)
```

Key-based redaction does not find secrets in free text or binary artifacts, so review
those under your own policy.

Export discovers artifact references **after** redaction and records each one as
included, omitted, missing, or corrupt. Spill extension snapshots map their result IDs
to artifact URIs. Only registered or captured artifacts are portable, so a filesystem
path inside an application string is left behind. Files are published atomically, with
private permissions, and without replacing an existing destination.

## Verification and playback

The version-1 archive contains `manifest.json`, `records.jsonl`, and optional
`artifacts/<sha256>` byte members. The manifest includes SDK and Python versions, the
registered `AgentRef`, the final state at export, unresolved operations, sizes, and
checksums. It records no dependency lockfiles or adapter code versions.

`verify()` returns a `VerificationReport`; malformed bundles return `valid=False`.
`playback()` returns the verified manifest, ordered records, report, and included
artifact bytes. It raises `IncidentError` for a malformed bundle. It never extracts
archive members or imports bundled application code.

**Valid** means the archive structure, checksums, sequence, identities, event
serialization, artifact catalog, and recorded final-state consistency all pass.

**Complete** additionally requires:

* the initial run record,
* unredacted payloads,
* all referenced artifacts,
* a terminal run state,
* no outstanding model or tool attempts.

Complete describes captured runtime boundaries, not activity inside a handler. A fully
recorded failed run can be complete. A metadata-only, redacted, live, interrupted, or
unresolved recording stays valid while incomplete; read `issues` for the missing
coverage.

`BundleLimits` defaults to 64 MiB for archive and expanded bytes, 32 MiB per member,
100,000 records, and 10,000 members. Oversized exports fail rather than truncate.
Imports reject unknown member paths, symlinks, duplicate names or JSON keys, encrypted
members, unsupported versions, mixed run IDs, and sequence gaps. Export builds its
snapshot and archive in memory, so use it for bounded incidents, not large histories.

## Run the example

The synthetic example needs no API key and no Datagol connector:

```sh
uv run python examples/07-quality/flight_recorder.py --output /tmp/invoice-incident.hx
```

It injects a model connection failure, computes a synthetic invoice total, applies
output middleware, and destroys the temporary runtime database. It then verifies and
plays back the bundle with no further model or tool calls.