Every stack ran on Claude Sonnet 5.5 and got the same system prompts, tools and user prompts. This page sums up what each one added or cost, with HarnessX highlighted.
Download the five-page brief (PDF) ↓Benchmark code, tasks and graders on GitHub ↗
How it was run
A recording proxy sat between each stack and the API and logged every call. Cost, latency, retries and token counts all come from that log, so no stack's own reporting is used. Tasks were graded by deterministic checks, plus an Opus judge for the one open-ended writing task.
| Tier | What it tests | What the agent has to do | Tasks |
|---|---|---|---|
| A | Tool use | Chain, fan out and paginate calls to simple tools, and keep state across them | 5 |
| B | Coding | Fix a bug, add a feature, refactor, debug, and implement from a spec, in a real sandbox | 5 |
| C | Corpus research | Read a 40-document corpus, join facts across files, spot contradictions and write a brief | 3 |
| D | Long context | Cross-reference 30 modules or 120 patches without losing the thread | 2 |
| E | Strict JSON | Return exactly the requested JSON from messy inputs and tool output | 3 |
| F | Adversarial tools | Transient errors, huge outputs, misleading results and prompt injection | 4 |
Each stack ran in two setups. Tuned is how a team would deploy it, with explicit caps, retries and caching. Defaults is the framework as it ships. The capability numbers below use tuned.
A separate fault suite had the proxy inject rate limits, server errors, overloads and timeouts. Each run was scored on whether it recovered, failed cleanly, gave a wrong answer, hung or crashed. That suite ran on Claude Haiku 4.5, because it tests the retry path rather than the model.
Capability
HarnessX scored 97%, tied with pi for first. The confidence intervals overlap a lot, so the top five can't really be told apart. Deep Agents came last at 87%.
The breakdown by tier shows where each stack gained and lost points.
HarnessX passed every run in five of the six tiers, including all 15 tool-use runs and all the long-context runs. HarnessX and pi were the only stacks with no tier below 83%.
Its one weak tier was corpus research, at 83% (5 of 6). The miss was the contradictions task: HarnessX found every planted contradiction but also listed a few extra ones, which the grader counted as false positives. The corpus seems to contain some real inconsistencies nobody planted, so that task is being audited. The raw baseline, CrewAI and TrueForge scored 67% in this tier.
HarnessX, task by task
Each tile is one task, showing runs passed out of runs, on the tuned setup.
21 of the 22 tasks passed every run. The only miss is the contradictions task above, wrong in 1 of 2 runs.
Cost and speed
| Sonnet 5.5, tuned | HarnessX | Raw SDK baseline |
|---|---|---|
| Capability score | 97% | 93% |
| Cost per successful run | $0.058 | $0.171 |
| Median cost per run | $0.0309 | $0.0322 |
| Median wall time per run | 12.7 s | 8.7 s |
| Framework overhead | 1% | 1% |
HarnessX had the second-lowest cost per successful run, about a third of what the baseline and CrewAI spent. Its framework overhead was 1%, tied for lowest, so the extra wall time is time spent waiting on the model. On short tasks each model call takes longer, partly because HarnessX spends turns on planning and sends a larger prompt.
Where tokens and time go
Median tokens per run, by tier:
Wall time per run, by tier, split into time on model calls, time in tools, framework overhead, and other time outside the model (such as native file access):
Reliability
This chart shows the share of fault scenarios in which the run still ended with a correct answer. The fault suite ran on Claude Haiku 4.5.
On the tuned setup, HarnessX recovered from all 4 fault scenarios, tied with CrewAI. As shipped, it recovered from 10 of 12 and failed cleanly on the other 2, the best result of the six stacks. TrueForge recovered from 3 of 12 as shipped, and one run hung.
Takeaways
- Capability tied for first, at 97%.
- Second-lowest cost per successful run, with 1% framework overhead.
- The best fault recovery on both setups, or tied for it.
Benchmark run on 10 October 2026. Every stack used Claude Sonnet 5.5 for capability and Claude Haiku 4.5 for the fault suite.