Tracing & evaluations

Evaluate an agent against a dataset with LangSmith's evaluate and aevaluate: tool selection, skill routing, delegation, and guardrails, either locally or in the LangSmith UI.

Everything on this page needs harnessx[langsmith]. Without it, harnessx.evals raises on import. A local run still calls the agent, so it costs model and tool spend; uv run python examples/07-quality/evaluating_with_datasets.py --offline needs no API key.

Enable tracing

Set LANGSMITH_API_KEY and add LangSmithExtension(project_name=...) to the agent. Without project_name it uploads to the LANGSMITH_PROJECT project, or to harnessx-agents when that is unset too. See Extensions & artifacts for what each span carries.

Tracing transmits prompts and tool payloads to LangSmith, so enable it only for data you intend to send. Trace delivery is best effort and is not an execution guard. Use the flight recorder when you need captured runtime boundaries for offline inspection.

Running an evaluation

import operator

from harnessx import Agent, AgentConfig, PermissionLevel
from harnessx.evals import build_example, contains_evaluator, evaluate_agent, tool_selection_evaluator

OPERATORS = {"+": operator.add, "-": operator.sub, "*": operator.mul, "/": operator.truediv}

def calculate(a: float, op: str, b: float) -> str:
    """Apply one arithmetic operator to two numbers.

    Args:
        a: The left number.
        op: One of + - * /.
        b: The right number.
    """
    return str(OPERATORS[op](a, b))

def make_agent(inputs):
    agent = Agent(config=AgentConfig(model="claude-sonnet-4-6"))
    agent.tools.register_tool(calculate, permission=PermissionLevel.ALLOW, replay_policy="safe")
    return agent

dataset = [
    build_example(
        inputs={"prompt": "What is 144 / 12?"},
        outputs={"expected_tools": ["calculate"], "contains_all": ["12"], "max_allowed_iterations": 3},
    )
]

summary = evaluate_agent(
    agent=make_agent,
    dataset=dataset,
    evaluators=[tool_selection_evaluator, contains_evaluator],
    experiment_prefix="math-agent-benchmark",
    offline=True,   # Disables LangSmith upload; the agent still calls its model.
)
print(f"Pass rate: {summary.pass_rate * 100:.1f}%")

Pass a built-in suite name ("tool_calling", "skills", "multi_agent", "guardrails", "memory", or "all") instead of a dataset to use the shipped examples. Experiments default to the prefix harnessx-eval; built-in datasets synced to LangSmith are prefixed harnessx-. default_evaluators() returns the standard evaluator set.

Evaluate from a running event loop

evaluate_agent_async(...) awaits the same evaluation from async code. It runs the synchronous runner in a worker thread, so pass a factory rather than an Agent bound to your loop. It cannot be cancelled once started and must not be nested inside another evaluation.

from harnessx.evals import evaluate_agent_async

async def benchmark():
    summary = await evaluate_agent_async(
        agent=make_agent,      # a factory, not an Agent bound to this loop
        dataset="tool_calling",
        offline=True,
    )
    return summary.pass_rate

Running evals from the CLI

# Scripted integration demonstration (langsmith extra, no API keys required)
uv run python examples/07-quality/evaluating_with_datasets.py --offline

# Run live model benchmarks without upload (model API key required)
python -m harnessx.evals.cli --suite tool_calling --offline

# Run against Anthropic and upload results + traces to LangSmith
export LANGSMITH_API_KEY="lsv2_pt_..."
python -m harnessx.evals.cli --suite skills --model claude-sonnet-4-6 --upload

# Run all benchmark suites with concurrency
python -m harnessx.evals.cli --suite all --concurrency 2

When uploaded, the CLI prints a clickable LangSmith URL to inspect row-level scores, side-by-side prompt diffs, and the complete nested execution tree for every turn.