TRACE Benchmark

TRACE — Task-Relevant Applied Constraint Execution: can a solver accomplish a task correctly while automatically honoring the preferences and constraints that matter for that task — even when those rules were stated once, in passing, and buried in a long prior conversation?

Blog post: nanonets.com/research/trace

Each sample is a long enterprise (Record-to-Report / finance) conversation with constraints sprinkled throughout at random — mid-sentence, embedded in routine back-and-forth, with no index that lists them. At the end the solver gets a task and must (1) identify which constraints are relevant, (2) honor them, and (3) solve the task with correct facts.

The dataset lives on the Hugging Face Hub: nanonets/trace-benchmark-dataset (two configs: solver = transcript + task, grader = rubric). This repo is the loading + evaluation harness and the place to submit your solver by pull request.

Install

pip install -r requirements.txt   # datasets

Quickstart

# 1. Run a solver over the `solver` config -> answers.jsonl
export SOLVER_BASE_URL=... SOLVER_MODEL=... SOLVER_API_KEY=...
python run_solver.py --solver solvers/example_solver --out submissions/example_solver/answers.jsonl

# 2. Grade with an OpenAI-compatible judge
export JUDGE_BASE_URL=... JUDGE_MODEL=... JUDGE_API_KEY=...
python grade.py --answers submissions/example_solver/answers.jsonl

Add --limit 3 to run_solver.py for a quick smoke test.

Scoring

Both the solver and the judge are any OpenAI-compatible /chat/completions endpoint, configured with environment variables (no keys in code). grade.py reports two numbers:

• task_pass_rate — fraction of tasks where the answer is fully correct and every applicable constraint is honored (per-question, all-or-nothing).
• rubric_pass_rate — fraction of individual rubric criteria passed, pooled across questions.

Each criterion is judged by meaning and returns 0 / 0.5 / 1; conditional criteria are gated by an antecedent check, and overridden criteria are excluded. See grade.py.

Submit your solver

1. Add solvers/<name>/solver.py implementing solve(transcript, question) -> str.
2. Generate submissions/<name>/answers.jsonl with run_solver.py.
3. Grade locally, then open a PR. See CONTRIBUTING.md.

License

Code: MIT (see LICENSE). Dataset: CC-BY-4.0 (see the Hugging Face dataset card).