tiny-rsi

Evolve a small agent harness and measure reward against test-time compute.

program.md         (autoresearch) outer Codex/Claude loop
harness.ts         seed harness: a Pi extension
harness-po-nn.ts   evolved harness
pi_agent.py        Harbor plumbing
eval.yaml          fitness environment

Each candidate runs eight fixed Terminal-Bench 2.1 tasks twice. Twelve runs measure reward against model cost per task and other test-time compute metrics. Uses Harbor for the benchmarks and Pi as the harness.

Fun notes

The candidates that the proposer generates are clearly more than just prompt engineering. It evolves observation compression, mutation and validation tracking, adaptive reasoning effort, a halfway-budget intervention, and early stopping after validation.

All candidates are developed and measured on the same eight tasks. a.k.a train on test :) The winning harness visibly specializes to them: its validation regex contains coqc, pdflatex, nginx -t, node ... extract.js, git commit, and other commands corresponding closely to the selected suite.

The obvious thing to do next is to run the final Pareto-frontier candidates once on unseen tasks. I suspect the gains won't be as pronounced.

The run script bootstraps a new repo with this README file excluded. An honest attempt at sandboxing wink wink.

How to run

./run.sh

Watch the jobs and candidate history from another terminal:

cd run
uv run harbor view jobs
git log --show-notes --oneline --decorate --tags='pi-*'

See the final plot in jobs/curve.svg.

Inspired by:

• Karpathy's autoresearch
• Meta-Harness Harbor pilot
• Pi Terminal Bench
• Lilian Weng, Harness Engineering for Self-Improvement