News / Model Watch · Agent infrastructure
HarnessPro: Agents That Evolve Their Own Harness — Better and Cheaper
A Tsinghua/Zhipu-led team (Jie Tang co-corresponding author) has open-sourced HarnessPro, a label-free framework where an agent recursively improves its own harness — instructions, tools, runtime policies — with task performance and execution cost optimized together. The idea lands squarely on this site's beat: cost discipline as architecture. Instead of more exploration buying more quality at more cost, HarnessPro diagnoses each round of execution traces for both failure patterns and budget waste, and ships three Pareto-selected operating points — Lite, Medium, Heavy — rather than a single harness. We verified the repository (MIT, nine selected harnesses across three benchmarks plus evaluation scripts) and the arXiv abstract directly; the per-benchmark table figures below are attributed to the paper as relayed by the AINLP summary.
Verified directly: repo pzs19/HarnessPro (MIT, created 2026-09-24) ships nine harnesses — Terminal-Bench 2, SWE-bench Pro and ProgramBench in Lite/Medium/Heavy — with runtime policies, hooks and evaluation scripts (runtime/policy.json holds the reasoning-intensity and tool-budget numbers). The arXiv abstract states the headline: across benchmarks with GPT-5.5×Codex, heavy mode averages +2.0 percentage points pass rate at −26.1% per-task cost vs RHO, and the advantage holds with GLM-5.2×Claude Code. Evaluation during optimization uses LLM-synthesized tests — no ground-truth solutions, no official tests.
From the paper tables (attributed): Terminal-Bench 2 heavy 76.3%→80.8% at $1.09→$0.72 per task; SWE-bench Pro medium 79% at $1.26 vs RHO 80% at $2.68; GLM-5.2 heavy on TB2 78.0%→81.4% at $1.00→$0.62; test-synthesizer ablation −3.4 to −10pp.
Not in the release: the GLM-5.2 evolved configurations — the README quickstart shows gpt-5.5 commands, and the AINLP summary notes the GLM-5.2 evolution configs are not yet published. Benchmark data, task environments and model access are bring-your-own.
The facts, graded (Nov 2)
- The repo, verified by direct read.
pzs19/HarnessPro: MIT license, created 2026-09-24, nine selected harnesses with per-tier links, runtime policies and hooks, verification helpers, and full evaluation scripts for all three benchmarks. Quickstart requires Linux, Python 3.12/3.13, uv, Docker and Codex CLI (their runs used 0.147.0). A live demo page is linked from the README. - The paper, verified from the abstract. arXiv 2610.26770, "HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs" (Zhuoshi Pan, ZhiYuan Feng et al.; Jie Tang on the author list). Aggregate claim: GPT-5.5×Codex heavy mode averages +2.0pp pass rate and −26.1% per-task cost vs RHO across the three benchmarks; the same advantage holds under GLM-5.2×Claude Code.
- Per-benchmark figures (paper tables via AINLP, consistent with the abstract's aggregate): Terminal-Bench 2 heavy 76.3%→80.8%, cost $1.09→$0.72 (−34%); SWE-bench Pro medium 79% at $1.26 (−53% cost, −1pp) and heavy 80.7% at $1.97, against RHO's 80% at $2.68; Lite 72.3% at $0.80, beating the stock Codex setup's 59% at $1.88; GLM-5.2×Claude Code on TB2 heavy 78.0%→81.4%, cost $1.00→$0.62.
- The evaluation trick. No official tests or ground-truth answers during optimization: candidate harnesses are graded against tests synthesized once from the task description and initial traces. On the paper's 20 preliminary check cases, model-only trajectory judgment caught 5 of 12 real failures; generated tests caught 10 — failure recall 41.7%→83.3%. Removing the test synthesizer cost 3.4–10pp in pass rate.
- Where the money is saved. The evolved harness picks reasoning intensity and tool budgets per task, uses runtime hooks to steer exploration vs wrap-up (in one TB2 medium profile: merge-parallel-calls reminders at 14 tool calls, pivot-to-verification at 26, hard stop after 44), appends nudges to tool outputs to preserve cache-friendly prefixes, and folds/compresses verbose logs. The ablation is the punchline: porting any single mechanism onto RHO makes things worse (dynamic routing alone 76%, budget hooks alone 75%, uniformly low reasoning 59%) — the gains come from the composition.
- Iteration behavior. TB2 heavy round 1: 74.6% at $0.91; round 3: 80.8% at $0.72. SWE-bench Pro medium only improved both objectives simultaneously by round 3 — quality-first and cost-first rounds can each temporarily worsen the other axis.
Verification on Nov 2, 2026: GitHub API + raw README read directly; arXiv abstract page read directly. Per-benchmark table numbers are the paper's own (as summarized by AINLP, a third-party media account — secondary source, consistent with the primary abstract's aggregates); no benchmark reproduction was performed.
Context: the same beat as Super-agent — spend discipline as design
This is the third entry this week on the same underlying story: agent cost engineering is becoming the differentiator. DeepSeek's Super-agent writeup (our full translation) put batching, event filtering and subscription pass-through into the system architecture; Vercel's CEO framed the human job as capital allocation; HarnessPro now automates a slice of that allocation at the harness layer — choosing reasoning intensity, tool budgets and stopping points per task, with cache-friendly prompt prefixes preserved. The Chinese-institution lineage matters here: Tsinghua and Zhipu (the GLM vendor) building cost-aware agent infrastructure, with GLM-5.2 as one of the two tested stacks.
For buyers, the Medium-tier numbers are the practical headline: on SWE-bench Pro, 79% at $1.26 vs RHO's 80% at $2.68 — one point of pass rate for half the bill. That is the same arithmetic we run on model pricing every week (DeepSeek V4.1-Flash off-peak at $0.22/1M, cache-heavy agent designs in the cheapest-API guide), now applied to the harness itself.
One honest caveat: the paper's comparisons are against RHO and the stock harness on three specific benchmarks, with per-task costs dominated by frontier-model pricing. Treat the percentages as benchmark-local, and the pattern — optimize both objectives jointly, evaluate with synthesized tests, select on the Pareto frontier — as the transferable part.
What this means for builders
The release is directly usable if you run Codex-style agents on the covered benchmarks: clone, uv sync, point CODEX_CONFIG at your provider and pick a tier. The harnesses are plain artifacts — instructions plus a runtime policy JSON with the reasoning-intensity and tool-budget numbers in the open — so even outside the benchmarks, they double as a reference for what a cost-tuned policy looks like at three price points.
The transferable lessons, no repo required: evaluate self-modification with fixed synthesized tests instead of model vibes (the 41.7%→83.3% failure-recall gap is the whole argument); never judge a change on quality alone — the SWE-Pro medium round-2/round-3 trajectory shows cost can lag then catch up; and do not cherry-pick mechanisms — every single-mechanism ablation made RHO worse while the composed system beat it.
Watch for: the GLM-5.2 evolved configs landing in the repo (the missing half for Chinese-model stacks); third-party reproductions of the table numbers; and any harness-tier artifacts emerging for other agent runtimes.
Access from outside China — and the usual disclaimer
This site tracks Chinese model APIs for international builders. The GLM side of this work runs on the same model family covered in the GLM API guide; the cost arithmetic belongs in the comparison table and the cheapest-API guide.
Independence disclaimer: ChinaModelAPI is an independent information site, not affiliated with the HarnessPro authors, Tsinghua, Zhipu or any vendor. Repo and abstract facts are from direct reads; table figures are the paper's own, relayed by a secondary summary; nothing here is a routing, availability or pricing promise.
Primary sources
- GitHub — pzs19/HarnessPro (MIT; 9 harnesses, runtime policies, eval scripts; README read directly)
- arXiv 2610.26770 — "HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs" (abstract read directly)
- AINLP (NLPer) summary, Nov 2026 — the per-benchmark table figures and ablation details, relayed
- 站内对照：DeepSeek Super-agent 全文译丛 · 资本配置访谈观察 · GLM API 指南
FAQ (2026)
What is HarnessPro?
An MIT-licensed framework from a Tsinghua/Zhipu-led team where agents recursively improve their own harness — instructions, tools, runtime policies — optimizing task performance and execution cost together, evaluated with LLM-synthesized tests instead of ground truth.
How much does it save?
Per the arXiv abstract, heavy mode averages +2.0pp pass rate at −26.1% per-task cost vs RHO (GPT-5.5×Codex). Per the paper tables: TB2 heavy $0.72 vs $1.09; SWE-Pro medium $1.26 for 79% vs RHO's $2.68 for 80%.
Does it work with GLM models?
The paper evaluates GLM-5.2×Claude Code with the advantage holding (TB2 heavy 78.0%→81.4% at $0.62). However, the GLM-5.2 evolved harness configs are not yet in the released repo — the release ships the GPT-5.5 evolution artifacts.
What are Lite, Medium, Heavy?
Three Pareto-selected operating points per benchmark: Lite favors cost, Medium balances, Heavy favors performance. Nine harnesses total across Terminal-Bench 2, SWE-bench Pro and ProgramBench, each a plain instructions-plus-policy artifact.
How are candidates evaluated without ground truth?
Tests are synthesized once from the task description and initial trajectories, then held fixed for all candidates. Model-only judgment caught 5 of 12 real failures in the paper's check set; synthesized tests caught 10 (41.7%→83.3% failure recall).
Can I just port one mechanism onto my setup?
Probably not. The ablations show single mechanisms grafted onto RHO make it worse (routing alone 76%, hooks alone 75%, low reasoning 59% vs 80%); the gains come from the composed system.