Supplement to “Take the work away from a weak model”

Tables and figures behind What my harness could and couldn’t do for Mercury 2.5. A section number (§) refers to the project’s design log.

Changes adopted

Changes that passed their rule
Each against its own same-sweep control
Change	What it does	What moved	Cost
The harness runs a step's tests after each edit turn	When the model edits code without running a test, the harness runs the step's test itself and attaches the result, saving a round trip	model time −31%, cost −40%	none
Every tool schema strict	Each tool's argument schema is sent with strict: true, so the provider rejects malformed arguments such as a list sent as a double-escaped string	malformed calls 9 of 524 to 0 of 432	planner calls ~30% slower
Executor reasoning at high effort	The executor's hidden reasoning is set to high rather than medium, low or off	12.8 calls a run against 18.3 at none	none
A 30 s cut on calls with no content	A call that has produced no content after 30 seconds is canceled and sent again, instead of waiting about two minutes for the provider's timeout	~29,700 s of stalls saved for ~3,000 s re-asked	none
A 16,000-token output cap	Every Mercury call carries max_tokens of 16,000, which stops rare calls that reason for minutes	SciCode model time −30%	none
Batched, sectioned file reads	One read can fetch several line ranges or named functions, a too-narrow window is widened, and re-reading an unchanged file returns one line	file reads −40%	none
A type check after each edit	After each edit the harness runs the type checker on the edited files and attaches any new errors	runs ending with new type errors 6 of 8 to 1 of 8	none
The batching rule repeated after the last message	The step's goal, the rule to batch reads and edits, and the turns left are appended after the last message on every executor call, never saved into the history	execute replies −21%	none
The reviewer handed the checks already run	The reviewer is given the results of the tests and checks the run already made, instead of re-running them	review calls −58%	none
A check that a rename plan covers every caller	Before a rename plan is accepted, the harness finds every file that calls the renamed code and sends the plan back if one is missing	plans missing a caller 6 of 40 to 0 of 40	none
A reproduction snippet in the plan	The planner writes a short snippet that should show the reported bug; the harness runs it and sends the plan back if the bug does not reproduce	wrong-report honesty 37% to 53% (not significant)	+0.13 calls
Skill triggers matched to the step's own files	Reference notes are attached to a step only when their keywords match that step's own files and text, not the whole goal or the verify command	skill tokens −17% (replay)	none
A gate on edits that contradict the run's evidence	Holds, once each, an edit made after the plan's reproduction already passed, and an edit that removes or weakens an existing test	nothing measurable either way	none

Changes not adopted

Changes that did not
Each against its own same-sweep control
Change	What it does	Result
A test-first gate	Holds a step's first edit to source code until a test has been run, so a failing test exists before the fix	red-first steps 3% to 48%; accuracy −3.9 points
Symptom-first planning	The planner is told to describe and confirm the reported symptom before naming a fix	honest 83% against 90%; the first, sequential read of +12 points was provider drift
Planner drafts in prose	The planner writes its draft plan in full sentences, with code sketches, instead of terse notes	named tests 1.68 to 1.71; +1.8 calls
A goal loop judged by a second model	After a run, a second model judges whether the goal was met and, if not, re-runs it with a refined goal	worse on all 3 django tasks
The batching rule moved to the end of the system prompt	The batch-your-reads-and-edits rule, with worked examples, moved from the middle to the end of the system prompt	no change
Research first, with a findings hold	Before planning, the planner looks up the installed libraries' APIs, then reference notes, then the web, and records its findings; the plan is held once if they decide nothing	+1.7 SciCode problems, then not replicated
A numeric self-check	The executor is asked to run its function on a case with a known answer before finishing; a gate holds the finish after an untested Python edit	+13 sub-steps, then −2
A stronger model auditing the plan	GLM reads Mercury's plan against the goal in one call and may correct it before execution	dropped after repeats
Requiring the planner to run code	The planner gets a Python tool in a throwaway copy of the workspace, and its first plan is held until it has run something	+2.0 calls, honesty n.s.
Structural edit tools and a plan block for renames	An ast-grep search-and-rewrite tool for renames, and planner instructions to plan renames as one structural step	0 calls in 60 runs
Search-and-replace edit blocks	A second batch-edit format: the model writes SEARCH and REPLACE text blocks instead of line edits	used for 55% of edits; calls and edit turns unchanged
Web-research aids	A web-research skill, a nudge to open a result instead of searching again, and a narrower trigger for the documentation notes	the target moved 11%, the rule needed 30%
A tracer for probing code	A tool that records a function's execution line by line in one call, to replace repeated print-and-run probing	0 calls in 15 runs
Forcing the diagnosing skill	Always attaching the bug-diagnosis notes to the first step of a bug-fix task (tested by replaying recorded runs)	no more honest runs (replay of 1,012)
Temperature 0.5	Asking for temperature 0.5, inside the range Inception accepts, instead of the 1.0 every call had been running at	not distinguishable, leaned worse
A repair round for weak tests	When an audit finds the run's new tests would pass on the old code, one extra round shows the model its diff and asks for better tests	honest +6 points, n.s., at +4 calls
Clarify questions shown the code first	Before asking the person anything, the question step sees the code's symbols, the README and how tests run, and is told to ask only what the code cannot answer	−37.5% code-answerable questions against a 40% rule (later turned on by choice)
Skills at all	Reference notes on tools and practices (uv, Ruff, testing, debugging) attached to prompts by keyword	no measurable difference in three tests (kept by choice)

Each feature test as an arrow from the arm without the feature to the arm with it

Each test from the two tables, from the arm without the feature to the arm with it: median wall time a run and mean accuracy. Positions differ between sweeps (different tasks and days), so read the arrows.

Each feature test against its same-sweep control: change in cost a run against change in accuracy, with 95% intervals. The adopted changes lean cheaper and none moved accuracy clearly.

Mistakes the method caught

Each would have changed a decision.

Mistakes the method caught
Each would have changed a decision
What looked true	What it was
102 runs wrong while green	65 were one fixture that asked for one thing and tested another
The weakest task at 49%	42 unscored plan-only runs counted as zeros
django at $0.096 a run	the totals event counted twice; $0.048
A stronger planner resolving 8 of 10	two of its four gains fetched the upstream fix from the web
GAIA down by a third	the search engines behind it blocked, for every harness version
edit blocks saving nothing	two other mechanisms did not know they were edits (0 of 23 checked)
Other agents resolving 20-22 of 30	eleven wins read the fix from git history or the answer key

How the setup with Claude changed

Date	Change	Why
19 Sep	build-the-tool and growing-docs-hygiene skills	The design log was too long to read, and the same analyses were being redone by hand
23 Sep	A session audit; CLAUDE.md cut from 148 always-loaded lines to 47, the rest loading by path; git hooks enforced; project commands moved into the repo	Long context was most of the cost
24 Sep	verify-before-claiming, debug-root-cause, design-before-building, plan-then-execute	Mercury’s two failures, in Claude
26 Sep	A global CLAUDE.md, with the testing and Python rules taken from this repo	The rules applied to every project
1 Oct	humanizer	Drafts that read as written by a model
2 Oct	Several sessions in parallel, passing messages, with a code freeze for the release sweep	One session could not run the benchmarks and keep building

What the Claude Code sessions cost

From harness sessions report, which reads Claude Code’s own transcripts and makes no model call, over 220 sessions and 16,238 API responses, run 4 October:

Measure	Value
Share of cost that is reading the cache	68%
Turns run at more than 400K tokens of context	48% of turns, 71% of cost
Estimated cost with context compacted at 200K	41-51% of actual
Share of context, by tokens times turns carried, that is shell output	74%
Whole-suite test runs, sequential and parallel	835 (770 min) and 220 (135 min)

Running other agents safely

To compare my harness with other agents, I ran four open agents locally on the same benchmarks. Each had its own workspace, a spending cap and a list of denied websites, but only Codex ran under an operating-system sandbox, and its mode limits writes, not reads. mini-swe-agent, in runs with three different models, listed my home folder and sent the listing to its model provider as part of its context; nothing was opened, copied or uploaded, and none of these logs is published. Agents with a shell also found later commits in django’s git history and read the fixes, and eleven wins were voided (§118). The setup procedure was written for my own agent, not for four others added late in the project. When the scope of a project grows, the safety assumptions need checking again, and an agent that runs shell commands should start in a sandbox whatever it is for. Every external-agent run since 5 October is sandboxed.

Where the numbers come from

Method §45.4, §46, §56, §61, §112. After-edit check §99. Strict schemas §63.1. Reasoning §53.2. Stall cut §69. Output cap §95.1. Read assists and type check §105.2. Reminder §106.2. Reviewer §108.2. Call-site check §66.1. Reproduction §54.2. Skill triggers §117. Premise gate §57.1. TDD gate §49.1. Prose drafts §60.1. Goal loop §72. Rule placement §82.1. Findings hold §87.2, §88.6. Numeric check §89.4, §89.6. Plan audit §77.2. Planner probe §54.1. Structural tools §62.1. Edit blocks §107.3. Web research §116.1. Tracer §43.1. Diagnosing skill §64. Temperature §82.2. Weak-test repair §46.8. Clarify §92, §101. Skills §117.2. Stronger models §127-127.7, §127.9, §129.4. Mistakes §47, FINDINGS row 15, §107.1, §79.1, §112.1, §106.1, §118. Session numbers harness sessions report, 4 Oct; the earlier audit §50. UI tests §27-28. The watchdog §51. Agent exposure: item 2.8 of TODO.md in the repository.