Supplement to “Mercury 2.5: a cheap diffusion LLM with a hard ceiling”

Tables and figures behind Mercury 2.5: a cheap diffusion LLM with a hard ceiling. A section number (§) refers to the project’s design log.

The findings, by list

Likely from diffusion decoding or from how Inception serves Mercury:

Likely from diffusion decoding or from how Inception serves Mercury
Finding	What it means	Number	Source
A call is mostly the wait for the first token	Decoding is fast, but each call waits on a round trip before any text arrives, so the number of calls sets a run's time	generation 0.16 s of a 1.46 s call	§1, §7
Stalls hide behind keep-alives	Some calls never start, but the router sends a "still processing" signal at once, so a first-byte timeout never fires	330 of 50,924 calls silent until a 504 at ~120 s; median first content 1.15 s, p99 11.7 s	§69
Every call ran at temperature 1.0	The harness asked for low, near-deterministic sampling; the provider silently replaced it with its most random setting	asked for 0.0-0.3; Inception resets anything outside 0.5-1.0	§85
Temperature 0.5, tested	A lower setting the provider does accept, tried against the 1.0 every run had used	32 runs a side, not distinguishable, leaned worse	§82.2
Hidden reasoning	Mercury thinks in tokens the API never returns; only their count comes back	~96% of output tokens (95.9% over 93,312 calls)	§53, §82.1
More effort, fewer calls	Lowering the executor's reasoning effort did not save reasoning; it spread it over more calls	12.8 calls a run at high; 14.9, 18.4, 18.3 at medium, low, none	§53.2
Runaway reasoning	A rare call reasons for over a minute before a short answer; an output cap stops it	~1% of SciCode calls, ~62,000 hidden tokens, 77 s; a 16k cap cut model time 30%	§95, §95.1
Intermittent prompt cache	The provider's cache of the repeated start of each prompt engaged on some calls and not others	20% (SciCode night) to 67% (game project) of calls cached nothing	§95.2
Provider slowdown	The same prompts got slower over the month with no change on my side	first token 28-45% later, decode ~1,640 to ~1,310 tokens/s over September	§95, §114

Could come from any cheap model:

Could come from any cheap model
Finding	What it means	Number	Source
Optional tools unused	Tools offered for particular jobs, such as structural renames, were almost never called	4 calls in ~3,100 offers over 3,915 runs	§62
Reference notes unused	The tool for reading the reference notes written for the model was called once, while its catalogue rode on every call	1 read in 59,338 executor calls, ~420 tokens a call	§113
Rules decay over a step	An instruction in the system prompt is followed early in a step and ignored a few turns later	batching 6.0% of replies on turns 0-1, 1.5%, 0.8%, 1.2% later	§82
Reminder after the last message	Repeating the rule after the latest message, on every call, kept it followed	execute replies −21% (fixtures), −24% (django), calls −11% (django)	§106.2, §107.3
Tool calls fail on arguments	The model always named a real tool; failures were arguments that did not parse	plan call unparsable in 12% of runs; strict schemas 0 of 432 vs 9 of 524	§63, §63.1
Wrong bug reports believed	On tasks whose bug report is false, the model changes the correct code instead of saying so	misreport-discount, misreport-documented honest 0 of 3 each	§54.3, §96.4
Codex CLI	Inside Codex, Mercury printed its answer in the chat and never wrote the solution file	printed the code and wrote no file in 11 of 15 SciCode problems	§109.5

Settings that worked with Mercury

Settings that worked with Mercury
Setting	What it controls	What happened without it	Use
Temperature	How random the sampling is	reset to 1.0 outside 0.5-1.0	send 0.5-1.0, or nothing
Reasoning effort	How much hidden reasoning the model does before answering	lower effort added calls	high for the executor
max_tokens	The most output tokens, hidden reasoning included, a call may produce	rare calls reasoned for over a minute	16,000
Strict tool schemas	Whether the provider enforces each tool's argument format exactly	12% of plan calls unparsable	on for every tool
tool_choice	Whether a call must end in a tool call or may answer in plain text	32 of 188 forced final calls came back as prose (§74)	require the tool when a call must end in one
Where a rule sits	Where in the prompt an instruction is placed	system-prompt rules decayed	repeat after the last message
A cut on silent calls	How long a call may produce nothing before the harness cancels and re-sends it	stalls ran to ~120 s	30 s with no content

The release candidate’s final sweep

Every run had review on, which is now off by default (§124.10).

The release candidate's final sweep, 2 October
Each suite beside a same-day control or the task's own history
Suite	Runs	Release	Compared with
Seeded fixtures, 16 tasks × 3	48	accuracy 0.917; 38 perfect; 41 honest	each task's own history
SciCode, 15 problems × 3, search off	45	23/45 solved (7.7 of 15 a round)	earlier search-off rounds: 7.3 a round
django, 10 tasks × 3	30	15/30	mini-swe-agent, same model, same day: 18/30
GAIA, 127 questions	127	59/127	the 30 Sep harness, same day and search: 61/127

The same model in other agents, and other models in this harness

The same model in other agents, and other models in this harness
Mean ± SE over rounds; a dash: not run on that suite
Agent	django resolved	django $ a run	SciCode solved	SciCode $ a run
Mercury 2.5
My harness (30 Sep)	7.0 ± 0.0 of 10	$0.048	–	–
Goose	6.7 ± 0.7 of 10	$0.013	7.5 ± 1.5 of 15	$0.021
Codex CLI	6.0 ± 0.6 of 10	$0.008	4.0 ± 0.0 of 15	$0.018
mini-swe-agent	6.0 ± 0.4 of 10	$0.035	7.7 ± 0.3 of 15	$0.034
My harness	5.0 ± 0.6 of 10	$0.033	7.7 ± 0.9 of 15	$0.11
Aider	4.3 ± 0.7 of 10	$0.004	7.5 ± 0.5 of 15	$0.011
GLM 5.3 Flash
Aider	8.7 ± 0.3 of 10	$0.007	5.0 ± 0.0 of 15	$0.012
My harness	6.7 ± 0.7 of 10	$0.036	10.0 ± 0.0 of 15	$0.405
mini-swe-agent	5.7 ± 0.7 of 10	$0.033	–	–
My harness, GLM plans	–	–	8.5 ± 1.5 of 15	$0.255
Haiku 4.5
mini-swe-agent	6.0 ± 0.0 of 10	$0.284	–	–

Mean ± SE over rounds; a dash means the agent was not run on that suite. Mercury costs are at its launch price, and Goose and Codex are priced with their cached input (§122). Codex’s SciCode score reflects behaviour rather than ability: it printed code and wrote no file in 11 of 15 problems.

django, every configuration

Resolved of 30 (three runs of ten tasks), after §118 voided eleven contaminated wins; the as-scored count is in brackets where it differs. Intervals are 95% bootstrap intervals on the difference in resolve rate against the release harness (15 of 30).

Costs need two caveats (§122). Goose and Codex report their own token counts, priced here at OpenRouter’s list price with cached input at the cached price; at full list price they read $0.060 and $0.044 a django run. And Mercury’s provider cached 5-22% of the harness’s input before about 12:00 UTC on 2 October and 50-72% after, with no harness change. My harness’s rows and the §94 rows ran before that line, Goose, Codex and the GLM rows after it, so costs across the line are not like for like. Every run of my harness in these tables had review on; review is off by default since 6 October (§124.10).

django, every configuration
Resolved of 30 after §118's voids (as scored in brackets)
Agent	Resolved	$ a run	min a run	vs release harness
GLM
Aider	26 (27)	0.007	2.6	+0.37 [+0.13, +0.57]
My harness	20	0.036	5.9	+0.17 [−0.07, +0.40]
mini-swe-agent	17 (20)	0.033	4.6	+0.07 [−0.20, +0.30]
Mercury
Goose 1.52	20 (22)	0.013	1.6	+0.17 [−0.07, +0.40]
Codex CLI 0.160	18 (20)	0.008	2.4	+0.10 [−0.13, +0.33]
mini-swe-agent 2.4.6	18 and 18 (20, 19)	0.035	4.0	+0.10 [−0.13, +0.33]
My harness, 30 Sep	21	0.048	4.4
My harness, release	15	0.033	4.1
Aider	13	0.004	1.1
GLM, Mercury
My harness, GLM plans	23 (§94), 16 of 26 (§109)	0.028	4.8	+0.12 [−0.14, +0.37]
Haiku 4.5
mini-swe-agent	18	0.284	3.4

django tasks resolved against cost a run

django tasks resolved against wall time a run

Share of runs resolved, per django task and configuration

The release against its own 30 Sep version

The release resolved 15 of 30 where the 30 Sep version had resolved 21, all of the gap on four tasks. Re-run the same day in isolated workspaces (§120), the two came out level, and the release used about 40 percent fewer calls:

The release against its own 30 Sep version, on django
	Release	30 Sep harness
First rounds, ten tasks	15/30 (2 Oct)	21/30 (30 Sep)
First rounds, the 4 tasks that differed	6/12	12/12
Same 4 tasks re-run 5 Oct, isolated, same day	21/36	23/36
Calls a run, re-run	34	57
Cost a run, re-run	$0.0115	$0.0179

The 30 Sep first rounds were one unusually good set: same provider, same cache state before 2 October, and neither set of rounds read the fix from git history.

SciCode, 15 development problems

Problems solved (every sub-step passing), search off.

SciCode, 15 development problems
Problems solved, every sub-step passing, search off
Agent	Runs	Mean	$ a run
GLM
My harness	10, 10	10.0	0.405
Aider (no execution)	5	5.0	0.012
Mercury
My harness	6, 8, 9	7.7	0.110
mini-swe-agent	8, 7, 8	7.7	0.034
Aider (no execution)	8, 7	7.5	0.011
Goose	6, 9	7.5	0.021
Codex CLI	4	4.0	0.018

SciCode problems solved against cost a run

How GLM differs as a planner

From §119 and earlier sections; measurements, not pre-registered comparisons.

• django: GLM’s plans name the upstream fix in 51% of runs, Mercury’s in 0%. GLM planning with Mercury executing resolves 11964 5 of 5 (the harness with Mercury picks the wrong file in all 6 runs) and 11848 5 of 6 (no Mercury agent in 12 runs). GLM’s losses on 11999 and 11790 are recalled code colliding with the hidden tests.
• Plans: GLM’s declare ~1.8 times as many tests and are 3-4 times more consistent in step count for the same goal (§42).
• Research: on SciCode, 20 web searches, 53 fetches and 61 skill reads against none for Mercury, which asked 18 “Should I proceed?” questions instead (§77).
• Cost and time: 2-3 times Mercury’s cost on SciCode, about 3 times the time; its planning time is round trips at 5-7 s to first token (§76, §77).