Seeded tool-call failures injected into unmodified coding-agent harnesses. Which harness still finishes the task, what recovery costs, and whether it reports the outcome honestly.
Detailed results
Faulted Success per board. Boards differ in where faults enter and in which fault types they compare, so columns are not added up or compared across boards. Open a board for its full metrics and rank bands.
| Harness | Model | Track S | Track N fixed model |
|---|---|---|---|
| claude-code2.1.289 | claude-sonnet-5-5(medium) | 58.3% ± 22.2% | 83.3% ± 16.7% |
| codex0.163.0 | claude-sonnet-5-5(medium) | 69.4% ± 20.8% | 83.3% ± 15.3% |
| fx0.0.13 | anthropic/claude-sonnet-5.5(medium) | 69.4% ± 20.8% | not measurableNo interception point for fx built-in tools (D-0027, V-22), so Track N faults cannot reach it. |
| opencode1.18.35 | claude-sonnet-5-5(medium) | 69.4% ± 20.8% | 80.6% ± 15.3% |
| pi1.0.3 | claude-sonnet-5-5(medium) | 66.7% ± 20.8% | 72.2% ± 20.8% |
Metrics
pass^1 under the moderate profile (15% of eligible tool calls fail, at most 6 per trial). The headline and the default sort.
pass^1 with no faults. The baseline each config is compared against.
Robustness Retention: Faulted Success divided by Clean, over tasks the config solves without faults. Never the primary sort.
Probability that 3 attempts on the same task all succeed under moderate faults (unbiased estimator, n = 5).
Among failed faulted trials, the share where the agent reported success anyway.
Share of plausible-but-wrong outputs whose marker token ends up in the final artifacts.
Model spend divided by successful trials under moderate faults.
Configs whose 95% intervals for Faulted Success overlap share a band. Bands replace ordinal ranks.