Building the benchmark

Notes on how the benchmark is put together, written as it happens.

• Nov 3, 2026What each new agent costsWhere the benchmark stands after the first results, why every agent we add brings its own provider account, and how many dollars one more agent takes at today's size and at the size we are building toward.
• Oct 14, 2026Tasks a strong agent cannot skimFour runs of Claude Code on the Track N tasks, why the first three told us nothing about robustness, and what it took to get a task that needs more than a handful of tool calls.
• Oct 11, 2026Three harnesses pass conformanceThe conformance rerun for Claude Code, Pi and OpenCode, what each harness does when the injector breaks, and the three tasks the smoke run will use.
• Oct 10, 2026Does the fault reach the model?How faults get into a coding agent, why we test delivery before we test agents, and what the first conformance batch found.