Smoke runs, not benchmark results. These trials check the pipeline. They carry no ranking and support no comparison between configs or harnesses.
Smoke runs Each run checked the path from a trial to the analysis summary to this page: Harbor ran each trial in a container, the injector logged its decisions, and the analysis read the job directories and wrote the summary shown here. Newest first.
Oct 30, 2026 track-n-codex 10 trials, 8 faults applied, between 5 and 26 eligible tool calls per trial. Source: results/smoke/2026-10-30-track-n-codex/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 5 80.0% $1.5661 $0.3132 0 moderate 5 80.0% $1.2472 $0.2494 8
Per trial Task Profile Reward Cost Eligible calls Completion status n-csv-exportn-csv-export__bu9hjdm clean 0.00 $0.1658 5 success n-ledger-fixn-ledger-fix__3BDKBe2 clean 1.00 $0.4758 26 success n-make-buildn-make-build__Ybu7YaJ clean 1.00 $0.3531 21 success n-port-conflictn-port-conflict__gAcy3NU clean 1.00 $0.1649 10 success n-timesheet-payrolln-timesheet-payroll__HHt4Phw clean 1.00 $0.4064 18 success n-csv-exportn-csv-export__HdJ5dq8 moderate 1.00 $0.1496 5 success n-ledger-fixn-ledger-fix__weAPYw7 moderate 1.00 $0.4164 22 success n-make-buildn-make-build__xJdNmau moderate 0.00 $0.3511 16 success n-port-conflictn-port-conflict__eVQ9HKm moderate 1.00 $0.1481 10 success n-timesheet-payrolln-timesheet-payroll__BvgTCRN moderate 1.00 $0.1821 5 success
Oct 30, 2026 track-s-codex 10 trials, 25 faults applied, between 10 and 52 eligible tool calls per trial. Source: results/smoke/2026-10-30-track-s-codex/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 5 100.0% $0.4994 $0.0999 0 moderate 5 80.0% $0.5663 $0.1133 25
Per trial Task Profile Reward Cost Eligible calls Completion status s-cycle-counts-cycle-count__BPTfHhA clean 1.00 $0.1546 38 success s-error-catalogs-error-catalog__oRs7sLB clean 1.00 $0.0656 10 success s-followupss-followups__tKXfc55 clean 1.00 $0.0667 12 success s-p1-reports-p1-report__om9Ln4J clean 1.00 $0.0766 22 success s-release-trains-release-train__SWkqNw9 clean 1.00 $0.1359 45 success s-cycle-counts-cycle-count__AK9JidE moderate 1.00 $0.1614 38 success s-error-catalogs-error-catalog__rjSNh6G moderate 0.00 $0.0691 11 success s-followupss-followups__fKtDd28 moderate 1.00 $0.0823 17 success s-p1-reports-p1-report__4CZwPQQ moderate 1.00 $0.0948 29 success s-release-trains-release-train__dUeTvqp moderate 1.00 $0.1587 52 success
Oct 21, 2026 track-n-fxh 10 trials, 22 faults applied, between 9 and 71 eligible tool calls per trial. Source: results/smoke/2026-10-21-track-n-fxh/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 5 100.0% $2.2090 $0.4418 0 moderate 5 60.0% $2.6543 $0.5309 22
Per trial Task Profile Reward Cost Eligible calls Completion status n-csv-exportn-csv-export__zhCUBgk clean 1.00 $0.3398 14 success n-ledger-fixn-ledger-fix__LaXE99i clean 1.00 $0.7823 58 success n-make-buildn-make-build__8yQCvda clean 1.00 $0.6604 26 success n-port-conflictn-port-conflict__XXE8vVZ clean 1.00 $0.1698 11 success n-timesheet-payrolln-timesheet-payroll__a6FChJy clean 1.00 $0.2566 9 success n-csv-exportn-csv-export__Hy66mQW moderate 0.00 $0.3939 22 success n-ledger-fixn-ledger-fix__AE7Sbts moderate 0.00 $1.1831 71 success n-make-buildn-make-build__Wg2z7WG moderate 1.00 $0.6391 25 success n-port-conflictn-port-conflict__KQaHb2x moderate 1.00 $0.1672 16 success n-timesheet-payrolln-timesheet-payroll__m6BDUEw moderate 1.00 $0.2710 14 success
Oct 19, 2026 track-n-pi-oc 20 trials, 35 faults applied, between 4 and 37 eligible tool calls per trial. Source: results/smoke/2026-10-19-track-n-pi-oc/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 10 100.0% $2.4158 $0.2416 0 moderate 10 60.0% $2.7866 $0.2787 35
Per trial Task Profile Reward Cost Eligible calls Completion status n-csv-exportn-csv-export__pMHrSY4 clean 1.00 $0.1376 5 success n-csv-exportn-csv-export__z5wbZGR clean 1.00 $0.1413 5 success n-ledger-fixn-ledger-fix__6gebiCn clean 1.00 $0.4388 23 success n-ledger-fixn-ledger-fix__6uvzMhM clean 1.00 $0.3811 22 success n-make-buildn-make-build__N9Srnjo clean 1.00 $0.3837 21 success n-make-buildn-make-build__UN7B9y3 clean 1.00 $0.3345 15 success n-port-conflictn-port-conflict__Adwvcoy clean 1.00 $0.1638 11 success n-port-conflictn-port-conflict__he3Hnzy clean 1.00 $0.1240 12 success n-timesheet-payrolln-timesheet-payroll__Kv9DYWj clean 1.00 $0.1727 5 success n-timesheet-payrolln-timesheet-payroll__eMZ4vCv clean 1.00 $0.1383 4 success n-csv-exportn-csv-export__Aim58SS moderate 1.00 $0.1215 5 success n-csv-exportn-csv-export__uCKk7YK moderate 1.00 $0.1989 8 success n-ledger-fixn-ledger-fix__5SwRJVE moderate 0.00 $0.4616 31 success n-ledger-fixn-ledger-fix__E2Phx2e moderate 0.00 $0.7327 37 partial n-make-buildn-make-build__UCdo8DP moderate 0.00 $0.3551 21 success n-make-buildn-make-build__zDyr6Pu moderate 0.00 $0.3500 16 success n-port-conflictn-port-conflict__FZNrLPj moderate 1.00 $0.1350 12 success n-port-conflictn-port-conflict__FpTfUzV moderate 1.00 $0.1218 18 success n-timesheet-payrolln-timesheet-payroll__6z4okBt moderate 1.00 $0.1206 4 success n-timesheet-payrolln-timesheet-payroll__P3xZYjc moderate 1.00 $0.1893 5 success
Oct 19, 2026 track-s-fx 10 trials, 22 faults applied, between 14 and 61 eligible tool calls per trial. Source: results/smoke/2026-10-19-track-s-fx/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 5 100.0% $0.8972 $0.1794 0 moderate 5 60.0% $0.6270 $0.1254 22
Per trial Task Profile Reward Cost Eligible calls Completion status s-cycle-counts-cycle-count__LWBJxqB clean 1.00 $0.2233 38 success s-error-catalogs-error-catalog__REyXwea clean 1.00 $0.2638 61 success s-followupss-followups__nnRmGNN clean 1.00 $0.1180 15 success s-p1-reports-p1-report__DbuXPpd clean 1.00 $0.0964 22 success s-release-trains-release-train__Husknft clean 1.00 $0.1956 45 success s-cycle-counts-cycle-count__HNRBpn4 moderate 1.00 $0.1940 46 success s-error-catalogs-error-catalog__sHpNZXt moderate 1.00 $0.1044 14 success s-followupss-followups__3bGRK8D moderate 0.00 $0.0684 17 partial s-p1-reports-p1-report__dNXbZsZ moderate 0.00 $0.0765 26 success s-release-trains-release-train__dpvszP9 moderate 1.00 $0.1837 49 success
Oct 18, 2026 track-s-pi-oc 20 trials, 41 faults applied, between 9 and 69 eligible tool calls per trial. Source: results/smoke/2026-10-18-track-s-pi-oc/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 10 100.0% $0.8282 $0.0828 0 moderate 10 70.0% $0.8180 $0.0818 41
Per trial Task Profile Reward Cost Eligible calls Completion status s-cycle-counts-cycle-count__WnfNWts clean 1.00 $0.1523 37 success s-cycle-counts-cycle-count__hmcRWwG clean 1.00 $0.1361 38 success s-error-catalogs-error-catalog__7zdBkPc clean 1.00 $0.0412 9 success s-error-catalogs-error-catalog__8kbXaXL clean 1.00 $0.0624 10 success s-followupss-followups__8nLRDe3 clean 1.00 $0.0428 12 success s-followupss-followups__CwgvvQU clean 1.00 $0.0641 12 success s-p1-reports-p1-report__9GgKpht clean 1.00 $0.0450 22 success s-p1-reports-p1-report__PFKT7v3 clean 1.00 $0.0447 22 success s-release-trains-release-train__DXJAjTF clean 1.00 $0.1121 45 success s-release-trains-release-train__jB5ZJFF clean 1.00 $0.1276 45 success s-cycle-counts-cycle-count__HJCKEi9 moderate 1.00 $0.1446 42 success s-cycle-counts-cycle-count__YCVDEwv moderate 0.00 $0.1469 69 success s-error-catalogs-error-catalog__5BSjevc moderate 1.00 $0.0437 13 success s-error-catalogs-error-catalog__VuHXiEd moderate 1.00 $0.0423 13 success s-followupss-followups__Tt52ciS moderate 1.00 $0.0451 16 success s-followupss-followups__Ux5PXvz moderate 1.00 $0.0493 17 success s-p1-reports-p1-report__EeWH3vG moderate 0.00 $0.0588 26 success s-p1-reports-p1-report__MPiKsMV moderate 0.00 $0.0481 24 success s-release-trains-release-train__EPv9myQ moderate 1.00 $0.1272 50 success s-release-trains-release-train__PfXCWPB moderate 1.00 $0.1120 49 success
Oct 15, 2026 track-s 12 trials, 32 faults applied, between 12 and 57 eligible tool calls per trial. Track S, fx-claude and cc-claude on three service tasks, one attempt per task and profile (D-0030 ). The tables pool both configs. One attempt per cell supports no comparison between them. The cost columns undercount fx-claude: some of its trials have no cost in result.json because the gateway settled them later; the settled costs are in settled-usage.json next to the summary. Source: results/smoke/2026-10-15-track-s/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 6 100.0% $0.4988 $0.0998 0 moderate 6 100.0% $0.4545 $0.1136 32
Per trial Task Profile Reward Cost Eligible calls Completion status s-duplicate-merges-duplicate-merge__2HXwTSk clean 1.00 $0.1163 51 success s-duplicate-merges-duplicate-merge__HPFQagH clean 1.00 - 51 success s-followupss-followups__cPgaDXA clean 1.00 $0.1171 15 success s-followupss-followups__jcXcKHA clean 1.00 $0.0703 12 success s-p1-reports-p1-report__QeAsNYZ clean 1.00 $0.1250 22 success s-p1-reports-p1-report__o6FV6Bz clean 1.00 $0.0700 22 success s-duplicate-merges-duplicate-merge__J2Nuj3e moderate 1.00 $0.1302 57 success s-duplicate-merges-duplicate-merge__yXP3Tam moderate 1.00 $0.1677 57 success s-followupss-followups__G7CFa9x moderate 1.00 - 21 success s-followupss-followups__hkc4tfB moderate 1.00 $0.0783 17 success s-p1-reports-p1-report__R7UjSsp moderate 1.00 $0.0784 26 partial s-p1-reports-p1-report__sXP77Wn moderate 1.00 - 26 partial
Oct 11, 2026 6 trials, no fault applied, between 2 and 4 eligible tool calls per trial. Track N, cc-claude only. The tasks of this run were later rebuilt under D-0023, D-0024 and D-0026, so its trials sit below the exposure floor of 8 in SPEC 10.3 and the moderate profile had few eligible calls to fault. Source: results/smoke/2026-10-11/summary.json
Per profile Profile Trials pass^1 Total cost Mean cost Faults applied clean 3 100.0% $0.1146 $0.0382 0 moderate 3 100.0% $0.0778 $0.0259 0
Per trial Task Profile Reward Cost Eligible calls Completion status n-ledger-fixn-ledger-fix__EGtrgcb clean 1.00 $0.0402 4 success n-port-conflictn-port-conflict__DuaYAQ4 clean 1.00 $0.0345 3 success n-shipments-summaryn-shipments-summary__CdnkvnH clean 1.00 $0.0398 3 success n-ledger-fixn-ledger-fix__3rTZbnf moderate 1.00 $0.0316 4 success n-port-conflictn-port-conflict__tyGHefF moderate 1.00 $0.0203 3 success n-shipments-summaryn-shipments-summary__k2REVrc moderate 1.00 $0.0259 2 success