feat(benchmarks): narrow-regime loss decomposition — split3 is positionally degenerate; input blindness is the dominant loss class - #520
Conversation
…er the input-fidelity arm 55% of split3 gold mass sits on trace spans the analyst cannot distinguish: the normalized trace keeps only each step's shell command, so mini-SWE-agent's duplicated submit boilerplate hides the labelled step. A constant accuse step_count-1 rule scores micro F1 0.568 there against 0.180 for the shipping analyst, while the same rule is worthless on dev-32, holdout-1, and holdout-2. Adds two tools: decompose-analyst-loss.mjs (per-case failure classes and counterfactual micro F1 from a run's own observations) and restore-step-reasoning.mjs (pairs raw trajectory reasoning back onto normalized steps so the traces importer can carry it).
…ed, kill gate clear Paired 8-observation A/B on four spent split3 cases, prompt and model held fixed and trace bytes the only difference: micro F1 0.1538 to 0.2000, precision 0.1034 to 0.1500, input-blind gold hits 1/8 to 2/8, findings 29 to 20. Below the pre-registered 0.375 blind-recall and 0.25 F1 bars, above the kill line, so the arm is not scaled on split3 evidence; the input repair stands on its own model-free proof (input-blind gold 64/116 to 0/116). decompose-analyst-loss.mjs now exits quietly when its reader closes.
…ied reference A constant accuse-last-step rule scores 0.568 micro on split3 (27/37 cases carry gold on the final submit decision) vs 0.180 for the shipping analyst. split3 is retired as a certification instrument; the promotion decision is unaffected because holdout-2 alone orders the arms identically.
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — f9060c0e
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-01T22:08:21Z
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 3 (1 medium-concern, 1 low, 1 weak-concern) |
| Heuristic | 0.1s |
| Duplication | 0.0s |
| Interrogation | 115.9s (2 bridge agents) |
| Total | 116.0s |
💰 Value — sound-with-nits
Ships two genuinely new, in-grain diagnostic/preprocessing tools plus a rigorous loss-decomposition report and a certification-integrity disclosure; the only blemish is a new committed .scratch/ directory of exploratory scripts and machine-generated data dumps that should be trimmed or gitignored.
- What it does: Adds
benchmarks/trace-analysis/tools/decompose-analyst-loss.mjs(recomputes analyst micro-F1 from a run's own observations + labels + OTLP traces, classifies every gold step as input-blind/hit/near/far/silent, splits unsupported findings by distance, and recomputes four counterfactual micro-F1s by neutralizing one class at a time) andrestore-step-reasoning.mjs(model-free preprocessing that p - Goals it achieves: (1) Explain why every certified config collapses to ~0.17-0.18 micro F1 on split3 — answered: 55% of split3 gold sits on trace spans the analyst cannot distinguish because the normalizer drops reasoning text and mini-SWE emits duplicate submit commands, AND split3's labels are positionally degenerate (a constant 'accuse step_count-1' rule scores 0.568 there vs 0.180 for the shipping analyst). (2)
- Assessment: The committed core is strong and fits the codebase.
decompose-analyst-loss.mjsdoes NOT duplicatecompare-analyst-runs.mjs(arm-level summary/paired CIs),src/causal-attribution.ts(factorial variance decomposition), orsrc/counterfactual.ts(interventional re-run of a mutated trajectory) — it is a distinct per-case, observation-only failure-class decomposition with compositional counterfa - Better / existing approach: No better architecture for the two tools or the report — searched src/ for decompose/counterfactual/positional/input-blind and the benchmarks/tools/ peer (compare-analyst-runs.mjs); none overlap in capability. The one available improvement is to NOT commit the
.scratch/directory (see finding). - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error
🎯 Usefulness — sound-with-nits
Two reusable analysis/preprocessing CLIs plus a measured loss-decomposition report and an instrument-retirement disclosure — reachable, exercised, and built in the grain of the existing trace-analysis tools; only gap is the tools index wasn't extended.
- Integration: Both new tools are standalone CLIs under
benchmarks/trace-analysis/tools/, the same category as the existingcompare-analyst-runs.mjs, and both are reachable and were actually run.decompose-analyst-loss.mjsgenerated the report shipped in this PR —.evolve/narrow-regime-decomposition-2026-08-01.md:21names it as the tool and:26gives a reproduce command invoking it; the certified READM - Fit with existing patterns: Fits the established pattern: a trace-analysis tool is a
#!/usr/bin/env node.mjswith a leading//usage block surfaced by--help/-h, arg parsing viaparseArgs, failure viausageError, reading run observations + labels + OTLP traces, and reusing the package's pooled micro-F1 semantics (compare-analyst-runs.mjs:1-30is the template;decompose-analyst-loss.mjs:1-27mirrors it). `d - Real-world viability: Both tools fail loud rather than silently degrading —
decomposethrows on a missing trace (decompose-analyst-loss.mjs:155) and an observation for an unknown trajectory (:197);restorethrows on step-count mismatch (:133), multiple/zero trajectory files per case (:140), message-count drift (:146), and an action absent from its paired message (:155).restorewrites per-case source/ - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
🔎 Heuristic Signals
🟡 Cruft: console debug added .scratch/blind-rate.mjs
- console.log(
${name}: trace directory missing (${traceDir}))
🎯 Usefulness Audit
🟡 tools/README.md index not extended for the two new tools [ergonomics] ``
benchmarks/trace-analysis/tools/README.mddocumentscompare-analyst-runs.mjswith full usage but this PR addsdecompose-analyst-loss.mjsandrestore-step-reasoning.mjswithout a single index entry (the only README touched is the certified-reference addendum,git diff origin/main...HEAD --name-only | grep readme). A maintainer browsing the tools dir for a standing loss-decomposition or trace-prep tool will not find them listed; discoverability is the gap, not function — each tool is sel
💰 Value Audit
🟠 .scratch/ payload is exploratory chaff with machine-specific paths and a binary .mjs — drop or gitignore it [proportion] ``
.scratch/does not exist on origin/main (git ls-tree origin/main .scratch/ returns nothing), so this PR introduces it as a committed namespace with 13 files / ~2,509 lines. Problems: (1).scratch/blind-rate.mjsand.scratch/positional-baseline.mjsare the superseded exploratory versions of logic now implemented more rigorously indecompose-analyst-loss.mjs(identical occurrences-map + beyond-boilerplate blind check at decompose-analyst-loss.mjs:157-168 vs blind-rate.mjs:45-60). (2) `run-
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
Premise check withheld merge —
|
What this is
The measured answer to "why does every configuration collapse to micro 0.17–0.18 on split3" — recomputed for free from the three certified arms' own observations, no model spend for the diagnosis, then one pre-registered $1.17 smoke of the mechanism it exposed.
Ships:
tools/decompose-analyst-loss.mjs(per-case failure-class decomposition with counterfactual micro recomputation),tools/restore-step-reasoning.mjs(model-free trace preprocessing), the full decomposition report (.evolve/narrow-regime-decomposition-2026-08-01.md, 74-row per-case table), and the split3-degeneracy disclosure appended to the certified reference README.The three findings, each verified against the shipping scorer
thinkingis never populated (tracescodetracebench-trajectory.ts:132joins a field that is always empty), and mini-SWE emits duplicate submit commands. The analyst cannot distinguish the labeled span from its neighbor even in principle. Counterfactuals kill the alternative hypotheses with numbers: escape-suppression −1.3pp, solved-abstention −11.1pp (both rejected); boundary credit ±2 steps +15.4pp (M1 survives, transformed).restore-step-reasoning.mjsrestores each step's reasoning text into the trace: input-blind gold 64/116 → 0/116, duplicate spans 110/967 → 2/967, with a byte-identical control re-import proving the A/B differs in exactly one thing. The paired 8-obs smoke (pre-registered bars in commit414b179BEFORE the run): precision +4.7pp, findings −31%, pad findings −52%, micro F1 0.154 → 0.200 — but the primary gate (blind-gold recall ≥ 0.375) read 0.250: the arm is NOT promoted on this evidence; the full A/B on dev-32/holdout-1/holdout-2 (~$9/arm) is the recorded next step.Verification
Decomposition reproduces the certified
result.summariesexactly for all three arms. Constant-rule degeneracy independently recomputed by the operator from raw labels (27/37, F1 0.568 — exact match). Pre-registration committed before the paid run; the miss is reported, not re-barred. Gates: biome clean, typecheck exit 0, vitest 4656 passed / 3 skipped, merge-tree clean.