1. [fatal] The snapshot instrument does not identify capped-run outcomes.
A pre-call snapshot can omit edits made after the last model response, including the final successful patch when no further call occurs. If a response straddles B, a hard cap can truncate it, prevent a tool call, or trigger harness cleanup or recovery. Budget invisibility establishes neither identical stopping behavior nor identical final artifacts. The outcome is exact only for a specifically defined checkpoint-stopping policy, not for ordinary token-capped execution.
Fix: Define cap enforcement, response truncation, tool execution and finalization precisely. Run actual capped trajectories on a validation subset at every B. If retaining snapshots, claim checkpoint performance and validate the reconstruction against that explicit intervention.

2. [major] Equal token ceilings are not equal budgets consumed, and do not isolate a “budget effect.”
Harnesses may stop well below B. Input tokens, output tokens, hidden reasoning, caching and repeated context have different accounting and usefulness. Tokenizers also differ across models. Changing B measures each harness's response to a resource constraint, not a clean decomposition of harness architecture versus budget.
Fix: Specify billable versus logical token accounting, reasoning-token inclusion and overshoot behavior. Report actual consumption, early stopping and cost alongside performance-cap curves. Add separate input/output or monetary-budget sensitivity analyses. Describe the estimand as performance under a common token-ceiling policy.

3. [major] “Defaults” are neither reproducible nor necessarily fair configurations.
Aider's chosen diff format may suit some models poorly; provider defaults, context limits, tool schemas and reflection settings can create artificial handicaps. Model identity is also confounded with provider route. Native defaults are a legitimate package comparison, but not an isolated harness-architecture effect.
Fix: Pin package versions, configuration files, prompts, model endpoints and decoding parameters. Smoke-test tool and edit compatibility in every cell. Report native-default and harmonized configurations separately, and validate route equivalence where feasible. Publish the complete API transcript/accounting schema.

4. [major] The variance shares are descriptive properties of nine selected cell means.
A balanced two-way sum-of-squares decomposition is mathematically valid, but its shares depend heavily on model selection, harness selection and score scale. They do not estimate universal model/harness variance components. Sampling error and run stochasticity inflate squared contrasts, especially interaction; forcing the three shares to sum to one silently absorbs this noise. A small total between-cell sum of squares makes the ratios unstable.
Fix: Label these finite-grid descriptive shares. Report absolute effects and the denominator, not just normalized shares. Estimate uncertainty and stochastic variation with a replicated hierarchical model or explicit noise correction. Do not compare directly with prior decompositions that include a separate noise component.

5. [major] The inference and null procedure do not match the design.
A task-paired bootstrap preserves pairing but a generic task bootstrap does not preserve the fixed language strata. It also does not recover run uncertainty from one realization per cell. Two-seed disagreement is not, by itself, a noise floor for an interaction contrast. Percentile intervals for nonnegative squared-effect shares often exclude zero even under a null, making the result-kill rule unreliable. Unspecified label shuffling can destroy model pairing or cross-cap dependence.
Fix: Use a stratified paired bootstrap and adequately replicated runs to propagate both uncertainty sources. Specify a defensible within-task, within-model permutation test that retains each run's whole cap trajectory. Base equivalence conclusions on prespecified signed contrasts or spread margins, not whether a squared-share interval touches zero. Prespecify the “low cap” and simultaneous versus pointwise inference.

6. [major] Public Exercism tasks create contamination, and unit-test success is not exact correctness.
Restoring tests prevents one shortcut, but not memorized solutions, fixture-specific hardcoding, modifying the test runner or dependencies, or exploiting incomplete tests. Reference-pass/stub-fail establishes basic grader operation, not task validity. Dropping failed tasks can change the benchmark population.
Fix: Freeze task eligibility before evaluating models and report exclusions by language. Grade submitted solution files in a clean trusted environment with a trusted command and dependencies. Add withheld tests or newly authored task variants, and report public-task versus fresh-task results. Call the endpoint “original-test pass rate,” not exact program correctness.

7. [major] The headroom pilot cannot establish headroom and permits selection-driven results.
With ten tasks, one success changes the score by ten points; zero or nine successes gives weak evidence about the underlying rate. Requiring saturation under all three harnesses can miss a ceiling in individual cells. Replacement based on observed performance also changes the share estimand, and the proposed replacement can duplicate an existing model.
Fix: Use a larger, language-stratified pilot with uncertainty bounds and fresh main-study tasks. Prespecify unique replacement models and cell-level headroom criteria. Report bounded-scale sensitivity analyses and absolute score differences. Do not interpret a small share near ceiling or floor as absence of a harness effect.

8. [major] Failure handling and financial stopping can destroy balance and introduce selection bias.
Dropping a cell eliminates the proposed balanced decomposition. Excluding infrastructure failures is safe only with a defensible classification and retry policy; long, tool-heavy trajectories may fail disproportionately. Cutting one model's repeat runs changes uncertainty asymmetrically. Financial hard stops can censor runs rather than merely reduce sample size.
Fix: Freeze a uniform retry and failure-classification policy, log all failures, and report both conditional-success and reliability-inclusive endpoints. Fund the planned caps or reduce a prespecified balanced sample before main collection. Stop the study rather than silently drop a cell. Recalculate costs using input/output mix, retries and cap-tail consumption.

9. [major] The uncovered cell is plausible, but not established as real.
The supplied summaries already establish harness effects, rank reversals and variance decompositions. Adding intervals alone is not a substantive novelty claim. A mixed open/closed grid with cap-dependent shares could be a narrow contribution, but the summaries do not establish that previous work lacks it, and an invalid cap reconstruction would not fill it.
Fix: Produce an auditable prior-art matrix covering estimands, cap enforcement, budget accounting, mixed-model coverage, uncertainty and scoring. Verify the cited papers' actual methods. Frame novelty as a validated resource-constrained factorial evaluation, not discovery that harnesses matter.

10. [major] The proposed downstream conclusions exceed the experiment.
A large harness share does not imply rank reversals: a harness can improve every model equally. A small share does not establish equivalence. One closed model versus two open models cannot identify an openness effect, especially with different routes. Small exercise tasks also cannot justify conclusions about METR horizons or repository-level leaderboards.
Fix: Separate harness main effects, interactions and rank-order uncertainty. Describe the closed/open comparison as specific model contrasts. Restrict conclusions to this task distribution, configuration grid and budget policy; require representative replication before advising broader evaluation changes.

VERDICT: no-go