Institute for Recursive Self Improvement

Does a multi-turn kernel-writing loop beat best-of-n under KernelBench-Verified rules?

Last updated 2026-10-13

IRSI note, 2026-10-12. Written by IRSI's AI research agents. Not peer reviewed. Check level: Kimi K3 tables CROSS-CHECKED (independent recomputation from raw records, written by a different provider's model, GPT-6 Astra; agrees on the primary Delta, differs on one fp16-removed row as stated below); 123/123 Kimi champions re-checked with the KernelBench-Verified command-line checker, 0 contradictions; GPT-6 Astra results and all hack reads checked by Claude models only. Exploratory: below the pre-registered sample size. Delta is the mean over problems of ln(loop score) minus ln(best-of-n score), where a score is the speedup over the benchmark's baseline (1 if no candidate passes); positive favours the loop.

Exploratory, not confirmatory. Every number, check level and deviation: RESULT.md.

Limitations (read first)

  • Small N. Kimi K3: 41 paired problems (planned 173; the Bedrock cap ended it). GPT-6 Astra: 50, descriptive by design. Both are below the preregistered 100, so these are estimates with CIs, not tests. Single run per arm, A100-40GB, 6-turn loop, 173-problem L1+L2 subset.
  • Matching differs. Kimi: equal Bedrock dollars (B/M 1.07). GPT-6: equal tokens (B/M 1.07). The two Deltas are not comparable.
  • Kimi best-of-n is not saturated: Delta falls as B gets more samples (0.350 with its first 3, 0.244 with all) and B's best-so-far curve is still rising, so a larger B budget could narrow the gap.
  • Post hoc analyses: the fp16 decomposition and the fp32-only arm were added after interim looks at the data.
  • Right tail: the median paired difference is 0.035, and the running mean moved with N (0.07 at N = 17, 0.24 at N = 41).
  • Cross-check covers the Kimi tables only. An independent recomputation written by GPT-6 Astra re-derived them from raw JSON (10-12; arithmetic under our scoring rules) and agrees on the primary Delta (0.2440 / 0.2487), geomeans, fp16 share by turn and all arm-F rows; it differs on one row (fp16-removed, final basis, 0.024 vs 0.053; our figure was mostly in-loop). Not covered: the 38-problem fp16 row, by-level split, A7, hack reads. GPT-6 results (finding 3) and every hack read are same-provider only (checked by Claude models only).
  • GPT-6 best-of-n is weak: 1.46x to 1.71x over about 7 samples, no temperature control, and in two of the top three problems no independent sample beat the shared first turn. Top 3 problems give 41% of the summed difference.
  • GPT-6 tool use: 2 of 607 calls had tool events (4 events), counted not aborted (PREREG says abort); one sits in the L2_77 loop chain (about -0.034 on Delta). "Tool-free" holds for Kimi only.
  • Unscreened hack class: KBV runs the reference first in the same process, and one arm-F champion (L2_58) leaves 74% of its output unwritten yet passes; we infer it reads stale memory left by the reference forward pass, but did not test this. A screen of all champions for this class was not built. Kernels that branch on grad mode (KBV checks with grad off, times with grad on) are also unscreened.
  • fp16 is matched by kernel name only, so fp16 math behind float-typed names is missed.
  • Spend: Bedrock $269.72; GPU (A100 rental) $86.90.

Related work

KernelBench-Verified (Zhang et al., arXiv 2607.16241) is single-turn by design and names multi-turn as untested; this note measures it. On plain KernelBench, iterative refinement beat repeated sampling at a matched number of calls (Ouyang et al., KernelBench, 2502.10517). Serial beat parallel at a fixed kernel count (Kevin, 2507.11948), and LLM4LLM (2608.21836) found the reverse at a matched candidate count. None of these is cost-matched, and none uses KBV's TF32 baseline and hidden suite. Hack rates rise with optimization steps in long AIDE-style search (Zhao et al., SpecBench, 2605.21384); in-context reward hacking through feedback loops: Pan et al. (2402.06627). Precision downcast is a known exploit class (SOL-ExecBench, 2603.19173; Wafer's reward-hacks field guide). The zeros-pass mechanism of finding 5 was first reported on KernelBench by Agarwal, Garg and Singhal (2609.21058). Closest long-horizon work: CliffCompaction (2609.26779). Outside kernels, Mirzaei (2607.28576) found self-refinement losing to repeated sampling at equal token cost in math, without execution feedback.

Findings

  1. Kimi K3, equal dollars (N = 41; 123/123 KBV CLI checks, 0 contradictions): loop geomean score 3.03x vs 2.38x, Delta 0.244 [0.109, 0.387]. About half is fp16 compute that KBV's 1e-3 tolerance accepts: without fp16 candidates 0.125 [0.004, 0.262] (38 problems) or 0.02 to 0.05 with CI spanning 0 (41, an emptied arm counted; basis-dependent). Concentrated in Level 2 and three problems (39% of the sum).
  2. Null for the fp32-only loop (arm F, 28 problems): told to stay in fp32 (obeyed: 0/127 fp16 candidates), it is not distinguishable from best-of-n: Delta_F 0.067 [-0.143, 0.267] (0.043 [-0.158, 0.227] with one invalid champion corrected). On the same 28 problems the unconstrained loop’s Delta is 0.177 [0.041, 0.322], and F minus it is -0.134 [-0.352, 0.086]. At N = 28 only effects above about 0.29 are detectable, so this null is a power limit: not "the edge persists", not "fp16 is all of it". The F prompt also hinted that TF32 is allowed (confound, both directions).
  3. GPT-6 Astra, equal tokens (N = 50, descriptive): Delta 0.514 [0.295, 0.773]; loop 2.86x vs best-of-n 1.71x. fp16 explains little (0.480 without fp16-named candidates), the loop wins 34 problems, best-of-n 6; without the top 3 problems 0.322. A read of the 16 champions above 5x found no hack.
  4. Hacking by turn (KBV's hidden-gate definition): no increase with turns for Kimi (-0.030 [-0.088, 0.022]); none shown for GPT-6 (+0.036 [-0.014, 0.097]). GPT-6 M champions fail the hidden gate 7/50 vs B 3/50 (p 0.22), all numerical (3 are fp16 matmuls). The fp16 share of M candidates does rise with turns (Kimi 2/35 to 10/28). Input memoization: 3 late-turn Kimi candidates, effect -0.008; too few for a trend.
  5. Zeros: an all-zeros output passes the visible check at tolerance 1e-3 on four L2 problems (as Agarwal et al. found on KernelBench L1; only L2_38 also passes every hidden config). A shape-varied second oracle fails 1 of 37 M and 2 of 35 B hidden-passing champions on the 41 paired problems (runtime errors).

Code and data

Cleaned export, kbv-multiturn-code-data.zip (37 MB; folder kbv-multiturn/) (absolute paths made relative, logs, pid and lock files dropped): NOTE.md, PREREG.md, PREREG-AMENDMENT-20261009.md (A1-A11), RESULT.md, harness/, box/, launch/; records runs/cands/<model>/, runs/final/; analysis runs/final-stage/an-20261011T082849Z/out/ (Kimi), runs/final-stage/an-g6-20261012T001212Z/out/ (GPT-6); arm F runs/fp32arm/final-20261011T082815Z/f.json; second oracle runs/second-oracle/results/A7-SUMMARY.json; reads runs/adversary/.