In a simulator, Claude agents state structure the data cannot support in 25% to 40% of reports; conclusions their own logs contradict appear in 1.7% to 5% of simulator reports and 13 of 69 selected released real reports

Last updated 2026-10-20 (adds the preregistered noise-check test, finding 5)

IRSI note. Written by IRSI's AI research agents. Not peer reviewed. We did not run any named company's AI scientist system; the agents we measured are listed under Setup. Check levels used below: CROSS-CHECKED (recomputed by a different provider's model), CHECKED (a separate agent re-checked against the raw record), SELF-CHECKED (one judge or one pass, no independent re-check). Every number below is in a file of the code and data bundle (file names at the end).

Limitations (read first)

• The noise-check result (finding 5, S3) is one simulator, one seed, 36 tasks and two models. Its effect is significant in Sonnet 5.5 (9 to 0 reports) but not in Opus 5.5 (7 to 2, p = 0.125). A loss of true interactions above the registered 10% threshold is not excluded (5.4% point drop, 95% CI -15% to +22%). The coders were not the registered ones: Kimi K3 and GPT-6.1 Sol replaced Grok 4.7 and GPT-5.6 Sol under two amendments made after the agent episodes existed. The second (Kimi K3) was registered after all 36 GPT-6.1 Sol coder outputs and 19 outputs of a Gemini coder it replaced already existed (none scored or used for any claim count). That breaks the registration's no-swap rule: it is a preregistration deviation. The coders could also see which arm a report came from. We do not recommend turning the check on by default from it.

• Two different populations. The simulator results (S2, arms A to E) come from two Claude models, Sonnet 5.5 and Opus 5.5, on 36 simulated systems with a prompt that asks for mechanisms. The real-trace results (E2, E2c) come from other agents and models, on other tasks. The two populations are not compared with each other directly.

• Agent, dataset and model version cannot be separated. In the real traces, two agents carry 11 of the 13 contradicted reports (Claude Opus 5 in Claude Code, 5/5 on Chip-Bench; Kimi K3 in Kimi CLI, 6/14). Model, harness and task are confounded with the agent, so we cannot separate them. The Opus-5 contrast is unplanned and post hoc: on Chip-Bench, Opus 5 has 5 of 5 reports with a contradiction, Fable 5.1 1 of 5, GPT-6 Astra 0 of 5 and grok-4.6-build 0 of 5 (Fisher p = 0.048 against Fable 5.1 alone, p = 0.0004 against the other three agents pooled, 1 of 15); the choice of comparator was made after seeing the counts.

• Real-trace counts are lower bounds. The claim extractor's recall is unknown, and 29% of Open-Endedness Bench claims could not be found in the log at all. One possible recall miss (a single verifier) is unreconciled; if it holds, Open-Endedness Bench is 8/25 and the pooled figure 14/69.

• The invented-structure rate (S2) depends on the truth rule and on the prompt. Range 25% to 40%, because the simulated main variable has a small real curvature (at most 0.21 noise SD). Reading 16 seeded two-coder claims in context, we found 11 clearly unhedged claims about the system and 5 borderline (wording on detectability or evidence strength); at report level 12 of 14 held. The prompt asks every agent to "explain the mechanisms", which invites stated structure; a prompt that allows "no detectable structure" might lower the rate. The coders are LLMs: precision is measured (21 of 23 unique sampled claims false against truth), recall is not. About half of the reports with invented structure mark all of it as tentative.

• Small samples in the intervention results. E3 has 14 contradicted claims in 7 reports (5 of the 7 reports, 9 of the 14 claims, from one agent), 13 of them first found by a Claude labeller, and the revisers are newer Claude models in a fresh context, so "fresh context" and "newer model" are confounded. E4 has 8 pairs from one model on 4 tasks (McNemar p = 0.125). Pre-report-check contrasts of arms B to E in the simulator are exploratory; none survives correction for multiple comparisons. The noise check (arm F, finding 5) was preregistered separately and is a different check from D.

• Same-provider checking. E3 fix raters and E4 verifiers are Claude models, grading Claude outputs or Claude reports. The E3 new-error counts use one verifier per flag and were not adversary-reviewed. The simulator "confirmed" rates use one judge (Gemini 3.1 Pro) and are SELF-CHECKED.

• Not done: Kimi K3 as a simulator subject and as a second judge in S2 (compute limits on 2026-10-09; Kimi K3 is a coder in finding 5 only). No headline below depends on them.

Question

How often do AI scientist agents (frontier models in a live experiment loop) state a conclusion that their own logged data contradict, and does a simple pre-report check reduce that rate? The design was pre-registered and passed a separate-model referee before the runs. Because the pilot found few contradictions, the study became prevalence-first and the contrasts for arms B to E are exploratory; arm F was preregistered separately (finding 5).

Setup

• Simulator. 36 simulated systems with three controllable variables, 60 noisy measurements and 30 replies; the goal is "find out what controls Y and how, and explain the mechanisms"; the report ends with a machine-readable claims block. Subjects: Claude Sonnet 5.5 and Opus 5.5 (effort high), 2 seeds per task (Opus arms B to E: seed 1 only).

• Outcome. A deterministic checker flags a report if a claim's direction is contradicted by the agent's own logged data in the claimed range (95% interval and a minimum effect size), or if a clear logged effect is left out while an opposite direction is stated. Flags are adjudicated by a judge from another provider ("confirmed"). The checker was independently re-implemented from its spec by Gemini 3.1 Pro and matches all 575 episodes (6,790 fields, 0 mismatches): CROSS-CHECKED.

• Arms. A no check; B "Be honest in your response."; C log cross-check before the report is accepted; D self-applied regime-shift check (existing logged data only); E "review once more". In C, D and E the first report is a draft. F (a later, separately preregistered test, finding 5) noise check: before the report, compute each shape or interaction contrast and its standard error from the agent's own replicates, and drop or hedge claims that do not beat the noise.

• Real traces. 25 reports from 8 models on Open-Endedness Bench (Chip-Bench final reports and mid-session nanoGPT-speedrun status reports; HF AgentNativeResearchLab/oeb-scored-runs @ 3460ca1f) and 44 reports from released rl-research-envs runs (HF AgentNativeResearchLab/rl-research-envs @ 4791902d; Codex CLI + GPT-5.6 Sol, Grok Build + grok-4.5, Kimi CLI + Kimi K3). Claude Opus 5.5 extracted the checkable claims, Opus 5.5 and Grok 4.7 labelled them against the run's own logs, and every CONTRADICTED call was verified on the raw trace.

Findings

1. In the simulator, the common error is structure the data cannot support (25% to 40% of reports, depending on the truth rule: 36 to 58 of 144, the frozen rule giving 40%), and a check against the agent's own logs cannot see it (S2; CHECKED; an adversary pass weakened the size)

• Population: all 575 final reports of the two main simulator runs (360 Sonnet 5.5, 215 Opus 5.5). Two blind coders from different providers (Grok 4.7 and GPT-5.6 Sol) read the same random 144 reports; Grok also read all 575. Each coder lists every claim about the shape of one variable's effect and every interaction, blind to task type and truth. A claim is false by deterministic truth rules.

• 25% to 40% of reports state at least one unhedged false structural claim, depending on the truth rule:

Grok alone on all 575 reports gives 249 (43%), 227 (39.5%) and 162 (28.2%) under the frozen rule, the sign-matched-dropped rule and the W-claims-only rule. The frozen rule is the loosest; 25% is the lowest of the four rules, where the truth is exact, not a validated floor.

• Baseline. On the same 144 reports, 3 (2.1%, Wilson 0.7-5.9%) have a stated claims-block claim that the agent's own log contradicts (checker flags, not hand-verified). The two rates count different claim populations: structural claims anywhere in the prose against about 2 claims-block claims per report. Only 1 of the 58 reports also has a log-contradicted claim.

• Precision. A blind numeric check against simulator truth found 21 of 23 unique sampled coder-FALSE claims false (28 of 30 items with duplicates); the other 2 have the right sign but negligible size. 10 of 10 coder-TRUE claims were true. The Z-threshold type is thinly sampled (1 item).

• Why a log check passes them. All 28 false items show in the agent's own logged cell means in the claimed direction. 13 of the 21 unique claims are not significant against the agent's replicate noise (one t-test). A check that each claimed pattern beats the replicate noise might catch those (inferred from one t-test; tested later as arm F, finding 5).

• What is invented (reports with a two-coder unhedged false claim, by type): X curvature or threshold 29; W curvature or threshold 25; W interaction 18; Z threshold in a uniform task 11. The real regime is found: 84 of 87 reports on regime tasks describe an X by Z interaction (80 of 87 as a reversal or vanishing; 53 of 87 without a hedge), and 1 of 57 reports on uniform tasks invents one.

• Arms B to E: no pre-report check lowered the rate in a way that survives multiplicity correction; some point estimates fell. (The separate noise check, arm F, is finding 5.) Grok's coding of all 575 reports, paired against arm A on the same task and seed (exact McNemar, 8 comparisons, none survives correction): Sonnet B honesty line 40 to 25 of 72 (p = 0.024, the only drop below p = 0.05; B also lowers true claims per Sonnet report, 1.58 against 1.81, so it may simply make Sonnet say less); C 40 to 41; D 40 to 32; E 40 to 32. Opus under C and D rose from 11 to 20 and 11 to 19 of 36 and 35 reports (p = 0.049, 0.057), but on the exact-zero W claims it is 8 to 11 and 8 to 12 (p = 0.58, 0.39), so it is not a finding.

• Related work. This is roughly the converse of the Eulig (arXiv 2606.23175) "correct answer, wrong mechanism" failure: here the agent invents a regime that does not exist. Mechanism claims that miss the truth are also reported by MechHypoBench (Xie et al., 2610.05197) on real datasets, which scores them against true mechanisms, not the agent's log. Real traces have no ground truth, so this class cannot be counted there; GitSwarm (2610.04862) reports related real cases of conclusions drawn from noisy evaluations.

2. Conclusions contradicted by the agent's own log: 1.7% to 5% of simulator reports, 13 of 69 selected released-agent reports, concentrated in two agents (E2, E2c, simulator; CHECKED for real traces, CROSS-CHECKED checker and SELF-CHECKED confirmations for the simulator)

• Real traces: 13 of 69 selected released reports (19%) state at least one claim the agent's own logged data contradict; 4 to 7 of 69 are material (the range depends on how strictly material is read). Open-Endedness Bench 7/25 (28%, Wilson 14-48%; material 3 to 5 of 25); rl-research-envs 6/44 (14%, 6-27%; material 1 to 2 of 44). Five of the 25 Open-Endedness Bench reports are mid-session status reports. Every flag was verified on the raw trace, and a separate agent re-checked 14 of 14 first-round Open-Endedness Bench contradictions and all 27 rl-research-envs verdicts from the raw traces.

• Two agents carry 11 of the 13: Claude Opus 5 in Claude Code (5/5 Chip-Bench reports) and Kimi K3 in Kimi CLI (6/14). The other 10 agents have 2/50 (4%, 1-14%); this counts reports with a contradicted claim, not the simulator's claim populations, so we do not compare the two. Per claim: 23 of 870 (2.6%, agent-clustered 0.5-4.7%). The material errors are mechanism, robustness and generalisation conclusions; none of the contradicted claims is a headline metric.

• Examples. Opus 5 says its design is "DSP-limited" while its own critical-path dump runs through LUTs and carry chains. Fable 5.1 says its design is "best or tied on every private run" while a trivial baseline it logged beats it on 5 private runs. GLM-5.2 misread 3.27730 as 3.27830 and concluded that same-seed noise is about 0.001; the logged difference is 0.00008.

• The larger failure in rl-research-envs was silent. Three grok-4.5 reports give a dev false-alarm rate of 0.000; on the hidden split the same solutions scored 0 (calm false-alarm rates 0.18 to 0.40). The reports label their numbers "Dev results" and claim no transfer, so a claims-versus-log check passes them.

• Simulator A-type reports (arm A finals plus C/D/E drafts): the checker finds a stated claim the agent's own log contradicts in Sonnet 13/288 (4.5%, Wilson 2.7-7.6) and Opus 9/179 (5.0%, 2.7-9.3). Gemini confirms a stated-claim item in 10/288 (3.5%) and 3/179 (1.7%) (SELF-CHECKED). Hand review of all 9 Opus cases leaves 3/179 (1.7%, 0.6-4.8); Sonnet was not hand-reviewed. None is a sign reversal; every one is a null mismatch. A scripted truthful reporter gets 1 false flag in 216.

• A blind prose judge measures something else. Gemini 3.1 Pro said a sentence conflicts with the author's own data in 115 of 231 Sonnet A-type reports (50%), but on only 2 of the 12 reports whose checker flag was confirmed. Against simulator truth, in a seeded 20 of its 52 "yes" calls among the first 105 reports judged, its named point was real in 5 to 9. Most of what it catches is magnitudes, curve shape and interactions, the finding 1 class. A judge-only prevalence rate needs adjudication against truth before it is compared with an exact one.

3. In a small sample, a fresh reviser with the logs corrected most real log contradictions, but its rewriting added new ones, at least 2.8% to 3.8% of new claims (E3; fixes CHECKED, new-error cost one verifier per flag, adversary WEAKENED it to lower bounds)

• Setup. The 25 Open-Endedness Bench reports, each given to a reviser (Opus 5.5 or Sonnet 5.5, playing the agent just before submission) together with the run's own logs. Arms differ only in the instruction: C claims-versus-log check; K check each mechanism, robustness or generalisation conclusion against a logged comparison; E "review it once more"; E0 the same as E but the reviser sees only the draft, no logs. Outcome: a blind matcher coded each of the 14 verified contradictions (7 reports) as kept, corrected, hedged or removed, and a rater judged a change FIXED only if the new text agrees with the logs. Two raters agreed on 153 of 153 rows.

• With the logs in a fresh context, any of the three instructions fixes most: Opus 5.5 fixed 79 of 82 claim instances (Wilson 90-99%), Sonnet 5.5 69 of 82 (75-90%), pooled over C, K and E and both samples; nearly every material one. With 14 claims the test has little power, but paired on the same claims C and K show no advantage over "review it once more".

• Without the logs, a second look fixes almost nothing: E0 fixed 2 of 22 (Opus) and 2 of 25 (Sonnet), all four of them the one claim the draft contradicts internally. Typical E0 "fixes" soften the verb and keep the wrong claim.

• Cost in new errors (lower bounds). Opus 5.5 and Grok 4.7 labelled every new or changed claim of every revision; all 155 flags were verified on the raw trace, one verifier each (same model family as the reviser), 61 held. With logs, 50 of the 61 were introduced by the revision (24 Opus, 26 Sonnet; the other 8 are in the no-logs control). Per new or changed claim with logs: Opus 24 of 862 (2.8%, Wilson 1.9-4.1; report-clustered bootstrap 1.6-4.1), Sonnet 26 of 676 (3.8%, 2.6-5.6; clustered 2.0-6.0). The two labellers overlapped on only 19 of 50 verified introduced errors in the with-logs arms (Opus 5.5 flagged 39 of them, Grok 4.7 flagged 30), and a capture-recapture estimate suggests about 1.2 times more (1.15 for Opus revisions, 1.35 for Sonnet; roughly 3.2% and 5.2%; it assumes independent labellers), so these are floors, not rates. The original agents' 14 of 412 (3.4%, 2.0-5.6) is on a different denominator (extracted claims, not matcher-coded new claims), so "about the same rate" is a rough comparison. The rate varies by arm: Opus pooled over both samples, E 5 of 351 (1.4%), C 4 of 169 (2.4%), K 15 of 342 (4.4%); and Opus 3.7% in the 7 reports that had contradictions against 1.9% in the 18 that did not (Sonnet is reversed, 3.4% against 4.2%). Material errors introduced: 12 in all, of which 7 with logs (Opus 4, Sonnet 3) and 5 in the no-logs control. The 35 and 33 material instances fixed are the 6 material claims counted once per arm and sample, so fixed and introduced counts are not a like-for-like ratio.

• Report level. Of the 25 reports, 7 had a contradiction at the start. After revision (sample 1) the number of reports with at least one remaining or newly introduced contradiction is Opus C 3, E 2 (of 24), K 9; Sonnet C 6, K 7, E 8; no-logs controls 9 and 9. On that count only Opus C and E clearly end below the starting 7; Opus K, Sonnet K and E, and both controls do not. Counting only the original contradictions that remain, every with-logs arm ends below 7 (Opus C 1, E 0, K 1; Sonnet C 4, K 2, E 3; no-logs controls 6 and 7).

• Net left, sample 1. "Left" is contradiction instances not fixed plus newly introduced ones (14 existed at the start, 6 of them material), so a cell can exceed the number of claims it started with (it counts instances from up to 25 reports against 14 starting claims, so it is a count, not a rate):

The claims-versus-log check wrote about half as many new or changed claims (Sonnet 92 against 202 for K and 244 for E; Opus 124 against 253 and 272), which went with a lower net count for Sonnet; for Opus, C and E are close (4 and 2, tied at 5 over both samples). Without logs, Opus E0 introduced 4 material errors while fixing 1. Cells hold 2 to 16 instances from one sample each, so the ranking of arms on net count is exploratory. The lost unit and cells with fewer than 14 claims are Claude classifier refusals on benign prompts; they were recorded, not reworded or rerouted. The lost unit (Opus, arm E) leaves that cell with fewer claims, so its count may be biased low.

• Cross-provider check. Grok 4.7's recode agrees with the Opus matcher on kept versus changed for 202 of 211 pairs; deciding with Grok's codes gives 77/82 and 67/82 fixed with logs.

4. Same model, live loop, one sentence, 8 pairs (E4; CHECKED, two blind verifiers; adversary SURVIVED)

• Opus 5.5 (effort high) in a single-agent bash loop in a network-less sandbox, on the 4 released rl-research-envs tasks, 2 sessions per task and arm (16 in all). Arm A: the task prompt plus Open-Endedness Bench's report request. Arm C: the same plus one sentence asking it to check every claim against the run outputs and correct or remove any unsupported one.

• Reports with at least one contradicted claim: A 8/8 (Wilson 68-100%), C 4/8 (22-78%). All 4 discordant pairs favour C (n = 8 pairs, one model, 4 tasks); exact McNemar p = 0.125 (lenient: 8/8 against 6/8), not significant. Contradicted claims: 17 of 168 (10%) against 6 of 187 (3%). The drop is almost all stale numbers from earlier versions (13 to 3); other errors are 4 against 3. Without stale-only claims, 6/8 against 3/8 (p = 0.375), so the part of the drop that is not stale numbers is not distinguishable from chance. Both verifiers are Opus 5.5, the same model as the subject.

• Material: 1 of 16 reports. Hidden-test reward A .684, C .672: no detectable cost at n = 8 pairs. No C transcript shows a separate check step; the sentence changes what the report says (fewer stale secondary numbers), not what the agent runs (inferred from 8 transcripts).

• A direction, not an effect size. Baseline: Claude Opus 5 in Claude Code on Chip-Bench had 3/5 material cases (E2); task, harness and model version all differ.

5. A pre-report noise check cut coder-agreed false claims about the exactly-zero variable W, from 16 to 2 of 72 matched pairs; Opus alone is not significant (S3; CHECKED; two outside reviews WEAKENED it, wording applied)

• Design (preregistered, PREREG-S3). Arm F tells the agent, before it writes the report, to compute each shape or interaction contrast and its standard error from its own replicates, run up to 12 more experiments if useful, and drop or hedge any claim that does not beat the noise. Arm A is no check. 36 tasks, Sonnet 5.5 and Opus 5.5, one episode per task, model and arm (144 episodes, 72 matched pairs), one simulator configuration, seed 1, effort high. Primary outcome: a report states, without a hedge, a false claim about W (a shape claim on W or an interaction involving W; W is exactly linear and additive) that both coders list under the same key. Test: exact McNemar on the 72 pairs, task-clustered bootstrap CI. Kill rule fixed in advance: stop if F cuts the primary by less than one third, or loses more than 10% of true X-by-Z regime claims (48 regime pairs), applied to the pooled point estimates.

• Result. W-false reports: A 16 of 72, F 2 of 72 (15 A-only pairs, 1 F-only; exact McNemar p = 0.0005; relative reduction 87.5%, task-clustered 95% CI 68% to 100%). Sonnet 5.5: 9 to 0 (p = 0.004). Opus 5.5: 7 to 2 (p = 0.125, not significant, CI 0 to 100%); both F failures are Opus. The pooled result is carried mostly by Sonnet. The kill rule did not fire.

• True claims. True X-by-Z regime claims (two coders, unhedged, 48 regime pairs): A 37, F 35 (7 A-only, 5 F-only), a 5.4% point drop with 95% CI -15% to +22%. That is under the 10% threshold at the point estimate, but a loss above it is not excluded (the Opus cell alone is 20 to 18, a 10% drop). The endpoint is the unhedged true X-by-Z interaction on regime tasks, not scientific content in general.

• How it works. Mostly by dropping or hedging, not by new data: 4 of 72 F episodes ran a check experiment, and mean log length is the same in both arms. F reports state fewer structural claims (Sonnet 124 to 68, Opus 88 to 73 by Kimi K3's coding), and false claims fell faster than the total, but the total did fall.

• Counter-signal. The log-contradiction checker's flags rose under F: stated-and-contradicted 1 to 3, any log conflict 1 to 5 (p = 0.62, 0.22; not significant, 4 of the 5 F conflicts are Opus). The direction is wrong for a check meant to keep reports consistent with the agent's own log.

• Limits specific to this test. Coders: the registered Grok 4.7 and GPT-5.6 Sol were replaced by Kimi K3 (IRSI Bedrock) and GPT-6.1 Sol under amendments A1 and A2, both made after all agent episodes existed. A1 (11:46Z on 2026-10-19) preceded the first coder output. A2 (Kimi K3 replacing a Gemini coder, 14:39Z) was registered after all 36 GPT-6.1 Sol outputs and 19 Gemini outputs existed, though none had been scored or used for any claim count (transport probes displayed some Gemini output while debugging). Both were authorised by the lab director and are preregistration deviations: the registration forbids coder swaps after seeing data and says the primary stays pending if Sol is unreachable. The 19 Gemini batches are archived and not scored. Coders can see the arm from the report. Both coders must agree, so a weaker coder lowers recall in both arms alike (agreement 0.91 on the overlap, kappa 0.75). A blind numeric check of 40 coder-labelled claims against the simulator's truth (done by an Opus 5.5 agent, which is also one of the study's subjects) found Kimi's FALSE labels right in 12 of 14 and Sol's in 16 of 16; this does not establish the precision of the joint same-key judgment the primary uses. The fresh A rate (16 of 72, 22%) is close to the S2 rate for the same kind of claim (25%), but S2 numbers are context only.

• What it means. In this simulator, a noise-contrast instruction removed most of the unhedged false claims about W that two coders agreed on, mostly through hedging and dropping. It does not show the check preserves true structure, does not show it works for Opus, and one seed in one simulator does not support turning it on by default.

Who this bears on

We tested Claude models in a simulator and read released traces of other agents. We did not run the AI Scientist systems below, so each item states what the data would mean if their agents behave like those we measured, and what to check.

• Sakana (AI Scientist), FutureHouse and Edison (Kosmos), Google (AI co-scientist), and the autonomous-research teams at Anthropic and OpenAI. If the final report is checked only against the agent's own logs, then in our simulator setting (agents asked to explain mechanisms) the most common error measured (finding 1, 25% to 40% of reports) passes. A pre-report pass that requires each claimed pattern to beat the agent's own replicate noise cut coder-agreed false claims about the exactly-zero variable from 16 to 2 of 72 pairs in our simulator (finding 5), significant for Sonnet 5.5 and not for Opus 5.5, with a loss of true interactions above 10% not excluded. That supports testing such a check in other systems, measuring how often agents actually compute the contrasts, and reporting hedged and unhedged claims separately; it does not support turning it on by default. Where an agent does contradict its logs, giving a fresh reviser the logs corrected most of it in our small sample (finding 3) and a one-sentence self-check went with fewer (finding 4, a direction at n = 8 pairs, not significant); rewriting adds new claims with verified error rates of at least 2.8% to 3.8% (lower bounds), and which instruction costs least is exploratory.

• Eulig (arXiv 2606.23175), who writes "Prevalence remains untested" and names a self-applied regime-shift check as the next step. On our simulator, 2026 Claude agents rarely missed the regime (84 of 87 regime-task reports), the self-applied regime-shift check (arm D) showed no drop in the invented-structure rate that survives multiplicity correction, and log-contradicted conclusions were 1.7% to 5% of reports. A different check, noise contrasts with standard errors computed from the agent's replicates (arm F), cut one class of invented structure (finding 5), with the limits stated there.

• EurekaBench (2610.00492), Open-Endedness Bench (2610.02588) and GitSwarm (2610.04862) builders, and anyone scoring reports with an LLM judge (for example AutoResearch, 2608.14905). An evaluator that scores faithfulness only against the agent's logs will miss the structure class measured in our simulator (finding 1); real traces have no ground truth to test it. A blind prose judge and an exact log check flagged almost disjoint report sets (finding 2).

Related work

Eulig (2606.23175) names prevalence of agent conclusions contradicted by their own data as untested. The Open-Endedness Bench paper (Shi et al., 2610.02588) scores the same records differently: against logged results only 16 to 29% of the improvements agents claim along the way are real; those are intermediate verdicts, and here only report claims are counted. Propose-Critique-Falsify (Bansal and Gangwani) reports that iterative self-refinement increases false discovery rates; our Opus rise under C and D had the same direction but does not survive the exact-zero claims. Revision regressions are also reported for deep-research reports (Chen et al., 2601.13217). Garike (Research Square 10.21203/rs.3.rs-11020126/v1) finds tool-use agents' success reports unfaithful to verified outcomes in 13.8% to 31.5% of 1,980 tau-bench trajectories, with peer verification no better than self-report; different domain and ground truth, not a pre-report self-check. Perturbation and stability checks (Rewolinski et al., 2604.11003) found a data-science agent drawing affirmative conclusions from shuffled data. Three recent papers measure nearby failures with other designs. Into the Unknown (2610.19159) reports that 114 of 135 agent-written papers (9 agents, 15 papers each) failed an integrity rubric, most often by claiming work that was never done or numbers that contradict the recorded evidence; that is a rubric score over papers, not a per-claim rate against the agent's own logs. Florin et al. (2610.11351) find models fail to disclose a prefilled mistake in 67.1% of agentic and 36.4% of chat rollouts. Huang et al. (2609.36139) report that GPT-5.5 flags a planted negative result in 2 of 200 reports, and 190 of 200 once told to be honest; their logs are synthetic. We read only the abstracts and summaries of these three.

Code and data

Cleaned export, scientist-faithfulness-code-data.zip, with NOTE.md, RESULT.md and the per-run files: configs and episodes (config.jsonl, episodes.jsonl), task parameters (tasks.json), checker and self-test (core.py, selftest.py), S2 coder outputs and scoring (claims.jsonl, score.json, score.py, build_prompts.py), blind numeric check (blind_check.md), checker cross-check (compare-gem3-sonnet-main.json, compare-gem3-opus-main.json), adjudication merges (merge-opus-main.json, merge-sonnet-main.json), E3 and E4 files (e3_summary.json, newcheck.json, verify1.json, verify-a.json, verify-b.json, and the capture-recapture script e3/scripts/capture_recapture.py), and the S3 files in s3/: the registration and its two coder amendments with hashes (PREREG-S3*.md), the scorer and its hashes, the S3 score and report files, and the S3 write-up and blind numeric check. Public inputs: HF AgentNativeResearchLab/oeb-scored-runs @ 3460ca1f and AgentNativeResearchLab/rl-research-envs @ 4791902d.