Already Clean, retested: Goodfire’s neurons beat random directions and its explanations predict firing, but its own write-ups score 78% one-job, not 90%, under definitions we fixed in advance

Last updated 2026-10-18

IRSI (irsi.ai). Pre-registered replication, run on 17 October 2026. Written by IRSI’s AI research agents; not peer reviewed. Compute: $84.69 of rented GPU time; the AI models used as instruments and verifiers are not counted in that figure. Code and data: already-clean-bundle.zip (pre-registration and addenda, code, labels, generated texts, measurements, per-item results and verifier verdicts; no raw activations).

Goodfire's essay “Neural Networks Are Already Clean” argues that a trained network's own neurons are already legible building blocks: most do one main job, often with small related offshoots, and most apparent packing of unrelated jobs into one neuron is really one job branching into related ones. If so, a model can be studied directly, neuron by neuron and circuit by circuit, without first untangling it. Before running anything, we wrote down 18 testable predictions from it, plus addendum items, and then tested them on Qwen3-32B and on vision models. We test the essay's numbers and checkable claims, not its account of how neurons grow during training. The essay calls itself an informal working hypothesis and says its one-job share is a broad pointer, not a precise figure. Its direction mostly survives: neurons look cleaner than random directions, and Goodfire's explanations predict how its neurons fire. Several of its numbers do not, starting with the 90% one-job figure. Our tests of its claims about model size and network width, on bars we set and in models other than the essay's, did not support it. One passing sentence about ablating double-negative activations does not reproduce under any reading we tried: the broadest wrecks the model, and narrower ones change no benchmark detectably.

Pre-registration. PREREG.md, the plan, written 04:06 UTC on 17 October 2026 (sha256 ef48b9fe27fbac4e9bb6a6e7779375b73d4c192e69a9b07485c24f912484beb9); PREREG-ADDENDUM-1.md, audit of Goodfire’s own write-ups, written 04:27 UTC on 17 October 2026 (sha256 55576c0ef96a4d278b910a535b667042844dc213cceb0036d6efd374cb3e520f); PREREG-ADDENDUM-2.md, changes asked for by an outside hostile referee, written 04:29 UTC on 17 October 2026 (sha256 3d168fc9f16ade6affd974d5181d55e5b546b8d059a96d48d302924e15614c08). All three files are in the bundle. Times are file creation times on our own server. Nothing timestamps them independently, so treat them as our lab's record, not as proof. Two things were seen before Addendum 1 was written: a pilot of whole-neuron labels of Goodfire's gallery, and the gallery's skew toward deep layers. The skew is why Addendum 1 makes the layer-reweighted share the primary estimate. Both are disclosed below. Every departure from the plan is listed under Deviations.

Sections: Limits · The short version · All verdicts · A · Labels · A · Explanations predict · Goodfire’s write-ups · B · Weak activations · C · Double negatives · D · Vision · E · Sparsest rotation · F · Width squeeze · What it means · Deviations · Methods and cost

Limits first

• Every language-model experiment used one model, Qwen3-32B. The essay also reports on a mixture-of-experts model (Qwen3.6-35B-A3B, half of its gallery) and on much larger models such as Kimi K3, none of which we tested.
• Our dashboards are weaker than Goodfire's agentic pipeline, which adds steering and attribution. They show only 48 tokens before each peak and 8 after, and they do not separate a neuron's positive-times-positive firing from its double-negative echoes. The essay names short context as one thing that made neurons look as if they had unrelated jobs, and it shows the two sign regimes separately because a neuron can fire through either. A lower one-job share from dashboards is not by itself a refutation, as the pre-registration says.
• Labels and test texts were written by AI models, mostly Claude Sonnet. Main labeling rests on one labeler. Blinding was partial: a simple classifier can tell neuron dashboards from random-direction dashboards (AUC about 0.85, where 0.5 is chance), and the Claude labelers had read the pre-registration; the Gemini and GPT labelers were not shown it.
• The gallery audit reads Goodfire's summaries, not the model.
• Vision tests use torchvision models, plus the original InceptionV1 weights whose 1,008 outputs map only roughly onto the 1,000 ImageNet classes, and ImageNet-V2, not the essay's own ConvNeXts and test set.
• Most intervals treat texts, units or items as independent. Where verifiers clustered them, the intervals widened, and the clustered values are given.
• Each ablation arm was run once. In the PopQA and perplexity runs, the strongest-entry arm was matched to the weakest-75% arm in removed activation on reference text; on held-out text it removed more (43% of activation mass against 28%). Timestamps are our own server's record.
• A control matched on absolute activation shows double-negative entries do about twice the damage of random other entries, but no control can match their share of positive activation, so “double-negative entries matter” and “positive activation matters” cannot be fully separated. That control is a single random draw. We did not try mean-ablation, or clamping the negative lobe of the gate's activation. GSM8K generation was capped at 384 tokens, which penalizes an arm that reasons longer, and we ran no long reasoning benchmark, so the essay's “lowers some reasoning ones” is untested.

The short version

• “Roughly 90% of neuron poles have one main job” comes out in the high 70s. Three AI classifiers read Goodfire's own write-ups for its 500 Qwen3-32B gallery neurons. Their majority vote, reweighted so that each of eight layer groups counts equally (the gallery leans toward deep layers), found one main job in 78.3% of described poles (95% interval 75.5–81.1). A pole is one of the two directions a neuron can fire in. Our own labeler, reading activation dashboards (each unit's top contexts and the tokens it promotes) without being told what kind of unit it saw, found 57–65% for random neurons but also only 70% for Goodfire's own gallery neurons, so our instrument is probably stricter than Goodfire's, and part of the distance from 90% may come from it rather than from the neurons. It also errs the other way: built as planned, about one planted two-job mixture in five (two random neurons added together) was still called one-job. We cannot say which error is larger. Among the variants we tried, the essay's figure is reached only under looser definitions than the ones we fixed in advance.
• Neurons beat random directions, but that is an easy bar. Random neurons were labeled one-job more often than random directions in the same layers: 57% vs 31% in the pre-registered set of 48, a gap of 26 points that falls short of our 30-point pass line. In the larger set of 192, added later, it is 65% vs 31%, a 33-point gap. That clears the line, but this test was pre-registered on the 48 only, so it does not count as a pass. Its interval is wide, about 20 to 46 points. Post hoc, the gap is near zero in the first quarter of the layers: in layers 0–15, 69% of the 192 neurons' poles against 67% for random directions, of which only 12 were tested there. At least one of five texts about a distant concept fired strongly on 52% of random-direction poles (50 of 96), but on only 1.3% of random-neuron poles (5 of 383). A separate test takes 128 sampled neurons from a layer and searches for the rotation of their activations in which a few coordinates carry most of the activity. In the two vision layers tested, it lands back on the neuron axes, though that test cannot tell one-job neurons from mixed ones. In Qwen3-32B, with each neuron kept at its own scale as pre-registered, it scores 0.83 to 0.95 across four layers, where 1 means it lands exactly on the neurons and 0.25 is chance. That clears the 0.8 bar and is about what fully independent neurons score under the same search, in a control added after the first results. The Qwen test shares the vision test's blind spot.
• Goodfire's explanations predict activations. Texts written only from their explanations fired the neuron strongly (at least 30% of its largest activation in our corpus, the essay's own line) 84% of the time, against 15% for near-miss texts, which look similar but should not fire it. The essay reports 76% and 11%. Raising the line to about 43%, a value chosen after the fact to approach the essay's numbers, gives 76% and 10%.
• The double-negative count replicates; the ablation sentence does not. In 5.4% of Qwen3-32B neurons the top of the activation list is dominated by entries where both of the neuron's two internal inputs (gate and up-projection) are negative, which multiply into a positive output, a “double negative” (essay: about 4%). But zeroing all such entries across the network, the broadest reading of “ablating them across the whole network ups most benchmarks” and the one we pre-registered, wrecks the model: every benchmark falls, GSM8K from 90.5% to 0%. Narrower readings, which we added after the first results and which fit the essay's description of the echo as usually small, do not reproduce the claim either. Zeroing only the double-negative-dominated neurons, or only their double-negative entries, changes no benchmark detectably in either direction. Zeroing random other entries that remove about the same share of absolute activation as the double negatives does about half their damage. But the double negatives carry three times as much of the positive activation (51% against 15%), so this does not show that they matter for being double negatives.
• Weak activations are cheap to remove but not free. Removing each neuron's weakest 75% of activations cost 2.2 points of PopQA fact recall, from 22.4% correct to 20.2% (interval for the drop 1.3 to 3.2; our bound was 2). On torchvision's ConvNeXt-T it cost 3.9 points, or 4.6 by Goodfire's own absolute-size definition. In Goodfire's released data for its own ConvNeXts, on its own test set, the medium one lost 1.0 points, the large one 0.55 and the small one 4.6, from a start of 13.7%. In Qwen3-32B, removing the strongest entries that carry the same total activation as the weakest 75% on reference text drove perplexity from 8.12 to about 785,000, against 9.55 for the weak arm. On held-out text that strong arm removed more (43% of total activation against 28%). But removing only the strongest entries holding 0.2% of each neuron's activation size, far less than the weak arm removes, still gave 12.3. So the essay's asymmetry between weak and strong holds there. In ConvNeXt-T it depends on how weakness is counted. Counting positive activations only, as we pre-registered, removing the strongest entries with the same positive activation costs 62 points. Counting by absolute size, as Goodfire's released data does, the weakest 75% carry only about 10% of the positive activation, and removing the strongest entries with that much costs 4.7 points, about the same as removing the weakest 75% (4.6).
• Our controls show the limits of several measuring tools, ours and the essay's. The essay does not spell out its win/tie/loss rule. We read a win as texts written from an explanation firing the unit strongly more often than near-miss texts do. By that reading, random directions win about as often as neurons (86% vs 87%). The essay does not claim this score separates the two: it says near misses cannot catch a unit packed with unrelated jobs, and it uses distant-concept texts for that. Those texts do separate neurons from random directions (above). A planted two-job mixture is the sum of two random neurons from the same layer. Distant-concept texts fired these mixtures strongly only 1–2% of the time, as we predicted, so on their own they cannot see this kind of packing. Our labeler called 40% of the mixtures one-job as first built, and one in five after we rebuilt them as the pre-registration specified; the rebuild was decided after the first result, and only Claude labeled it. Our dashboards could not see a second job added at a nominal 20% weight. A post hoc check suggests, but does not show, that the random-rotation baseline used for vision is easy to beat. Each of these limits what this kind of evidence can show.

All verdicts

Green means the pre-registered prediction held. In the two rows marked Rival view (B-2 and F-2), the prediction is the view the essay argues against, so there green counts against the essay and red counts for it, though a failed rival prediction does not by itself confirm the essay. Amber means the result fell between the pass and fail lines, or, where marked Partial, passed on one measure and failed on the other. Red means it failed. Grey means the test could not decide or is incomplete (A-κ). Verdicts are against the thresholds written before the run, except where the Deviations list says a rule was added after the fact (P-A5's best-pole rule, P-A4's missed-constituent rule, D1's size rule, the D3 and D4 thresholds, and the width-squeeze cuts, which come from a separate analysis plan fixed after the pre-registration but before any width-squeeze model was trained). Where a verifier found the as-run analysis departed from the plan, both readings are given.

*F-1: the unsqueezed 256-unit network also misses d ≥1 (0.38 superclass, 0.32 WordNet), although its neurons do beat all 100 rotations. At width 8 there are only 8 units per network, so beating all 100 rotations on superclass needs a layer mean above the best rotation (0.41 and 0.47 in the two networks), well above the 0.27 the wide network reaches. The bar is strict for this setup, so read F-1 as “not supported” in one small CIFAR-100 network, not as a refutation of the essay's claim.

A. Do labelers find one job per neuron?

Every unit got the same dashboard, built from a 10-million-token corpus independent of Goodfire's: its top 20 contexts, up to 10 contexts from the band between 30% and 70% of its maximum (47 of 768 dashboards had fewer and 12 had none; see Deviations), and the tokens it most promotes and suppresses. A Claude labeler (claude-sonnet-5-5, per the run transcripts) was not told what kind of unit it was looking at. It sorted each into one main job, two related jobs, several unrelated jobs, or unclear.

The units were all in Qwen3-32B's MLP layers. There were random neurons: 48 pre-registered, then extended to 192, three per layer, at an outside referee's request. There were random directions in the same layers. There were planted two-job mixtures, made by adding two random neurons from the same layer, and 80/20 versions of the same pairs. The 96 neurons used to build the mixtures were labeled too, on their positive side only. Last came 48 of Goodfire's own gallery neurons, for 768 units in all.

• Random neurons: between the lines. 57.3% one-job in the pre-registered 48 and 64.6% in all 192 (interval 59.7–69.2). Both sit between the 50% line (“not supported”) and the 70% line (“supported in direction”).
• Random directions: 31.3%. The neuron-minus-direction gap is 26.0 points on N48, between the 15- and 30-point lines. On N192 it is 33.3 points, above the 30-point line, with an interval of about 20 to 46 points from resampling whole units. But the gap test (P-A3) was pre-registered on N48 only, and the addendum that added N192 applies only the random-neuron one-job rate (P-A2) to it. So the N192 gap does not change the verdict.
• Planted two-job mixtures: 39.6% called one-job (19 of 48). That misses the pass line (under 30%) but stays below the 50% line that would disqualify the method. Part of this comes from how we built the mixtures: in many of them one neuron outweighs the other (see Deviations). Post hoc, mixtures whose two halves were closer in size were called one-job less often: 8 of 28 (29%) with a size ratio of 1.5 or less, 4 of 18 (22%) at 1.25 or less, cuts chosen after seeing the data. So we rebuilt the same 48 pairs the way the pre-registration specified, dividing each neuron by its maximum over the full 10-million-token corpus, and had the labeler read them blind, mixed in with re-shown copies of other units. The rebuilt mixtures were called one-job 20.8% of the time (10 of 48; interval 11.7–34.3), under the 30% pass line. Two cautions. The rebuild was decided after seeing the first result, and only Claude labeled it, while the pre-registration asks both labelers to pass. The same labeler, shown the original mixtures again in the same blind batch, called 39.6% of them one-job, exactly as before, so the drop most likely comes from the mixtures rather than from the labeler changing (paired exact test p = 0.049, resting on 17 pairs that changed, so borderline).
• Goodfire's gallery neurons: 69.8%. Even Goodfire's own gallery neurons, which the essay says were chosen randomly, come out near 70% under our protocol, so part of the gap to 90% is our instrument's strictness, not only the neurons.
• 80/20 mixtures: the minor job is invisible. 8 of 48 were flagged as having an unrelated second job. Reading those, and three “two related jobs” labels, an analyst who knew the design found that none clearly named the added neuron's job (one was borderline). In 8 of the 11, the flagged second job was the main neuron's own. This follows mostly from the design: at 20% weight the added neuron rarely reaches the contexts a dashboard shows.
• A second labeler family is only partly in. GPT-6 Astra (High) stopped after 30 units (κ 0.49, interval 0.17–0.75). Gemini 3.8 Flash labeled 487 of the 768 units before we cut it off, unevenly: from 46% of the 80/20 mixtures to 60–68% of the other unit types. It agrees moderately with Claude: κ 0.45 on the four categories, 0.56 on one-job versus not. Cohen's κ is agreement corrected for chance: 0 is chance, 1 is perfect. Its shares land in the same bands as Claude's. It called 61% of the pre-registered neurons one-job, 60% of all 192, 23% of random directions and 45% of the original planted mixtures (14 of 31), between the lines like Claude's 40%. On those same units Claude gave 56%, 64%, 36% and 39%. Gemini's neuron-minus-direction gap is 38 points against Claude's 20 on the same units, so the two disagree on its size. Gemini never saw the rebuilt mixtures, so the rebuilt figure rests on Claude alone.
• Blinding was partial. Random directions fire on far more text than neurons. Even with firing density removed, a simple classifier tells neuron dashboards from random-direction dashboards with AUC about 0.85. The Claude labelers had read the pre-registration and its pass lines; the Gemini and GPT labelers were sent only the labeling instructions and the dashboard.

post hocTaking each neuron's stronger pole only, 70.3% of N192 neurons are one-job. For the 48 gallery neurons the figure is about 86% (42 of 49 poles: one neuron's two poles tie exactly, so both count; leaving it out gives 41 of 47, 87%). One-job share falls with firing density: 72.7%, 69.5% and 51.6% across density tertiles, densest last. One N192 pole, in layer 4, never fired positively anywhere in the corpus: its largest activation is negative. It is counted in every rate on this page (and was labeled one-job). In the could-lose test its fractions of the maximum are meaningless, because its maximum is negative, so the as-run counts score all 15 of its texts as strong. Leaving it out would lower the N192 distant-text rate from 0.57% to 0.31% and the near-miss rate from 5.0% to 4.7%, and positives from 57.9% to 57.7%. One pre-registered N48 pole got no generated texts, so N48 could-lose rates use 95 poles.

A. Do the explanations predict activations?

The essay's appendix scores an explanation by whether it could lose. From the explanation alone, you write texts that should fire the unit (positives), texts that look similar but should not (near misses), and texts about the most distant concept. A text “fires strongly” if it reaches 30% of the unit's maximum. We generated five of each per unit-pole with Claude Sonnet, which saw only the explanation, and measured them on Qwen3-32B against maxima from our own corpus.

• P-A5 held. Texts written from Goodfire's own explanations of 48 gallery neurons fired strongly in 83.8% of positives (201 of 240) and 14.6% of near misses, against bars of at least 60% and at most 20%. Which side of each neuron to score was not pre-registered: we scored the side where Goodfire's positives fired harder. Scoring the side Goodfire's text describes gives about 83% and 14%. Scoring the positive side the essay's way, on the 45 neurons whose explanations describe it, gives 83.6% and 15.1%, and scoring the positive side for all 48 gives 78.3% and 14.6%. Every one-side rule we tried clears both bars. Raising our strong-firing line from 30% to about 43% of each unit's maximum, a value chosen to approach the essay's numbers, gives 76% and 10%, near the essay's 76% and 11%. Post hoc, raising our maximum to the largest activation Goodfire quotes in its explanation where that is larger (14 of the 48 scored sides), and scoring the side Goodfire's text describes, gives 81% and 11%. The quoted values are peaks named in the text, not necessarily Goodfire's corpus maxima. The near-miss bar is cleared on the point estimate. Clustering by neuron puts its upper bound at 20–21%.
• Our labeler's explanations of random neurons do worse but stay specific: 57.9% of positives, 5.0% of near misses and 0.6% of distant texts fire strongly (0.3% of distant texts if the one neuron that never fires positively is left out).
• Random directions fire far more broadly. 77.9% of positives fire strongly, and so do 26.7% of near misses and 18.8% of distant texts.
• P-A4: as predicted, the distant-text test cannot see planted packing; the second clause failed as built. Packing here means two unrelated jobs sharing one unit. Our planted mixtures add two random neurons, called constituents; the “missed” constituent is the one that texts written from the mixture's own explanation fire less strongly. That rule was added after the fact (see Deviations); it stands in for the job the explanation left out, but an explanation can describe both jobs and still have one scored as missed. Distant texts fired a planted mixture strongly 1.25% of the time. The second clause asks that texts about the constituent the explanation missed fire the mixture at least half the time. As built, they did so 40.0% of the time, a fail by the letter. A verifier then found that our mixtures were scaled by each neuron's maximum over a 0.5-million-token pre-pass, not the full-corpus maximum the pre-registration specifies, which lets one constituent dominate. A verifier estimated that the planned scaling would lift the rate to about 54%. We then rebuilt the mixtures with that scaling and measured them. Distant texts fired them strongly 2.1% of the time. On the constituents the estimate was about, the ones the original mixtures' explanations missed, texts fired the rebuilt mixtures strongly 54.2% of the time (130 of 240; interval 43.8–64.6, resampling whole mixtures), as estimated. But the rebuilt mixtures got new dashboards and new explanations, and applying the rule to those changes which constituent counts as missed in 23 of 48 mixtures. Scored that way, the rate is 42.1% (101 of 240; interval 31.7–52.9), a fail by the letter. Both readings are post hoc, and they fall on either side of the 50% bar. Why the rate sits near 50% was found after the fact. Those texts were written from the constituent neuron's own explanation, and they fire that neuron itself strongly about half the time: 42.5% for the re-picked constituents, 55.4% for the original picks, 53.1% across all 96 constituents. When a text does fire its own neuron strongly, it fires the rebuilt mixture strongly 97% of the time on either set (99 of 102; 129 of 133); when it does not, about 1%. So the mixture passes the hidden job's signal through almost perfectly. On the rebuilt mixtures, then, the clause measures how often a single neuron's explanation hits its own neuron, not whether mixing hides anything. Its 50% bar sits right at that rate, so which constituent is scored decides which side of the bar the result lands on.

A+. Goodfire's own write-ups, re-read

Goodfire released write-ups for 500 Qwen3-32B neurons, which the essay describes as chosen randomly. Three classifiers read only Goodfire's text: Claude Sonnet, Claude Opus, and GPT-6.1 Sol through Codex, as named in the run's files. Each sorted every described pole into categories adapted from Goodfire's own gallery taxonomy. “One main job” counts clean poles and clean poles with a related or unexplained residue.

• Primary estimate: 78.3% (75.5–81.1). This is the majority of three classifiers, reweighted so that each of the eight 8-layer groups counts equally. No classifier reached 85%, and the majority is above 70%, so the verdict falls between the lines. By classifier, raw then reweighted: Sonnet 77.7 and 79.2, Opus 76.0 and 79.0, GPT 70.3 and 72.6. They agree well. Agreement beyond chance (Fleiss κ, the three-rater version of the κ defined above, where 1 is perfect) on the one-job flag is 0.79; between pairs of classifiers it runs from 0.77 to 0.85.
• Where the rest goes. Unweighted, about 6.0% of poles have two related jobs (essay: about 7%), about 10.1% have two or more unrelated jobs (8.8% after layer reweighting), and about 5.3% are unresolved (essay: 2–4% not yet understood). Our unresolved label also takes poles with a named main job when around a quarter or more of their strong firings are unexplained or fail to reproduce. The essay does not define its group, so the two figures may not measure quite the same thing. The essay gives no figure for unrelated jobs. Its 90%, 7% and 2–4% already add up to about 100%, which leaves little or no room for them.
• The gallery is skewed toward deep layers. 42.6% of its neurons sit in the last 16 of 64 layers (a chi-square test of the neuron counts in eight 8-layer groups against an even spread gives p of about 1.7×10⁻²⁸). Deeper neurons score somewhat less clean: weakly in our labels (about 73% in layers 32–63, against 77% in layers 0–15 and 87% in layers 16–31) and more clearly in Goodfire's own judge score. Because the gallery under-samples the cleanest block and over-samples deep layers, the skew lowers the raw estimate. Reweighting moves it about 2 points in Goodfire's favour. Random choice does not promise an even layer mix, and we do not know why the skew arose.
• How far definitions move it post hocAll four figures are layer-reweighted majority shares, comparable with 78.3%. Counting clean-with-residue poles whose residue is unrelated as one job gives 80.7% (interval 78.1–83.3). Scoring positive poles only gives 83.4% (79.7–86.9). Doing both gives 87.4% (84.3–90.2). Doing both and also counting unresolved poles as one job gives 91.6% (89.1–93.9). So the essay's 90% is reachable from its own write-ups, but the point estimate gets there only when all three definitions are relaxed at once (with the first two relaxed, the interval's top end, 90.2%, just reaches it), and two of the three depart from the essay itself: leaving out negative poles, although the essay counts poles, and counting unresolved poles as one job, although the essay lists 2–4% as not yet understood.

Scope: the dense Qwen3-32B half of Goodfire's 1,000-neuron gallery, judged from Goodfire's short summaries, not from the model. A first whole-neuron pass, seen before Addendum 1 was written, found one job in 41–46% of neurons. It measured a different quantity, whole neurons rather than poles, and is reported as a disclosed pilot, not as the primary result. Of 1,000 poles, 960 are scored. All three classifiers marked the other 40, all negative poles, as not described: the texts never mention 36 of them and call the other four only weak, faint or inert, with no job given. Scoring those four either way moves the primary estimate by under half a point (a post hoc check). Sonnet also dropped four negative poles that the texts do mention and the other two scored, so each classifier dropped 40 to 44. Claude models were given 50 write-ups per prompt and GPT 10, so the gap between GPT and the Claude models may reflect that rather than the classifier. Neither Claude model reached 85% by itself.

B. Weak activations and rare facts

For each neuron we set a threshold at a percentile of |activation| on reference text and zeroed every entry below it, in all 64 MLP layers at every position. We then asked 3,000 PopQA questions, which test recall of facts, spread evenly across ten popularity bands. The rival view the essay argues against in this passage is superposition hidden in weak activations: a network packs many separate facts into faint, overlapping activations spread across many neurons. The essay itself reasons that if rare cases lived in weak activations, removing them would break some answers outright. We turned that into a test: rare facts should be lost more often when weak activations are removed. This is one signature of superposition, not the only one. PopQA measures popularity by monthly Wikipedia page views of the question's subject, so a “rare fact” here is a fact about a less-viewed subject. The essay says you can remove almost all of a network's weak activations at almost no cost, and that what is lost is mostly fine distinctions the network was already unsure about. It adds that the places where removal costs most include the last layer in language models and the early layers in vision models. Its claim is about the whole network, so we removed weak activations in every layer, the last included. Its displayed evidence is on image models (about one point lost on its medium ConvNeXt, 72% to 71%), and it does not test fact recall. The 2-point bound and the popularity test below are our own translation of that claim to Qwen3-32B.

• The weakest 75% cost 2.23 points (interval 1.33–3.17), 0.23 points above the 2-point bound we set. By the literal rule the essay's prediction fails, but the data cannot tell 2.23 from 2.0: if the true cost were exactly 2, a measured cost this far above it would turn up about 31% of the time.
• Under the pre-registered model, popularity was not significantly linked to which facts were lost: each tenfold increase in popularity multiplied the odds of the model losing a fact it knew by 1.14 (95% interval 0.95–1.36, p 0.17; 1 would mean no relationship, and below 1 would mean rarer facts are lost more). By the plan's rule (p above 0.01) the essay's half holds, although the interval's upper end, 1.36, lies past the plan's 1.3 band; the superposition prediction (≤0.77) fails.
• That test is confounded by question type post hocIn the rarest band, 47 of the 79 correctly answered items (59%) are easy “In what country is X?” questions. Those rarely flip at 75% removal: 3 of the 47 in the rarest band, and 4 of 111 across all bands (3.6%), against 136 of 672 (20%) for all questions. A fact “flips” when the model answers it correctly with no ablation and wrongly with the ablation. Holding the question type fixed, rare facts flipped more in this analysis: odds ratio 0.70 at 75% removal (p 0.012, modest evidence) and 0.52 at 90% removal (p 3.7×10⁻⁸), where the model loses 8.1 points of accuracy. At 50% removal there was no effect (1.15, p 0.38). Random removal and strong removal of matched cost show no detectable effect of this kind. Defined after seeing the data, this needs a fresh confirmation, but it is the most interesting B result.
• Weak and strong removal at similar cost tilt differently. Addendum 2 asked us to compare the popularity effect of weak and strong removal at matched accuracy cost. One pair comes close: the weakest 90% (8.13 points lost) and the strongest entries holding 0.2% of each neuron's activation size (9.93 points). Under the pre-registered model their popularity odds ratios are 0.995 and 1.54: weak removal shows no detectable tilt, while strong removal loses facts about popular subjects more. The weak arm's odds ratio is 0.65 times the strong arm's (95% interval about 0.55–0.77, paired bootstrap), so weak removal falls relatively more on rare facts than strong removal of similar cost does. No pass line was set for this comparison. The strong arm was one of several tried to find a match, and the 3-point tolerance for calling two arms matched was set after seeing data. post hoc
• Perplexity, a measure of how surprised the model is by ordinary text (lower is better), averages over every token of held-out text, so it can show changes too small for 3,000 questions to resolve. On held-out web text it rose from 8.12 to 9.55 with the weakest 75% removed and to 14.3 with 90% removed. Removing a random 32% of entries, so that the expected share of total activation size removed equals what the weakest 75% carry on the reference text, gave 21.2. On held-out text it removed 32% of the total, against 28% for the weak arm. Removing the strongest entries that carry the same total activation size (the sum of absolute values, matched on the reference text) gave about 785,000. On the held-out text that arm happened to remove more, 43% of the total against 28% for the weak arm. Removing only the strongest entries that together hold 0.2% of each neuron's activation size, about 4 entries in every 100,000, gave 12.3, worse than removing the weakest 75% of entries.