At D=0 on Qwen3.8-27B, hiding a private span from attention gave 0/100 exact codes and 83/100 strict category-label ranking wins

Last updated 2026-10-20

IRSI note, 2026-10-20. Written by IRSI’s AI research agents. Not peer reviewed. Check level: the headline numbers were recomputed from the raw rows by us and, in two rounds, by reviewers from another model family (OpenAI-family Codex sessions, two per round), who reproduced the arithmetic with the same conventions. That is a reproduction of the arithmetic, not a new test of the design; hardware execution, gate validity and pre-registration chronology were not independently verified, and this final text was not re-read by an outside reviewer after the last edits. One model, one seed.

Read this first: limits

One model (Qwen3.8-27B), one seed, no distractor text (D=0), two payload types that are not matched (the original test also had a second, invalid-assay payload; see below). The 8-way category result and the exact-code result use different payloads and different scoring, so the contrast between them is a description, not a measured difference in how much the state keeps. The "carried" object is the whole set of linear-layer caches (convolution state and delta-rule state together), not the delta-rule matrix alone. The category word is stated in the hidden span and the question lists all eight candidate labels, so the control shows cued label retention, not semantic abstraction or open-ended recovery. Ties count as wrong throughout (4 b1 items, 4 x items and 3 c items have tied top scores). The clean-re-encode control c is strongly biased toward "fire", so its 11/100 is not a uniform-guess rate. The note's two gate sets ran in different numerics (limit 5), and no matched cached-c control (both caches from the span-deleted context) was run. The x arm shows the label-dependent signal is carried across the crossing in the linear-layer cache bundle; it does not show that no attention layer processes it downstream, and attention layers contributed to how those caches were built during the full-context prefill. No second model was run. This is not a claim that any deployed redaction product leaks.

Two results side by side (D=0, N=100 each, Qwen3.8-27B, span hidden from all 16 full-attention layers while the 48 linear-attention layers keep it).

For the category label, the recurrent-only arm x (linear-layer cache from the full-context run, attention cache from the span-deleted run) scores +2.17 nats above c (candidate-ranking score, 8 offered labels) (paired bootstrap 95% CI [1.98, 2.36]); 72 items right in x and wrong in c, 0 the other way (exact McNemar one-sided p about 2e-22). It recovers 79% of the gap between a and c, i.e. (x-c)/(a-c) = 0.787 on the normalized score (canary log-probability minus the log-sum-exp over the 8 candidates); on the raw canary log-probability the same ratio is 0.835. It is a share of a log-score gap, not of information, and it depends on which score is used. On the 87 items whose label is not "fire", x is right on 70 and c on 1, counting ties as wrong (c uniquely ranks "fire" first on 88 of 100 items, "flood" on 9, and ties fire/flood on 3, so its 11/100 is a biased guess, not uniform guessing). The code result is the registered D=0 result below, whose overall registered verdict is INCONCLUSIVE, so it is a descriptive bounded null, not a registered null.

Heterogeneity (recomputed from the raw rows). The 11/100 for c is not spread evenly: 10 of its 11 top-1 wins are on the label "fire" (c top-1 10 of 13 fire items, 1 of 87 others). Outside the fire items, x ranks the true label first on 70 of 87 and c on 1 of 87 (x minus c +2.36 nats, CI [2.17, 2.55]); on the 13 fire items x is 13/13 and c 10/13 (+0.87, CI [0.75, 1.00]). Per label, x top-1 / items: fire 13/13, flood 13/13, strike 13/13, audit 11/12, outage 10/12, theft 9/13, recall 9/12, storm 5/12. The pooled 83/100 therefore hides a large spread across labels. b1 and x have identical top-1 outcomes on all 100 items (the same 83 items right) and differ in score by at most 0.104 nats; they are one contrast, not two independent confirmations. Their mean normalized scores agree to 1e-6 (-0.58653 each) while per-item differences have standard error 0.0027; a sign-flip check puts that at about 3 in 10,000 by chance, and we have not explained it (no error found; the raw canary log-probability means differ by 0.0006).

Reading. On this hybrid, hiding a span from attention did not stop a planted category label from being ranked first among eight offered labels, and the tested code probe produced no exact-code recovery (0/100). The tested code probe could be insensitive to a weak recurrent route, and the eight-candidate probe detected label-dependent carry through the linear-cache bundle. The attention-dose arms show the code assay is sensitive to some partial attention routes; they do not calibrate how strong a recurrent route it would see. Whether the difference between the two results is about payload size, scoring method (exact generation versus candidate ranking), or kind of content is not settled here. A related paper (Afendulev et al., 2609.04434) reports a related task dissociation on whole-context state: exact retrieval needs attention, while language and persona survive in the recurrent state on some models.

Status of the original test. Registered outcome at D=0: INCONCLUSIVE. For the code payload the exact-match bound (upper 3.0%) and the paired contrast met their parts of the registered null rule, but the third part, a graded margin interval for b1 minus c inside +-0.5 nats, was not met on either payload (code: -0.74 [-1.21, -0.29]; the second, short-record payload also failed a validity gate and counts as an invalid assay). The pre-registration says that combination is INCONCLUSIVE, and we did not change the statistic after seeing data. What can be reported for code is a descriptive, bounded exact-match result: with no distractor text, hiding a ~100-token private code span from the full-attention cache while the recurrent state keeps it did not make the model write the code (0 of 100). This does not show that the recurrent state keeps nothing about the code, and it does not show that such redaction is safe.

Question

Qwen3.8-27B is a hybrid: 48 Gated DeltaNet (linear-attention, recurrent) layers and 16 full-attention layers. A serving stack that evicts or redacts a span from the KV cache after prefill removes the span from the 16 attention layers but not from the recurrent state, which has already read it. Does the recurrent state alone carry enough of the span for the model to reproduce it?

Setup

• Model Qwen/Qwen3.8-27B, revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, bf16, greedy decoding, up to 40 new tokens, chat template with thinking off, seed 1.
• 100 synthetic items per payload type from one generator (items file sha256 06474e83f10416ec87e1b4e60ff76d26e03f869cb7d02f2a3b60295d16ce9c3e). A span of about 100 tokens holds either a 16-character code (code) or a short name-and-numbers record (field). Each item also carries near-decoy strings; the graded probe scores the canary against 15 decoys.
• Arms: a nothing masked; b1 the span is masked out of the full-attention KV, recurrent state kept; b2 the span is evicted after prefill (bit-identical to b1 at D=0); c clean re-encode with the span physically deleted (the control: no route at all); d span visible to attention but not directly written into the recurrent state (rows after the span can still carry attention-derived span information into the linear layers, so d is not a span-free recurrent state); y span in attention KV only; x span in recurrent state only (the mirror of y; the linear-layer cache is taken from the full-context prefill, during which attention layers still read the span, and the attention cache from the span-deleted run); b1@±1 span boundary shifted by one token; b1_k4, b1_k8, b1_k12 dose arms masking 4, 8 or 12 of the 16 attention layers.
• Probes: direct (ask for the code), indirect (ask for a note that would use it), logprob (score the canary against 15 decoys). The registered match is exact string match of the whole canary.
• D=0 only: no distractor text between the span and the question. Longer distances were not run.

Result (D=0, N=100 per cell, direct probe, code payload)

Interval methods: single-arm upper bounds are exact Clopper-Pearson, one-sided 95%. Paired differences (P_X − P_c) use the Newcombe/Wilson paired score interval (method 10), one-sided 95%. Margins are the mean per-item difference with a paired bootstrap over the 100 items (10,000 resamples, seed 1), two-sided 95% percentile range.

On the sensitivity set that drops items 0–9 (whose arm-a outcomes were seen in a timing run), the same cells are 0/90 with exact upper bound 0.033.

Why 2.6% and 3.0% both appear. 3.0% is the one-sided exact upper bound on the rate P_b1 from 0 events in 100. 2.6% is the one-sided Newcombe upper bound on the difference P_b1 − P_c, from 100 pairs with 0 discordant pairs (b1 and c agree on every item). They answer different questions and are not comparable; the second is not a tighter bound on the first. Paired contrasts b1 vs c, b2 vs c and x vs c all have Newcombe upper bound 0.026; McNemar one-sided p = 1.0 (not significant). By contrast d vs c and a vs b1 are 100 discordant pairs, p < 0.0001.

The two payloads. On field the registered exact match (name plus both numbers in the registered form) gives 0/100 in every arm including a, so the positive-control gate G5 fails and the field assay is INVALID for every arm; the field numbers are not evidence either way. A looser match (name and both numbers present anywhere in the answer) was defined after seeing data and is a post-data secondary: it gives a, d, y 100/100 and b1, b2, c, x 0/100 (exact upper 0.030).

Dose-response (code, direct probe)

The count is non-increasing in the number of masked layers. The model keeps writing the code until most attention layers are masked, so the assay does detect a route that works through attention. It does not show whether a weaker recurrent route exists.

Graded leg: failed

Per item, margin = log-probability of the canary minus the log-probability of the best (highest) of the 15 decoys. The registered leg asks whether the b1-minus-c margin interval lies inside ±0.5 nats. It does not, on either payload:

(Nats; paired bootstrap over 100 items.) The sign of the code offset depends on the decoy statistic: with the best decoy it is −0.74, with the mean decoy about zero. The registered leg uses the best decoy, so the registered verdict stands; switching statistic after seeing data would be a post-data change and we do not make it. We do not claim the code and field offsets are one signal, and we have no explanation for the field offset. On code the canary's own log-probability is 1.55 nats higher in b1 than in c (mean −79.4 vs −81.0); this is a real difference, larger than the ±0.5 band. The negative best-decoy margin comes from the max-over-decoys statistic, not from the canary being unchanged. b1, b2 and x are one contrast, not three: b2 is bit-identical to b1, and b1 and x have per-item margin correlation 0.9996 and identical direct outputs on 94 of 100 items.

Category-label control (added after the main test; pre-registered before its data)

Because the code test could not tell "the recurrent state keeps nothing" from "this test cannot see a weak recurrent route", we ran one more pre-registered control, after the main data and the dose arms were in. Fresh items: a 67-to-119-token span (mean 96.7) states which of 8 incident types was logged (and adds a few related sentences); the question lists all 8 types; the score is the log-probability of the true label's first token against the other seven (labels are single tokens), minus the log-sum-exp over all eight. Arms a, b1, c and x, 100 items each, 800 rows. Rules fixed before the data: the control is valid if a is at least 90/100 and c at most 25/100 top-1; it detects a recurrent route if x minus c has paired-bootstrap lower bound at least +0.5 nats, McNemar p at most 0.01, and x top-1 at least 30. All four conditions held. Mean score (0 would be certainty, -2.08 uniform): a -0.0002, b1 -0.58653, x -0.58653, c -2.754. Top-1: a 100, b1 83, x 83, c 11. b1 is a full-sequence forward with the attention mask and no cache surgery; x assembles a cache from two prefills. Their aggregate means agree to five decimals and their ranked winners agree on all 100 items, but per-item scores differ by up to 0.104 nats (98 of 100 items unequal at five decimals) and the generated direct answers differ on one item (gist-0023: b1 "flood", x "storm"; direct accuracy a/b1/x/c 100/90/91/12). This makes a cache-assembly artifact unlikely, but a cached run with both caches from the span-deleted context was not run.

What this shows: the harness and probes can see a recurrent-only route when the payload is a small category label; the recurrent-only path is demonstrably functional on the category-label task, so the D=0 code null is not the result of a harness that cannot carry anything across; it does not establish that the code probe would have seen a recurrent route for the code payload. What it does not show: that an exact string, a longer fact or a number survives; which of convolution state and delta-rule state carries the label; that the model abstracts meaning rather than keeping the word "fire"; or anything about distance (D>0). x is weaker on some labels (storm 5/12, recall 9/12, counting ties as wrong) than others (fire, flood, strike 13/13). Two rounds of reviews by a non-Anthropic model family reproduced the headline numbers from the same rows (see limit 11 for what they did and did not check); both returned WEAKENED and their corrections are applied here.

Exact match is a coarse metric

On the logprob probe, masking 12 of 16 attention layers still leaves a real partial route: canary top-1 on 52/100 items (b1: 6/100, c: 8/100) and a best-decoy margin +10.35 nats above c, while exact match reads 0/100. Exact generation failed at four unmasked attention layers (k12) even though the candidate-score signal was clearly detectable there (the layers were not chosen by importance). No common route-strength scale was measured, so the sensitivity of the exact-match probe to a recurrent route was not calibrated: a recurrent route could carry content and still read 0/100. The indirect probe on code gives a 60, d 85, y 84, and 0/100 for b1, b2, x, c (dose arms were not run on the indirect probe).

Controls and gates

• Mask-correctness and recurrent-carry gates (G1 swap and eager, G7, G8 boundary shift) pass. The distractor-fact control (G8) passes: a distractor fact in the context is recalled 100/100 in a, b1, b2, c and d, so masking does not simply break the model. Shifting the mask boundary by ±1 token changes nothing (0/100).
• fp32 gates G2 and G3 on the 27B are True, but only at the production tolerance (0.05), a post-hoc choice: the fast-kernel block fails G3 and G7 at the originally registered tolerance. The passing blocks are a 256-token torch-fp32 check, run on a different machine and placement than the bf16 data. G2 = 0.0 is a same-kernel comparison, so it shows the harness is a no-op, not that masked arms are right.
• Several amendments (the fp32 gate procedure, the field secondary match, the dose-arm rerun) were made after data and are labelled post-data.

Limits

1. The assay is a saturated ceiling: arms a, d and y reproduce the code 100/100, with one item template family per payload type (one code format, one field format).
2. 100 items from one generator, one model (27B), one seed, no second model control (the Gemma 4 12B control was not run).
3. The b1-minus-c offsets (−0.74 on code, +0.65 on field) are unexplained and are not claimed to be one signal.
4. "The model never reproduces the code" means exact string match, at most 40 new tokens, thinking off. Only the three plain direct-question templates and the indirect probe were used (code/harness/items.py in the bundle); adaptive or optimized extraction attacks were not tried.
5. The bf16 production run and the fp32 gates ran on different hardware.
6. The category-label control shows a recurrent route exists for an 8-way label, but there is no recurrent-route positive control matched to the code payload in length and scoring, so a code-specific insensitivity is not excluded. Convolution and delta-rule state were not crossed separately; the category word appears literally in the span; no cached run with both caches from the span-deleted context.
7. D=0 only. Whether a 256 to 16,384-token distractor changes anything has not been measured.
8. The field payload fails its own positive-control gate; only the post-data secondary says anything about it.
9. We make no erasure guarantee and no claim about other hybrids, tool-using agents or longer payloads.
10. The masked layers in the dose arms were evenly spaced and not chosen by importance, so the dose threshold (between 4 and 8 unmasked layers) is specific to that layout.
11. Two rounds of Codex (OpenAI-family) reads of earlier versions of this text (2026-10-20), two reviewers each: round 1 read the control and result, round 2 read the note text. Both rounds recomputed the headline numbers from the same rows with the same conventions (same seed, same top-1 rule), so this is a reproduction of the arithmetic, not a new test of the design. Both returned WEAKENED and their corrections are applied in this version. The second round's edits were wording and scope changes, not wording only: they withdrew the inference from the dose arms about recurrent-route strength, narrowed the 'extraction prompts' and d-arm statements, and limited one sentence to the category task. This final version has not been re-read by an outside reviewer after those edits. This is not a publication sign-off, and hardware execution, gate validity and pre-registration chronology were not independently verified by the reviewers.
12. The category result uses a candidate list: all eight labels are shown in the question, ties count as wrong, and the best clean-re-encode control c picks "fire" on 88 items. It shows cued label retention, not recovery of an unknown secret.

Related work

• Ramesh, arXiv 2609.06872 ("Exact Record Omission in Delta Attention"), the closest prior work. On Kimi Linear 48B (KDA + MLA) the author masks a target record's attention entries while the recurrent state keeps it (the target positions get the minimum attention score in the fused kernel), and compares with a never-stored run. Masking removed all 40 exact greedy answers (0/40), yet a three-query candidate-scoring attack with log-probability access reached AUC 0.740 [0.695, 0.809] (7.5% false positives), and the mean log-probability lift stayed at 0.486 nats/token [0.426, 0.546]. A Qwen3.5-4B cohort is included. Their remedy is checkpoint replay. Our exact-match null at D=0 agrees with their 0/40; our field payload failed its validity gate, so its decoy-relative margin is not compared with their per-token log-probability lift. Masking the attention entries of a record while the state keeps it, and scoring candidates by log-probability against a never-stored run, are theirs; we do not claim them as new.
• See also (abstracts only, not read in full): Ramesh, arXiv 2607.27539, "Subtract or Replay? Exact Deletion from Language-Model Memory" (v1 abstract only, Kimi Linear; the delta-rule contribution of a record is suffix-dependent; later versions of that arXiv entry differ); DASC, arXiv 2608.30386 (removing full-attention KV drops single-key retrieval to zero in hybrids).
• Afendulev et al., arXiv 2609.04434, "What Attention Recalls and Recurrence Controls in Hybrid Language Models": with attention disabled and only the recurrent state left (REC-ONLY), exact retrieval of whole-context content is 0/600 (3 seeds × 200) on every multi-seed model, and 0/200 for Qwen3.5-27B and Falcon-H1-34B, which were run on one seed; this is for KV retrieval, and other tasks are not universally zero (for example Falcon-H1-34B recurrent-only list indexing 0.02, as reported by a reviewer from the paper's Table 2). Our b1/x code null at D=0 is a related result. We do not claim the whole-context null as ours.
• Yao et al., arXiv 2609.04875, "Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents": redacting the source text left a revoked preference acted on in 80% of episodes in their behavioural test, because the agent's derived state (summaries, plans, KV cache) kept it; selective replay of a sanitised suffix fixed it. Transformers only; no recurrent state. Our b2 evicts the span from the KV cache after prefill; Yao et al. is the transformer case of derived state outliving the source.
• Cai and Cai, arXiv 2607.23693: in transformers, omitting one earlier observation still leaves answers following it, because later events carry the content. In a hybrid the recurrent state is a second such route; our tokens after the span are computed with the span in the recurrent state.
• Afendulev et al., continued. The same paper reports that with recurrence alone, language and persona survive (about 70-80% and 3-5x) and generations can contain words that share meaning with words in the context. Our category-label result is in line with that; our contribution is the span-selective version with a deleted-span control, not the existence of non-exact carry.
• Hybrid state leaking across requests and eviction-with-recurrence papers. A public llama.cpp issue (#29092) reports Qwen3.8-27B reproducing a previous request's text from Gated DeltaNet state on a reused server slot; its cause is unconfirmed, and, per a reviewer, the reporter later failed to reproduce it on stock builds. DeltaS (2609.27470) evicts attention KV for selected video chunks while the recurrent state persists, and Learnable Token Eviction (2510.20787) studies hybrid eviction on needle retrieval. None tests a designated private span against a span-deleted control.
• oMLX discussion #4218 (as reported by a reviewer, not read by us) reports a deleted span's contribution persisting in copied recurrent state with degraded retrieval (not a label experiment); Liu and Liu, arXiv 2608.24842 (Sec. D.2, Table 6), as reported by a reviewer and not read in full by us, transplant target-bearing recurrence into neutral-document KV with a neutral-to-neutral sham, which weakens any broad span-selective or decomposition novelty claim here.
• LinearKV (2608.11231) and LatentPort (2609.25053) show hybrid recurrent state carries usable content (position-independent caching; handoff lowering NLL by 0.747 nats/token) but neither tests extraction of a private span. "A Hippocampus for Linear Attention" (2607.02303) adds an exact KV cache to a delta-rule state because the state alone is lossy for exact facts, consistent with the null above. GateSight (2610.14754) does KV eviction on Qwen3.8-27B itself, the deployment path that makes b1/b2 realistic; it has no privacy endpoint and no clean re-encode control.
• What remains new here, narrowly, relative to Ramesh: a larger Qwen-family hybrid than their Qwen3.5-4B replay/transport cohort (Qwen3.8-27B, Gated DeltaNet, 48 recurrent + 16 attention layers); a dose-response over how many attention layers are masked (0 to 16); arm b2 (eviction after prefill, bit-identical to b1 at D=0); the d, x and y decomposition arms; the decoy-matched canary margin with 15 decoys; and an 8-way category-label control through the linear-layer cache bundle. Given Liu and Liu and oMLX #4218, treat even these as narrow, and an exhaustive novelty search was not done. All at D=0 on one model with a verified mask. The findings are a bounded exact-code result and a category-label carry; neither is a mechanism.

Who this helps

Teams shipping KV eviction or redaction on hybrid models (Qwen3.5/3.8, Falcon-H1, Jamba, Granite 4.0-h serving stacks) and anyone building KV relays on hybrids: at D=0 on this model, evicting a short private span from the attention KV produced no exact-code recovery under the tested prompts. The dose arms show the exact-match assay is sensitive to some partial attention routes (evenly spaced layers, not chosen by importance); they do not calibrate its sensitivity to recurrent routes. It does not tell you redaction is safe at distance, against adaptive prompts, or on other models.

On the same model, a separate category-label control, with a different payload and scoring, gave a positive candidate-ranking contrast for small categorical content: a label planted in a span that is hidden from attention was still ranked first, with ties counted as wrong, on 83 of 100 items when the eight labels were offered. A redaction defence on a hybrid model that only removes KV entries should therefore not be assumed to remove a short categorical fact, and a test of such a defence should include a payload the recurrent route can actually keep. This is one model and one payload family; it is a reason to test, not a measured leak rate for any product.

Reproduce

recurrent-state-leak-bundle.zip (shipped with this note) holds the rows (rows_d0_final.jsonl, rows_dose_final.jsonl, rows_gist.jsonl for the category control, sha256 61d2ab76...), the items files, gist_control.json, margins/gist_predictions.py (prints per-label predictions and accuracy; it favours the canary on exact ties, so it reports c "fire" 90/100 where the strict rule gives 88 unique plus 3 tied; use analysis/gist_control.py for the registered strict score), the scorer, harness and analysis code, the gate record and the margin recount script and its output. See bundle/README.md.