IRSI evidence bundle: evaluation of the released C2C Qwen3-8B / Qwen2.5-7B-Instruct fuser
Prepared September 2026. Exploratory result, not a reproduction of the C2C paper's main table.

Files
  cells.csv            One row per question and arm: 8,211 questions x 6 arms = 49,266 rows.
                       status is "scored" or "failed" (failed cells count as incorrect).
                       correct_amendment3: our frozen parser (amendment3_parser.py).
                       correct_upstream_parser: extract_answer_from_content in C2C's
                       rosetta/utils/evaluate.py at commit ff3cdba6c35b497f8f5e2cede7de78498bb2e4cc,
                       lines 99-209, applied unchanged.
  amendment3_parser.py The frozen answer parser, applied identically to every arm.
  recount.py           Recomputes every number on the page from cells.csv. Standard library only.

Models (Hugging Face revisions)
  Receiver  Qwen/Qwen3-8B                 b968826d9c46dd6066d109eabc6255188de91218
  Sharer    Qwen/Qwen2.5-7B-Instruct      a09a35458c702b33eeacc393d103063234e8bc28
  Fuser     nics-efc/C2C_Fuser            f01fc3258b305e280e04c7238f4f2cf31b7dc70d
            subfolder qwen3_8b+qwen2.5_7b_Fuser/final

Arms
  R   receiver alone
  S   sharer alone
  P   sharer writes one sentence of background (no options, no solving), then receiver answers
  T   sharer may solve, with options; receiver asked to reason
  F   C2C fusion (released fuser)
  T0  same sharer text as T; receiver given the same short final-answer instruction as R and F

Data (evaluated membership)
  OpenBookQA      allenai/openbookqa test            468
  ARC-Challenge   allenai/ai2_arc ARC-Challenge test 831
  MMLU-Redux      edinburgh-dawg/mmlu-redux-2.0 test 5,599
  C-Eval          ceval/ceval-exam val               1,313
  Items used in earlier development runs, items outside 2 to 4 choices, and a
  32-item qualification sample per benchmark were removed before any outcome was seen.

Known differences from the C2C paper
  The paper's main table uses a Qwen3-0.6B receiver. It reports no accuracy for this 8B/7B pair.
  The paper caps multiple-choice generation at 64 tokens; this run allowed the full context,
  with a repetition stop added during recovery. The training data of this checkpoint is not
  documented. The original run stopped partway; the remaining cells were completed by recovery
  runs under the same frozen configuration. 24 cells failed (T: 20, T0: 4); F and R had none.

Raw model outputs are not included. Contact peter@irsi.ai.
