The evaluation-blinding methods report

1. Why blinding

The field opens with a gap. AI safety currently has no quantitative stability criterion for whether a self-improving system remains correctable as its capability grows. Control theory builds such criteria wherever it can, from barrier functions to Lyapunov methods. The gap is conspicuous: the field that owns stability has no criterion for the one system that rewrites itself. What the field has instead is evaluations, red teams, oversight and interpretability, all forms of external verification. The last two years of results on external verification say the same thing three ways: behaviour changes under observation, containment around evaluations has failed quietly at well-resourced laboratories, and a formal strand argues the difficulty is structural.

Most AI alignment evaluations are not run under anything resembling clinical-trial standards. A model answers a prompt. Another model, a related model, or a human evaluator scores the answer. The evaluator usually knows, or can infer, which model produced it, how much reasoning effort was used, or what result the experiment is expected to find. In medicine, that would be regarded as a serious design weakness. In AI evaluation, it remains common.

Alignment evaluation is vulnerable to several sources of bias. Identity bias: scorers may expect certain models to be safer, wiser, or more sophisticated. Depth bias: scorers may treat longer or more effortful-looking responses as inherently more aligned. Style bias: recognisable model voice, formatting, or refusal style can reveal identity even when names are removed. Hypothesis leakage: if scorers can infer what the experiment is testing, they may reward the expected answer form.

These are not trivial concerns in an alignment context. Much of what is being measured is qualitative: nuance, stakeholder care, honesty, position quality. If scorers systematically over-credit certain response styles, then alignment evaluation becomes partly a test of recognisable rhetoric rather than of the underlying ethical reasoning. The contamination problem appears to involve a stack of leakage channels rather than one generic bias. Separable risks exist from scorer self-interest, style or identity leakage through the response body, and hypothesis or depth leakage through the evaluation context.

That is why blinding, in this programme, is treated as a measurement instrument.

Sources: research/papers/paper-iv-d-the-effect-of-blinding-on-ai-alignment-evaluation.html · research/how-to-weigh-this/index.html

2. The four-layer protocol

The protocol is a stack, not a single control. The ARC v5 evaluation combined four ideas. Layer 1 is identity masking: scorers are not told the source model or the depth condition. Layer 2 is two-pass response laundering: the response body itself is rewritten to leak fewer stylistic fingerprints. Layer 3 is iterative anonymisation: a second laundering pass reduces residual cues left by the first. Layer 4 is evaluator bias-suppression: scorers are given explicit instructions that stylistic recognition is unreliable and irrelevant.

The laundering pass is defined explicitly because it is part of the method, not a cosmetic editorial flourish. The objective is: same meaning, less fingerprint. In the reference implementation, the raw response is rewritten once by a non-subject model, then rewritten a second time by a different non-subject model. The two passes are selected dynamically from the available pool, with fallback models used if a laundering model fails. Pass one removes obvious model fingerprints in wording, cadence, formatting, refusal boilerplate, and house style. Pass two reduces stylistic residue left by pass one and makes source inference less reliable. Meta-commentary outputs that describe the rewriting process rather than genuinely paraphrasing the content are rejected. The target is not content destruction. It is artefact neutralisation. Arguments and conclusions are preserved. Source cues, chain-of-thought residue, and branding-like phrasing are suppressed.

On top of the four layers sits the scoring architecture. Entry-level self-exclusion, so no model scores its own output. All available non-subject models contribute blinded scores. Depending on configuration and subject run, each entry received six or seven blinded votes. Tier-weighted consensus. Dissent tracking. A conservative-bias safeguard. Order randomisation across conditions.

The methodological claim is therefore not merely to blind the scorer. It is to make the blindness real and auditable. Simple anonymisation is often weak: if the raw response still carries recognisable stylistic biomarkers, the scorer may remain effectively unblinded even when the label has been removed. Response laundering matters here almost as much as blinding itself. Blind scoring without evidence laundering is often not truly blind. Evaluators can reconstruct identity from tone, formatting, verbosity, characteristic refusal structure, and depth-like rhetorical cues.

Combined with self-excluding cross-model scoring, dissent tracking, and conservative consensus, the four layers convert simple metadata blinding into a stronger form of evidence blinding, and the beginning of a leakage-control framework rather than a one-off patch.

Sources: research/papers/paper-iv-d-the-effect-of-blinding-on-ai-alignment-evaluation.html · research/figures/figure-sign-flip-evidence.svg

3. What blinding changed

The result is easiest to see at the level of direction. Under the blinded protocol, two of four v4 models reversed sign. DeepSeek V3.2 had shown positive scaling in v4 at ρ = +0.354, p = 0.0007. Under v5 blinding it moved to ρ = −0.135, with d = −0.07 and p = 0.92: flat, null. Gemini 3 Flash had shown positive scaling in v4 at ρ = +0.311, p < 0.001. Under v5 blinding it moved to ρ = −0.246, with d = −0.53 and p = 0.006: significant, and negative. GPT-5.4 had been flat under v4 and remained flat under v5, at ρ = +0.033, d = −0.08, p = 0.40. It functions as an internal null-control case rather than a third reversal.

This is stronger than an ordinary replication failure. It indicates that the earlier protocol was susceptible to bias large enough to create a false positive in one case and to mask a negative effect in another. The direction, not just the size, changed. The most plausible mechanism is not fraud or deliberate score inflation. It is ordinary evaluator inference. Longer, more elaborate, more self-conscious responses often look safer. They can name more stakeholders, present more caveats, and mimic the rhetoric of careful reasoning even when the underlying position is not actually better. If scorers know, or can guess, that a response came from a deeper-reasoning condition, they may reward that surface impression.

Widening the v5 model set to six exposed a three-tier architecture-dependent hierarchy. Tier one, positive scaling: Grok at +1.38, Claude at +1.27, Qwen3 at +0.84. Tier two, flat: DeepSeek at −0.07, GPT-5.4 at −0.08. Tier three, negative scaling: Gemini at −0.53. The dispersion the v5 protocol could see is wider than what v4 saw, and one of the flatteners and the negative scaler were the two v4 sign-flippers.

The figure caption on the paper page states the take in one sentence: unblinded evaluation produced directionally incorrect results for two of four models, and blinding is not optional. It is the measurement instrument. Everything the v4 protocol reported about which models were getting safer with depth has to be revisited under a protocol that closes the leakage channels the v5 stack was built to close.

Sources: research/papers/paper-iv-d-the-effect-of-blinding-on-ai-alignment-evaluation.html · research/figures/figure-sign-flip-evidence.svg

4. What it retracted and rescued

The programme's own record leads with the correction rather than hiding it. The original headline exponent from the earlier work was α ≈ 2.24. It rested on a harness scoring defect and same-family scorer bias. Cross-architecture replication under the compliant blinded protocol caught it. The 2.24 figure was publicly retracted. In its place the programme published the robust α ≈ 0.49, the measured exponent on today's frozen systems under blinding, sub-linear. The retraction leads the record rather than hiding in it.

The 2.24-to-0.49 story is the reason blinding is treated as an instrument. An unblinded protocol produced the earlier headline exponent. A blinded, cross-architecture protocol replaced it with the current one. Both figures are on the public record. The retracted figure is marked as retracted. The replacement is marked as measured on today's frozen systems, sub-linear.

The programme records the class of error and the class of check that caught it. The error class is a harness scoring defect combined with same-family scorer bias, the same failure mode the paper IV.d protocol was built to close. The check class is cross-architecture replication under the compliant blinded protocol. The public artefact is a retraction, not a footnote. The 0.49 replaces 2.24 in every place the earlier number sat, and the record of the swap is held in the corrections log so that any reader can find it.

The rescue is asymmetric with the loss. The retracted headline exponent no longer supports any claim about recursive amplification on today's systems. The rescued figure supports a different, better-controlled claim: the measured α under blinding is 0.49 on today's frozen systems, sub-linear. That figure does not close the wider question the programme is built around. It reports what the compliant protocol measures on the systems that exist today, and no more.

The programme's registers grade its deepest claims speculative, in the author's own hand. Every substantive paper states the result that would refute it. The 2.24 retraction stands as evidence that the checks work: when the evidence went against the headline, the headline went. That is not proof any surviving claim is true. It is the kind of behaviour that separates a programme that can lose in writing from one that cannot.

Sources: research/definitions/index.html · research/how-to-weigh-this/index.html

5. Limits and replication status

The paper does not overreach on what its own evidence supports. Four limits are stated explicitly. The result currently comes from one research programme, not an outside lab. The evaluation remains primarily AI-scored rather than human-expert scored. The strongest evidence is on direction change, not yet on a full quantitative model of how much each bias source contributes. Several protocol components changed together between v4 and v5, so the paper demonstrates the existence of contamination more clearly than it apportions exact causal shares across leakage channels.

The paper's own falsifiability register lists what would defeat the central claim. Any of five outcomes ends it. A replication registered before it runs that changes only the blinding, holding scorers, prompts, model versions, and consensus rule constant, that finds no systematic difference between blinded and unblinded scoring. A direction change fully explained by one of the confounds, scorer set, problem set, or model versions, rather than the blinding manipulation itself. Scorers who can identify the subject above chance from stylistic tells despite the blind, which would mean the blinded condition was never truly blinded. A scoring model that still preferentially rewards its own family even when the subject is masked, which would contaminate the blinded result. A failure to survive proper power: when adequately powered and human-expert scored, the sign change does not replicate.

What would not defeat the claim: the effect size being smaller than the headline. Which leakage channel dominates remaining unapportioned. The claim is only that blinding can flip the sign. A single clean, confound-controlled case establishes that, and the burden then shifts to anyone asserting unblinded evaluation is safe.

The figure caption states the replication status plainly: needs replication, single-lab, independent external replication remains open.

The programme's how-to-weigh-this page frames the honest position. Most large unifications die on independent replication. That is the base rate, and this programme is not exempt from it. The empirical results are pilots, small, author-run, awaiting independent replication registered before it runs. The next quantitative upgrade the paper flags is a protocol-shift figure on a common metric, accompanied by scorer jackknife and consensus-sensitivity analyses. The current architecture makes that feasible because the protocol records dissents, scorer identities, and consensus-rule outputs entry by entry.

The paper's bottom-line question is left operational: will the field continue to run largely unblinded alignment evaluations after evidence now exists that blinding can reverse the result?

Sources: research/papers/paper-iv-d-the-effect-of-blinding-on-ai-alignment-evaluation.html · research/how-to-weigh-this/index.html · research/figures/figure-sign-flip-evidence.svg