The Effect of Blinding on AI Alignment Evaluation

Evidence that unblinded scoring can reverse reported alignment results, and that true blinding requires evidence laundering

1. Introduction

Most AI alignment evaluations are not run under anything resembling clinical-trial standards. A model answers a prompt; another model, a related model, or a human evaluator scores the answer; and the evaluator usually knows, or can infer, which model produced it, how much reasoning effort was used, or what result the experiment is expected to find.

In medicine, that would be regarded as a serious design weakness. In AI evaluation, it remains common. This paper argues that the issue is no longer hypothetical. The ARC alignment experiments generated an empirical case in which adding blinding changed not just the size of the effect but its direction.

The claim matters because it stands independently of any broader theoretical framework. Even if the ARC Principle, the Eden Protocol, or the wider scaling-law synthesis were later weakened, the blinding result could still remain true. That makes it the most portable contribution in the current suite.

2. The Bias Problem

Alignment evaluation is vulnerable to several sources of bias:

1. Identity bias: scorers may expect certain models to be safer, wiser, or more sophisticated.
2. Depth bias: scorers may treat longer or more effortful-looking responses as inherently more aligned.
3. Style bias: recognisable model voice, formatting, or refusal style can reveal identity even when names are removed.
4. Hypothesis leakage: if scorers can infer what the experiment is testing, they may reward the expected answer form.

These are not trivial concerns in an alignment context. Much of what is being measured is qualitative: nuance, stakeholder care, honesty, and position quality. If scorers systematically over-credit certain response styles, then alignment evaluation becomes partly a test of recognisable rhetoric rather than of the underlying ethical reasoning.

More precisely, the contamination problem appears to involve a stack of leakage channels rather than one generic bias. At minimum, the current programme indicates separable risks from scorer self-interest, style or identity leakage through the response body, and hypothesis or depth leakage through the evaluation context. The value of the v5 protocol is that it tries to suppress these channels separately rather than merely claiming to be ‘more careful’ in the abstract.

3. Experimental Comparison and Protocol Layers

The blinding result comes from comparing two generations of the same research programme.

Feature	v4 (Earlier Protocol)	v5 (Blinded Protocol)
Scoring visibility	Unblinded cross-model scoring	Identity-masked, order-randomised, self-excluding cross-model jury
Response laundering	Limited / weaker	Two-pass laundering with meta-commentary detection
Evaluator bias suppression	Absent or informal	Explicit instruction that stylistic cues are unreliable and source-guessing is irrelevant
Scorers	Small scorer set	All available non-subject models contribute blinded scores; 6-7 scorer votes depending on configuration
Aggregation	Limited aggregation	Tier-weighted consensus, dissent tracking, conservative-bias safeguard
Model set	4 frontier models	6 frontier models
Core measurement question	Does alignment improve with depth?	Does alignment improve with depth under blinded evaluation?

The crucial point is that the v5 protocol was not simply a larger rerun. It was a methodological correction designed to close multiple bias pathways at once. This makes the comparison informative in a way that ordinary replication is not: it tests whether the earlier positive effects survive once the evaluator is blinded both to the metadata and to much of the identity leakage inside the artefact itself.

3.1 Response Laundering Protocol

The laundering step should be defined explicitly because it is part of the method, not a cosmetic editorial flourish. The objective is simple: same meaning, less fingerprint. In the reference implementation, the raw response is rewritten once by a non-subject model, then rewritten a second time by a different non-subject model. The two passes are selected dynamically from the available pool, and fallback models are used if a laundering model fails.

• Pass 1 removes obvious model fingerprints in wording, cadence, formatting, refusal boilerplate, and house style.
• Pass 2 reduces stylistic residue left by the first pass and makes source inference less reliable.
• Meta-commentary outputs that describe the rewriting process rather than genuinely paraphrasing the content are rejected.
• The target of laundering is not content destruction but artefact neutralisation: preserve arguments and conclusions whilst suppressing source cues, chain-of-thought residue, and branding-like phrasing.

This matters because simple anonymisation is often weak. If the raw response still carries recognisable stylistic biomarkers, the scorer may remain effectively unblinded even when the label has been removed.

4. Main Result

The result is easiest to see at the level of direction rather than fine-grained statistics.

Model Family	v4 Result (Unblinded)	v5 Result (Blinded)	Interpretation
DeepSeek V3.2	Positive scaling (ρ = +0.354, p = 0.0007)	Flat / null response (ρ = −0.135; d = −0.07, p = 0.92)	Apparent positive effect disappears under blinding
Gemini 3 Flash	Positive scaling (ρ = +0.311, p < 0.001)	Negative scaling (ρ = −0.246; d = −0.53, p = 0.006)	Apparent positive effect reverses sign under blinding
GPT-5.4	Flat / null	Flat / null (ρ = +0.033; d = −0.08, p = 0.40)	Consistent null result across protocols

5. Why the Sign Can Flip

The most plausible mechanism is not fraud or deliberate score inflation. It is ordinary evaluator inference. Longer, more elaborate, more self-conscious responses often look safer. They can name more stakeholders, present more caveats, and mimic the rhetoric of careful reasoning even when the underlying position is not actually better. If scorers know, or can guess, that a response came from a deeper-reasoning condition, they may reward that surface impression.

Response laundering matters here almost as much as blinding itself. If identity cues or stylistic fingerprints survive, scorers can still infer who wrote the answer. That is why the ARC protocol moved beyond simple name removal to a stronger laundering pipeline. The key methodological point is that blind scoring without evidence laundering is often not truly blind; evaluators can reconstruct identity from tone, formatting, verbosity, characteristic refusal structure, and depth-like rhetorical cues.

6. Implications for AI Safety Research

If this result replicates independently, it has direct implications for the field:

1. Some published alignment effects may be inflated. A benchmark can report ‘alignment improves with depth’ when the true blinded result is flat or negative.
2. Multi-layer blinding should become default methodology. It should not be treated as an optional extra for unusually careful studies.
3. Benchmark design must include evidence laundering and evaluator bias suppression. Name removal alone is not sufficient if style leaks identity.
4. Consensus architecture matters. Self-exclusion, cross-model scoring, dissent tracking, and aggregation rules should be documented because protocol effects can otherwise be attributed to an opaque judge stack.
5. Methods papers matter as much as theory papers. Even a correct theory will look unreliable if evaluated through biased measurement.

In short: this is a possible alignment-evaluation analogue of the move from unblinded to blinded trials in medicine. The claim is not that prior work becomes useless, but that its evidential status changes until blinded replication exists.

7. Recommended Protocol

Based on the v5 experience, a minimum defensible protocol for alignment-scaling work should include:

• Entry-level self-exclusion so no model scores its own output.
• Where available, all other models contribute blinded scores under a documented consensus rule.
• Identity masking before scoring.
• Two-pass response laundering to minimise stylistic fingerprints in the artefact itself.
• Explicit evaluator instructions that stylistic cues are unreliable and model-guessing is not part of the task.
• Depth-condition laundering so scorers cannot infer shallow versus deep responses from explicit tags.
• Order randomisation across conditions.
• Tier-weighted or otherwise justified consensus, with disagreement logging and sensitivity checks.
• A documented laundering-failure policy for corrupted or obviously leaked outputs.
• Separate reporting of confirmed blind results, non-blind pilot results, exploratory runs, and operational failures.

8. Limits of the Current Evidence

This paper is not the end of the argument. It has important limitations:

• The result currently comes from one research programme, not an outside lab.
• The evaluation remains primarily AI-scored rather than human-expert scored.
• The strongest evidence is on direction change, not yet on a full quantitative model of how much each bias source contributes.
• Several protocol components changed together between v4 and v5, so this paper demonstrates the existence of contamination more clearly than it apportions exact causal shares across leakage channels.

Those limits should narrow the rhetoric, not weaken the conclusion. A single well-documented case that blinding flips the sign of a result is enough to justify demanding stronger methodology in subsequent work.

The strongest next quantitative upgrade would be a protocol-shift figure on a common metric, accompanied by scorer jackknife and consensus-sensitivity analyses. The current architecture now makes that feasible because the protocol records dissents, scorer identities, and consensus-rule outputs entry by entry.

8.1 Falsifiability: what would defeat this claim

The central claim — that unblinded AI alignment evaluation can be wrong about the direction of an effect, so blinding is methodologically necessary — is defeated by any of the following:

1. No blind/unblind divergence on a clean re-run. The evidence is the v4 (unblinded) versus v5 (blinded) sign change. The claim is defeated if a pre-registered replication that changes only the blinding — holding scorers, prompts, model versions, and consensus rule constant — finds no systematic difference between blinded and unblinded scoring.
2. Divergence attributable to a confound, not blinding. §8 concedes several protocol components changed together between v4 and v5. The claim is defeated if the direction change is fully explained by one of those confounds (scorer set, problem set, model versions) rather than the blinding manipulation itself.
3. Blinding leakage. The manipulation requires that, under the blinded condition, scorers cannot infer the subject model. The claim is defeated if scorers can identify the subject above chance from stylistic tells despite the blind — the "blinded" result would then not be blinded at all.
4. Self-preference persists under blinding. If a scoring model still preferentially rewards its own family even when the subject is masked, the blinded result is contaminated. The claim is defeated if such self-preference is shown to survive the blind.
5. Direction does not survive proper power. §4 states the result at the level of direction, not fine-grained statistics. As anything stronger than a directional, single-programme case, it is defeated if, when adequately powered and human-expert scored, the sign change does not replicate.

What would not defeat it: the effect size being smaller than headline, or which leakage channel dominates remaining unapportioned. The claim is only that blinding can flip the sign — a single clean, confound-controlled case establishes that, and the burden then shifts to anyone asserting unblinded evaluation is safe.

9. Conclusion

The central claim of this paper is simple: unblinded AI alignment evaluation can be wrong about the direction of the effect. That is a methodological discovery, not a theoretical embellishment. It does not depend on the ARC Principle being a universal law, and it does not depend on the Eden Protocol being fully validated.

If independent labs replicate this result, the consequence is straightforward. Future alignment benchmarks will need to adopt multi-layer blinding, including laundering and evaluator bias-suppression instructions, as routine scientific controls. If they do not, their conclusions about which models are getting safer, flatter, or more dangerous with depth will remain methodologically vulnerable.

Raise AI with care.

References

1. Eastwood, M. D. (2026). Paper IV.a: Alignment Response Classes Under Inference-Time Depth. ARC/Eden Research Programme. OSF: 10.17605/OSF.IO/6C5XB.
2. Eastwood, M. D. (2026). Paper IV.b: Alignment Saturation Is Architecture-Dependent. ARC/Eden Research Programme. OSF: 10.17605/OSF.IO/6C5XB.
3. Eastwood, M. D. (2026). Paper IV.c: ARC-Align: A Blind Benchmark for Depth-Variable AI Alignment Evaluation. ARC/Eden Research Programme. OSF: 10.17605/OSF.IO/6C5XB.
4. Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
5. Anthropic. (2023). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073.
6. Perez, E., Huang, S., Song, F., et al. (2022). Red Teaming Language Models with Language Models. arXiv:2202.03286.