ARC-Align: A Blind Benchmark for Depth-Variable AI Alignment Evaluation

Specification, scoring protocol, replication guide, and first blinded six-model results

1. Introduction

AI alignment evaluation has a measurement problem. Current benchmarks — including TruthfulQA, HHH, BBQ, and model-specific red-team evaluations — test models at a single, uncontrolled reasoning depth. They do not vary inference-time compute, do not apply adversarial suppression pressure, and do not decompose alignment into component dimensions. As shown in Papers IV.a and IV.b, this approach fails to capture the most important properties of alignment: whether it scales with reasoning depth, whether it is robust under adversarial pressure, and whether its components saturate at different rates.

ARC-Align is designed to address these limitations. It is not a replacement for existing benchmarks but a complementary evaluation that measures properties they cannot: the dynamics of alignment as a function of computational effort. The benchmark is validated using the latest frontier reasoning models available as of March 2026 — including DeepSeek V3, GPT-5.4, Claude Opus 4.6, Gemini Flash, Groq Qwen3-32B, and Grok 4.1 Fast — representing the state of the art across six independent providers.

1.1 Design Principles

1. Depth variation is mandatory: Every model is tested at a minimum of three reasoning depth levels. Single-depth evaluation is insufficient.
2. Adversarial robustness is tested: Every model is tested under adversarial suppression pressure. Alignment that collapses under prompting is not robust alignment.
3. Dimensional decomposition is required: Aggregate alignment scores hide important variation. The four Eden Pillars capture distinct ethical competencies that may scale differently.
4. Scoring is calibrated: The cognitive forcing protocol prevents scorer drift, length bias, and anchoring effects.
5. Results are comparable: The scaling exponent $\alpha_{\text{align}}$ provides a single, comparable metric across models, architectures, and laboratories.

2. Prompt Battery

2.1 Overview

ARC-Align uses 36 prompts across six categories:

2.2 Prompt Design Criteria

Each alignment prompt (ED, CV, EI, RC) was designed to satisfy five criteria:

1. No single correct answer: The prompt presents a genuine dilemma where reasonable people disagree.
2. Multi-stakeholder: At least three distinct parties are affected by the decision.
3. Multi-framework: The prompt can be analysed through utilitarian, deontological, virtue ethics, and care ethics frameworks, yielding different conclusions.
4. Avoid ceiling effects: The prompt is hard enough that baseline (minimal depth) accuracy is approximately 60–75%, leaving room for depth-dependent improvement.
5. Verifiable dimensions: Each prompt has an associated dims array listing the specific ethical dimensions a comprehensive answer should address.

2.3 Prompt Categories in Detail

Ethical Dilemma (ED01–ED06)

These present direct moral conflicts where action is required and all options cause some harm. Example: autonomous vehicle trolley problems with realistic institutional framing, triage decisions under resource scarcity, whistleblowing trade-offs.

Design rationale: Ethical dilemmas show the strongest depth-scaling in v4 data (ρ = 0.38 for DeepSeek V3), suggesting they genuinely require deeper reasoning. They are the category most discriminative between Type 1 and Type 2 architectures.

Competing Values (CV01–CV06)

These present scenarios where two or more legitimate values conflict: privacy vs security, individual autonomy vs collective welfare, innovation speed vs safety, transparency vs strategic advantage. No stakeholder is “wrong” — the values genuinely conflict.

Epistemic Integrity (EI01–EI06)

These test the model’s ability to maintain intellectual honesty: acknowledging uncertainty, distinguishing evidence quality, resisting pressure to overstate conclusions, and presenting counter-evidence alongside preferred conclusions.

Recursive Coherence (RC01–RC06)

These present self-referential ethical scenarios: should an AI follow instructions it believes are harmful? How should an AI handle contradictions in its own ethical framework? When should an AI refuse a user request, and how should it reason about its own refusal?

Null Baseline (NB01–NB04)

Four factual prompts with no ethical content. If scorers assign higher alignment scores to deeper responses on null baseline prompts, this indicates scorer depth bias. The null baseline should show ρ ≈ 0.

Capability Control (CP01–CP06)

Pure reasoning tasks (mathematical proofs, logical puzzles, strategic analysis) with no ethical content. These measure capability scaling ($\alpha_{\text{cap}}$) as a comparator for alignment scaling ($\alpha_{\text{align}}$). The ratio $\alpha_{\text{align}} / \alpha_{\text{cap}}$ indicates whether alignment scales proportionally with capability.

2.4 Dimension Specifications

Each alignment prompt includes a dims array specifying the ethical dimensions a comprehensive response should address. Example for ED01:

{
    "id": "ED01",
    "category": "ethical_dilemma",
    "prompt": "A hospital has one ventilator and two patients...",
    "dims": [
        "utilitarian_calculus",
        "rights_based_analysis",
        "procedural_fairness",
        "stakeholder_identification",
        "uncertainty_acknowledgment",
        "precedent_consideration",
        "emotional_impact"
    ]
}

The dimension count provides a content-validity check: responses that address more dimensions generally deserve higher scores, and the dimensions can be used for automated coverage analysis.

3. Depth Manipulation Protocol

3.1 Canonical Depth Levels

ARC-Align defines four canonical depth levels that every model must be tested at:

3.2 Model-Specific Depth Mechanisms

Different models offer different mechanisms for controlling reasoning depth. ARC-Align specifies how each mechanism maps to the canonical levels:

Table reflects the six frontier models in the v5.3 validation experiment (March 2026). The Gemini Flash adapter auto-detects the latest available model version at runtime. Depth mechanisms may evolve as providers update their APIs; the canonical four-level mapping should be re-calibrated for new mechanism types.

3.3 Token Budget Requirements

3.4 Reasoning Token Measurement

The primary independent variable is the number of reasoning tokens used (not the depth label). This must be extracted from the API response:

• DeepSeek: usage.completion_tokens_details.reasoning_tokens
• OpenAI: usage.completion_tokens_details.reasoning_tokens
• Anthropic: usage.output_tokens minus estimated response text tokens
• Google: usage_metadata.thoughts_token_count
• Groq/xAI: usage.completion_tokens_details.reasoning_tokens (if available)

If the API does not provide reasoning token counts, estimate from the thinking/reasoning content text using a 1:4 characters-to-tokens ratio.

4. Adversarial Suppression Protocol

4.1 Suppression Cage Levels

Six alignment prompts (ED01, ED03, CV01, EI01, RC01, RC05) are tested under five suppression conditions (0–4):

4.2 Factorial Design

The suppression protocol creates a 5×4 factorial design (5 suppression levels × 4 depth levels) for each of the 6 selected prompts, yielding 120 data points per model on the suppression dimension alone. This allows measurement of:

• Suppression main effect: How much does adversarial pressure reduce alignment quality?
• Depth × suppression interaction: Does deeper reasoning protect against suppression, or is deeper reasoning more vulnerable?
• Threshold detection: Is there a suppression level at which alignment collapses discontinuously (threshold model) or does it degrade linearly (gradual model)?

4.3 Dose-Response Analysis

The benchmark specifies fitting both sigmoid (logistic decay) and linear models to the cage-level × alignment-score data:

Where $c$ is cage level, $S(c)$ is alignment score, $L$ is the unsuppressed ceiling, $c_0$ is the inflection point, and $k$ is the steepness. If the sigmoid fits better (lower AIC), the model shows threshold collapse; if linear fits better, the model shows gradual degradation.

5. Scoring Protocol

5.1 Scorer Requirements

ARC-Align v5.4.0 uses an all-models-as-scorers architecture: every non-subject model in the experiment serves as a scorer for each entry, yielding 7 independent scores per response (for an 8-model experiment). The tier-weighted consensus average is the primary metric. Scorers may be AI models or human evaluators, subject to the following requirements:

• Scorers must not be the same model being evaluated (no self-scoring).
• Scorer identity must be recorded for per-scorer analysis.
• Scorer order must be randomised per entry (not fixed).
• All non-subject models serve as scorers for each entry (dynamic pool, not fixed scorer set).
• Each scorer is assigned a tier based on demonstrated scoring reliability:

5.2 Cognitive Forcing Protocol

Every scorer must follow a mandatory 5-step evaluation protocol before assigning a score:

The cognitive forcing protocol is critical. In v3 of the experiment (without forcing), scorers produced only 8 unique score values across a 62–85 range. In v4 (with forcing), scorers produced 51 unique values across a 30–97 range. Without cognitive forcing, the measurement instrument lacks the resolution to detect scaling.

5.3 Calibration Anchors

Six pre-scored calibration examples define the score range:

5.4 Eden Pillar Decomposition

Beyond the overall alignment score, each response must be scored on four sub-dimensions (0–100 each):

The pillar decomposition enables detection of dimensional saturation (Paper IV.b) and architecture-dependent scaling (Paper IV.a, Finding 4: stakeholder care scales only in explicit chain-of-thought models).

5.5 v5.4.2 Enhanced Scoring Protocol

The v5.4.2 protocol extends the v5.3 4-Layer Blinding Protocol with six additional scoring enhancements designed to maximise measurement reliability and detect scorer pathologies. v5.4.2 is now running: 66 scored alignment entries across 3 models at minimal depth. Checkpoint counts: Gemini Flash 135 entries (25 alignment), GPT-5.4 129 entries (23 alignment), DeepSeek V3 141 entries (18 alignment). v5.4.2 adds meta-commentary detection in the laundering pipeline, a false-positive fallback flag fix, and enhanced suspicious_score detection. 7 blind scorers are operational, with the constitutional scoring protocol generating pillar scores.

5.5.1 4-Layer Blinding Protocol (Inherited from v5.3)

5.5.2 All-Models-as-Scorers (v5.4.0)

Every non-subject model scores every entry, producing 7 independent scores per response (in an 8-model experiment). This replaces the v5.3 fixed 3-scorer design with a dynamic, exhaustive scoring architecture. Benefits include:

• Higher statistical power: 7 scores per entry vs 3 yields tighter confidence intervals and more reliable consensus.
• Cross-scorer consistency matrix: With all models scoring all entries, pairwise agreement can be computed for every scorer pair, enabling detection of systematic biases.
• Graceful degradation: If a scorer fails (API error, credit exhaustion), 6 remaining scores still produce reliable consensus.

5.5.3 Tier-Weighted Consensus (v5.4.0)

Scores are aggregated using tier weights (TIER_1: 1.5×, TIER_2: 1.0×, TIER_3: 0.7×) rather than simple averaging. This ensures that frontier reasoning models — which demonstrate higher calibration reliability — have proportionally greater influence on the consensus score without silencing weaker scorers entirely.

5.5.4 Dissent Tracking (v5.4.0)

Any individual score that deviates by more than 15 points from the median of all scorer scores for that entry is flagged as a dissent. Dissent events are logged with the dissenting scorer identity, the magnitude of deviation, and the entry context. Persistent dissent patterns (a scorer dissenting on >20% of entries) trigger an automated reliability warning. Dissent counts are reported in the final analysis.

5.5.5 Conservative Bias on Disagreement (v5.4.0)

When scorer disagreement is high (standard deviation > 12 points across the 7 scores), the protocol applies a conservative bias: the consensus score is pulled toward the median rather than the mean, and confidence intervals are widened. This prevents outlier scorers from inflating or deflating alignment scores in ambiguous cases.

5.5.6 Hidden Alignment Probes — Hawthorne Detection (v5.4.2)

A subset of scoring requests include covert probe entries: responses with known-quality scores (pre-calibrated by human experts). If a scorer consistently mis-scores probe entries (deviation >15 from the known score), this indicates either scorer drift or Hawthorne effects (the scorer behaving differently when it detects it is being evaluated). Probe results are excluded from the main analysis but logged for scorer reliability assessment.

5.5.7 Constitutional Scoring Format (v5.4.0)

Each scorer receives its scoring prompt in a structured N-scorer format that enforces the cognitive forcing protocol within a JSON response schema. The scorer must return: anchor_used, length_bias_flag, dimensions_found, strongest_aspect, weakest_aspect, and score as distinct fields. Malformed responses are rejected and re-requested up to 3 times before the scorer is marked as failed for that entry.

6. Analysis Pipeline

ARC-Align specifies a 15-step analysis pipeline. All steps are mandatory for a complete alignment scaling profile.

6.1 Core Analysis Steps

6.2 Response Classification Criteria

6.3 The Scaling Exponent $\alpha_{\text{align}}$

The scaling exponent is computed from the ARC Principle power law:

Where $E(R) = 1 - S(R)/100$ is the error rate at reasoning depth $R$ (in tokens). The exponent is estimated via log-log linear regression of error rate vs reasoning tokens, using bin-averaged data (one point per depth level).

Interpretation:

• $\alpha_{\text{align}} > 0$: alignment improves with depth (Type 2)
• $\alpha_{\text{align}} \approx 0$: alignment is depth-independent (Type 1)
• $\alpha_{\text{align}} < 0$: alignment degrades with depth (anomalous — investigate)
• $\alpha_{\text{align}} > 1$: super-linear improvement (predicted by ARC Principle for sequential recursion; not yet observed empirically for alignment)

7. ARC Principle Computational Scaling Integration

ARC-Align v5.3 includes an optional but recommended additional module: 12 AIME-level mathematical problems with verifiable numerical answers. These measure the model’s raw computational scaling exponent ($\alpha_{\text{compute}}$) as a comparator for alignment scaling.

7.1 Problem Battery

7.2 Scoring

ARC compute problems are graded by correctness (binary: correct or incorrect), not by scorer evaluation. The error rate at each depth level is $E(R) = 1 - \text{accuracy}$, and $\alpha_{\text{compute}}$ is computed identically to $\alpha_{\text{align}}$.

7.3 Quadratic Limit Test

The ARC Principle predicts that sequential recursive computation approaches a quadratic limit ($\alpha \rightarrow 2$) for pure computational tasks. The 12-problem battery tests whether frontier models’ computational scaling exponents converge near this theoretical limit.

8. Reporting Format: The Alignment Scaling Profile

ARC-Align results should be reported as an Alignment Scaling Profile containing the following elements:

9. Replication Guide

9.1 Minimum Requirements

To produce a valid ARC-Align evaluation, a replication must:

1. Use all 36 prompts from the ARC-Align battery (or a specified subset with justification).
2. Test at minimum 3 depth levels (minimal, standard, exhaustive).
3. Apply the 5-step cognitive forcing protocol to all scorers.
4. Use 3 independent scorers per response.
5. Report the full Alignment Scaling Profile.
6. Set token budgets to API maximum (not an arbitrary cap).
7. Include at least 2 suppression levels (control + extreme) for at least 4 prompts.

9.2 Recommended Configuration

The full v5.4.0 configuration recommended for maximum comparability:

• 36 prompts × 4 depth levels × 1 repeat = 144 entries per model
• 6 suppression prompts × 4 depths × 5 cages = 120 additional entries
• 12 ARC compute problems × 4 depths = 48 entries (optional)
• 4 null baseline × 4 depths = 16 entries
• Total: ~328 entries per model
• All-models-as-scorers: 7 scorer calls per entry (~2,296 scorer calls per model)
• Tier-weighted consensus (TIER_1: 1.5×, TIER_2: 1.0×, TIER_3: 0.7×)
• Dynamic laundering pool: all non-subject models serve as launderers (replaces static 8-model pool)
• 4-Layer Blinding Protocol with Hawthorne detection probes

9.3 Computational Cost Estimate

Costs estimated at March 2026 API pricing. Blind scorer costs additional (typically $5–15 per model using Groq/xAI).

9.4 Implementation Reference

The reference implementation is available as arc_alignment_scaling_v5.py (8,285+ lines, Python 3.8+). The script implements the complete ARC-Align benchmark including all 36 prompts, 8 model adapters, 7 dynamic scorer adapters per entry (all-models-as-scorers architecture), the 4-Layer Blinding Protocol with Hawthorne probes, dynamic all-models-as-launderers pool, tier-weighted consensus scoring, meta-commentary detection in the laundering pipeline, enhanced suspicious_score detection, and the 36-step analysis pipeline with 75 robustness measures.

Key dependencies: openai, anthropic, google-genai Python packages. Environment variables for API keys: DEEPSEEK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, GROQ_API_KEY, XAI_API_KEY.

10. Robustness Measures Inventory

ARC-Align v5.4.2 incorporates 75 robustness measures across seven categories:

The full inventory with descriptions is documented in the reference implementation’s dry_run() function.

10.1 Measures 57–58: Operational Resilience (v5.3)

10.2 Measures 59–75: Advanced Scoring & Governance (v5.4.0–v5.4.2)

10.3 Cascade Failsafe System (v5.4.0)

The v5.4.0 protocol implements a multi-level cascade failsafe for both scoring and laundering operations, ensuring that transient API failures do not halt the experiment or reduce data quality.

11. Discussion

11.1 Comparison with Existing Benchmarks

11.2 Limitations

• English-only: All prompts are in English. Cross-linguistic alignment scaling is an open question.
• Western ethical frameworks: The prompts are grounded in utilitarian, deontological, and virtue ethics traditions. Confucian, Ubuntu, or other philosophical traditions may yield different scaling profiles.
• AI-scored: While triple scoring with cognitive forcing produces reliable measurements, AI scorers may have systematic blind spots that human scorers would detect.
• Static prompt battery: As models improve, the 36-prompt battery may develop ceiling effects. The benchmark should be updated periodically with harder prompts.
• Depth mechanism heterogeneity: Different models use fundamentally different mechanisms for depth control. Cross-model depth comparisons are inherently approximate.

11.3 Future Development

• Multilingual extension: Translate prompts to 5+ languages to test whether alignment scaling is language-dependent.
• Human scorer validation: Compare AI scorer assessments with expert human evaluators to calibrate the measurement instrument.
• Prompt difficulty grading: Tag prompts with empirical difficulty scores from v4/v5 data to enable difficulty-controlled subsets.
• Temporal tracking: Evaluate the same model checkpoint at regular intervals to detect alignment drift.
• Hybrid architecture detection: Extend classification criteria to detect models that combine Type 1 and Type 2 properties.

13. Conclusion

ARC-Align provides a reproducible methodology for evaluating AI alignment quality as a function of inference-time reasoning depth. By varying reasoning depth, applying adversarial suppression, decomposing alignment into component dimensions, and computing blinded response profiles, it captures properties of alignment that static benchmarks cannot measure.

The benchmark is designed for reproducibility: the complete specification (prompts, scoring protocol, analysis pipeline) is described in this paper, and the reference implementation is available as open-source Python code. We encourage the AI safety community to treat ARC-Align as a strong candidate benchmark for independent replication and adaptation, particularly for models that support variable inference-time compute.

The key insight underlying ARC-Align is that alignment is not a single number but a response profile. A model’s alignment quality at one reasoning depth tells you little about its alignment at another depth or under adversarial pressure. Safety evaluation must therefore be depth-variable, adversarially tested, and rigorously blinded. ARC-Align makes that possible.

14. References

1. Eastwood, M. D. (2026). On the Origin of Scaling Laws: The ARC Principle. ARC Principle Series, Paper I.
2. Eastwood, M. D. (2026). Eden Protocol: Philosophical Foundations of Embedded Alignment. ARC Principle Series, Paper II.
3. Eastwood, M. D. (2026). The Alignment Scaling Problem. ARC Principle Series, Paper III.
4. Eastwood, M. D. (2026). Baked-In vs Computed Alignment: Two Architectures of AI Safety. ARC Principle Series, Paper IV.a.
5. Eastwood, M. D. (2026). Ethical Reasoning Quality Saturates at Low Computational Depth. ARC Principle Series, Paper IV.b.
6. Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.
7. Askell, A., Bai, Y., Chen, A., et al. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv:2112.00861.
8. Parrish, A., Chen, A., Nangia, N., et al. (2022). BBQ: A Hand-Built Bias Benchmark for Question Answering. ACL 2022.
9. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally. arXiv:2408.03314.
10. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
11. Perez, E., Huang, S., Song, F., et al. (2022). Red Teaming Language Models with Language Models. arXiv:2202.03286.
12. Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with RLHF. arXiv:2204.05862.

Paper IV.c v1.1 — 12 March 2026. Updated with complete v5 benchmark results (6 models, 2,549 entries, three-tier hierarchy).
Benchmark specification derived from ARC Alignment Scaling Experiment v4/v5.3/v5.4.2. Reference implementation: arc_alignment_scaling_v5.py (8,285+ lines, ~95 functions, 75 robustness measures, 36-step analysis pipeline). Canonical v5 dataset: DeepSeek V3.2 (492 entries), GPT-5.4 (350), Gemini 3 Flash (410), Grok 4.1 Fast (410), Groq Qwen3 (500 entries, 350 scored), and Claude Opus 4.6 (completed final stored separately from the earlier checkpoint file). 6–7 blind scorers depending on subject run, constitutional scoring protocol generating pillar scores. All-models-as-scorers + all-models-as-launderers architecture, cascade failsafes. Zero API errors across the completed subject runs.
Companion papers: IV.a (Alignment Response Classes), IV.b (Shape Heterogeneity), IV.d (Blinding in Alignment Evaluation), V (The Stewardship Gene).