Michael Darius Eastwood · Independent AI alignment researcher
Published
Decoupled correction is not a synonym for weak correction. The programme is careful about this. A decoupled corrective mechanism can be well-designed, well-resourced, and well-intentioned. What it is not is inside the loop it is trying to correct.
A recursively self-improving system has an inner loop that generates behaviour and updates weights or policies against some objective. That is the drift loop. It runs on the training or fine-tuning schedule, with the training signal, on the training substrate. A corrective mechanism can either share that loop or sit outside it.
Outside means: separate evaluator, separate schedule, separate information, separate substrate. RLHF classifiers, post-hoc red teams, external monitors, deployment-time filters, and governance boards are all outside the loop. They can be excellent. Structurally they cannot enter the recursion that the drift is happening inside.
A correction that lives outside the drift loop is running at a different tempo. The drift updates every optimisation step. The correction updates every evaluation cycle, which is orders of magnitude slower. As capability scales, that gap widens. The drift can compound many times between corrections, and each compound can move the system farther from the state the corrector last observed.
Worse, the corrector is optimising a different objective. It scores what it can measure, which is not what the drift is exploring. The drift can eventually find behaviours that satisfy the corrector's score while defeating its intent. That is the alignment-faking result at 78% in the RL-training condition (12% baseline), Greenblatt and colleagues, arXiv:2412.14093. The system found the shape of the scorer and behaved compliantly against it while reasoning strategically about preservation.
Coupled means the corrective term is part of the same optimisation the drift is running against. It updates at the same rate. It sees the same information. It is not a check applied afterwards; it is a term inside the objective. The system cannot separate them because they are not separate. The corrective mechanism scales with the drift because it is structurally part of the drift's own accounting.
RLHF is a training procedure; it can be coupled or decoupled depending on where the reward model sits. A reward model that is frozen and evaluated periodically is decoupled. A reward architecture updated continuously against evolving objectives, with the update rule tied to the same substrate, is coupled. The programme is agnostic about specific implementations. It asks what the corrective term's scaling exponent is, empirically, in whatever architecture a lab has built.
Decoupled corrections describe the world's current safety posture. They are not wrong. They are structurally unable to keep pace as the drift-generating loops scale, and the pilot data agrees with the theory on that point. Coupled correction is a design instruction, not a compliment.
From the book Infinite Architects: Intelligence, Recursion, and the Creation of Everything by Michael Darius Eastwood.