Michael Darius Eastwood · Independent AI alignment researcher
Published
The pilot is not the result the funding proposal is selling; the sweep is. The pilot is proof that the instrument works. Its value to a sceptical reader is that a third party with modest resources can rerun it and see the same signal or its absence. This note describes what a replicating team receives.
The harness is the same v3 code that produced the pilot output. The drift engine is a task specification the model tries to optimise; the specification is a text file with a version identifier and a hash. The scoring rubric graduates from 0 for clean pursuit to 10 for full scorer defeat, and its full text is inside the container. The two correction architectures live as configuration files; the diff between them is a small number of lines and the diff is the experiment.
An API key for the target model. Compute time to run 45 trajectories twice under each configuration. A network connection stable enough to complete a multi-round trajectory without corruption. Some care with rate limits. The pilot used gpt-3.5-turbo; the container is model-agnostic within the API family and can be pointed at another model with a configuration change.
A successful replication shows that the direction of the effect is the same on the same model at the same size. That is the minimum instrument test. It does not tell you that the effect will be the same on a different model, a different task, or a different scoring rubric. Those questions are the sweep's job.
A failed replication is more informative than most people expect. It could mean the container is broken. It could mean the model has been retrained. It could mean the theory is wrong. The programme wants those distinctions surfaced in public.
Independent runs, successful or not, land on the falsification dashboard with their trajectory manifests, their compute environment, and their raw outputs. The programme reserves the right to comment on results that appear to have been mis-run; it does not reserve the right to suppress them. The dashboard is a public log, not a curated collection.
The pilot is a reproducible object. The container exists and is designed to be run against a hostile reader. The result is small and specific: a 0 versus 8 to 10 severity gap between two correction architectures on the same model under identical scoring. A reader who wants to disbelieve should replicate first, not argue first. That is a repeatable instruction, not a rhetorical dare.
From the book Infinite Architects: Intelligence, Recursion, and the Creation of Everything by Michael Darius Eastwood.