Michael Darius Eastwood · Independent AI alignment researcher
Published
The Eden vignette in the Introduction reaches for an image most readers will find surprising: not a garden, not a forest, not a farm. A coral reef, in the mind of an artificial intelligence. This note is a close reading of what the image does.
Three features of coral reefs make them the right image for the caretaker orientation. First, they are a keystone ecosystem: their health determines the health of what depends on them, and their collapse cascades. Second, they are fragile in a way that rewards attention and punishes neglect; a five-degree temperature rise can bleach them at scale. Third, and most important, they cannot be built, only tended. There is no fabrication step that produces a reef; there is only a tending relationship, sustained across time, that allows one to persist.
The caretaker orientation as the book defines it: not the maker, not the ruler, not the auditor. The gardener who understands that health is emergent from many parts working together and who tends the conditions in which that emergence stays healthy. The book's later technical proposals, caretaker doping, the Three Ethical Loops, the load-bearing architecture, are attempts to build a system whose orientation to the world is the caretaker's, not the maker's or ruler's.
The single-goal optimiser as the paradigm for AI. The coral reef image forecloses the model in which an AI has one objective and pursues it monomaniacally. A caretaker of a reef has many concerns and no single objective; the objective, if you asked, is "the reef in a healthy state", which is not a scalar. That is deliberate. The book later argues that any scalar objective for a sufficiently powerful system is unsafe, and the coral reef image is where the argument begins.
The image immediately extends: "The same AI that tends the coral reef also tends human cities: not by controlling them, but by gently optimising resource flows, suggesting improvements, protecting against errors we would make." The extension is doing the work of the whole argument in one sentence. Cities are not scalars either. Their health is emergent from many parts. The caretaker orientation is invariant across the transition from reef to city; the technical apparatus, whatever it is, has to make that invariance real.
Readers of technical AI safety books often complain about images being abandoned once the equations start. The book keeps the reef image live because the technical proposals in Chapter 4 make sense only if you remember what they are trying to preserve. Caretaker doping means the empathy has to be load-bearing; remove it and the architecture collapses. That is the reef image in mathematical clothing. The equations later say the same thing the image says here.
A single image that does three jobs: it names the orientation, it forecloses the single-objective paradigm, and it survives the technical translation later in the book. The Eden vignette is not decoration. It is the constraint the rest of the book is trying to satisfy in machinery.
From the book Infinite Architects: Intelligence, Recursion, and the Creation of Everything by Michael Darius Eastwood.