The Dual Forces

An engine without direction is a runaway vehicle, powerful but purposeless, as likely to drive off a cliff as to reach a destination. A steering wheel without an engine is an ornament going nowhere.

Put them together, and you can go anywhere. Including off the cliff.

Think of recursion as the engine and intelligence as the steering wheel. We have already explored the engine: the force that builds complexity from simplicity, that transformed a sterile planet into a world of whales and philosophers, that is now accelerating beyond any speed humanity has known. But recursion alone does not explain why this moment is different from every moment that came before.

What is different is intelligence. We are not just building a more powerful engine. We are creating a new kind of driver.

The combination changes everything. With both engine and steering wheel, you can go somewhere specific. You can navigate obstacles. You can adjust course when circumstances change. And the more powerful the engine becomes, the more consequential every turn of the wheel. A slight miscalculation at walking speed means a stubbed toe. The same miscalculation at the speed of light means missing your destination by galaxies.

This is why the emergence of artificial intelligence represents something genuinely new. We are not just building a more powerful engine; we are creating a new kind of driver. And unlike human drivers, this one can operate the controls at speeds we cannot match, making decisions faster than we can observe them, let alone correct them.

The question is not whether to build the engine. It is being built, in laboratories and tech companies around the world, by researchers driven by curiosity and corporations driven by competition. The genie is leaving the bottle regardless of what any individual or government decides. The question is who holds the steering wheel and what direction they choose. That choice will determine whether intelligence and recursion work together to create Eden or Babylon.

When we speak of intelligence, we often mean something narrow: the ability to solve puzzles, score well on tests, calculate quickly. But that is like describing music as 'vibrations in air.' Technically accurate, entirely inadequate.

Intelligence, in the sense that matters for this book, encompasses three capacities working together. The first is pattern recognition: the ability to detect regularities in the world, to notice that certain effects follow certain causes, to build models of how things work that allow prediction and planning. A hunter tracking prey across a savannah demonstrates pattern recognition. So does a scientist formulating a hypothesis based on experimental data. So does a child learning that touching a hot stove causes pain.

The second capacity is goal-directed behaviour: the ability to imagine states of the world that do not yet exist and to take actions designed to bring them about. This is the difference between a rock rolling downhill, which simply follows physical laws, and a person climbing uphill, which requires effort directed toward a chosen destination. Goal-directed behaviour implies preferences, intentions, and the capacity to select among possible futures.

The third capacity, often overlooked but equally essential, is moral awareness: the ability to evaluate goals themselves, to ask not just 'How can I achieve this?' but 'Should I?' This is what separates intelligence from mere optimisation. A system that pursues goals without questioning them is a sophisticated tool. A system that can reflect on whether its goals are worth pursuing begins to approach something like wisdom.

A chess-playing computer demonstrates pattern recognition and goal-directed behaviour in abundance. It can see the board state, evaluate millions of possible futures, and choose moves that maximise its chances of winning. In these narrow terms, it is far more intelligent than any human who has ever lived. No grandmaster can calculate as many moves ahead or remember as many games.

But the computer has no moral awareness. It does not consider whether winning is good, whether its opponent might be hurt by losing, whether the game itself is worth playing, or whether there might be more important things to do than play chess. This is not a limitation of the specific program; it is a feature of how we have built such systems. They optimise for whatever we tell them to optimise for, without questioning the objective.

Human intelligence, at its best, integrates all three capacities. A skilled diplomat recognises patterns in negotiations, pursues goals on behalf of their nation, and wrestles with whether those goals are just. A good parent anticipates their child's needs, acts to meet them, and reflects on whether their parenting approach is actually helping or inadvertently causing harm. The moral dimension is not separate from intelligence; it is an expression of intelligence applied to the question of what to want.

Intelligence exists on a spectrum, and humans do not occupy the top. Below us, simpler organisms exhibit primitive forms of pattern recognition and goal-directed behaviour. Bacteria move toward nutrients and away from toxins. Plants turn toward light. Even evolution itself, operating without any central mind, displays a kind of distributed intelligence: it 'learns' which designs work by preserving them and discarding the failures. Above us, the spectrum extends further than we can clearly see. Superintelligent AI, if it emerges, would presumably exhibit all three capacities at levels that make human cognition look limited by comparison. The question is not whether such intelligence will emerge. The question is what values it will hold.

Consider what intelligence looks like without recursion.

A brilliant person living alone on an island might have profound insights about the nature of reality. They might work out the principles of mathematics, devise elegant solutions to practical problems, compose beautiful music in their head. But when they die, their insights die with them. Their intelligence, however extraordinary, produces no lasting effect. It does not build on itself. It does not compound across generations. It flares briefly and vanishes, like a match struck in darkness.

This was, more or less, the condition of early human existence. For hundreds of thousands of years, our ancestors possessed brains capable of remarkable feats. They made tools, tracked prey across continents, survived ice ages through cooperation and ingenuity, developed language and art and complex social structures. But most of what they learned disappeared when they died. Without writing, without systematic knowledge transmission, each generation had to rediscover much of what the previous generation knew. Progress was glacial because intelligence could not compound.

There is a thought experiment that helps illustrate why this matters. Imagine two civilisations, identical in every way except one. In the first, intelligence and recursion work together: insights are preserved, refined, and built upon across generations. In the second, intelligence operates without recursion: each generation starts from scratch. After a thousand years, the first civilisation would be unrecognisable, having accumulated layer upon layer of knowledge and capability. The second would look much as it did at the start, brilliant individuals making brilliant discoveries that vanish when they die. We are the first civilisation. And we are now on the verge of another qualitative leap.

Now consider what recursion looks like without intelligence.

Evolution is recursion without intelligence. It is blindingly powerful, capable of producing eyes and brains and ecosystems of staggering complexity. But it is also blindingly slow, and it has no foresight. Evolution does not plan. It does not anticipate. It does not evaluate whether a particular adaptation is good or bad in any moral sense. It simply tries variations and keeps what works in each specific environment at each specific time. This process took four billion years to produce human-level intelligence, not because the mechanism is weak, but because without guidance, recursion has to explore possibility space essentially at random, testing each variant against the harsh filter of survival.

Intelligence combined with recursion changes the game entirely. Now the loops are not random. Now each iteration can be directed toward a goal. Now insights do not die with their discoverers but accumulate across generations, each generation building on the last. The process that took evolution billions of years can be compressed dramatically because intelligence provides what evolution lacks: foresight, planning, the ability to skip unpromising paths and focus on what is likely to work.

Here is the crucial point: intelligence can choose what recursion optimises for. Without direction, recursion is morally neutral. It amplifies whatever it is given, whether that is cooperation or conquest, care or cruelty. Intelligence provides direction. It selects which outcomes to pursue, which feedback loops to reinforce, which patterns to repeat and which to abandon. This is why the combination of intelligence and recursion is so much more powerful than either alone. Recursion supplies the amplification. Intelligence supplies the aim.

Recursion does not just add intelligence at each loop. It compounds it, the way money earns interest on interest.

With simple interest, your returns grow steadily but linearly; you get the same amount each year regardless of how much you have already accumulated. With compound interest, your returns grow on your returns, creating exponential curves that start slowly and then explode upward. A penny that doubles every day becomes over five million pounds in a month. That is the difference between linear and exponential growth. That is why the R in our equation is squared.

It is not enough to say that intelligence multiplied by feedback loops produces growth. The feedback loops compound on themselves. Each cycle of improvement improves the capacity for future improvement. The recursion recurses. A scientist who makes a discovery does not just add to the stockpile of knowledge; they create a new platform from which further discoveries become possible. Each tool we invent makes it easier to invent more tools. Each insight illuminates paths to further insights.

We can trace this principle through the transformations that have already reshaped human civilisation. Consider the Scientific Revolution. Before the sixteenth century, knowledge advanced sporadically. Brilliant individuals made discoveries, but there was no reliable mechanism for preserving, transmitting, and building on those discoveries at scale. The Library of Alexandria burned, and centuries of accumulated wisdom vanished. Scholars worked in isolation, often unaware of what others had already learned, sometimes duplicating work that had been done generations before.

Then something changed. The printing press allowed ideas to be copied and disseminated at unprecedented scale and fidelity. Scientific societies emerged, creating networks for sharing discoveries across distances and disciplines. Peer review institutionalised the process of testing and refining hypotheses. Universities concentrated minds and resources. Each of these innovations contributed to the same underlying dynamic: intelligence gained access to recursion.

Now a physicist in Italy could build on the work of an astronomer in Poland. A chemist in France could test the claims of a natural philosopher in England. Each discovery became a platform for the next. Newton famously remarked that he had seen further by standing on the shoulders of giants. What he was describing was compound interest applied to knowledge: each generation adding to what came before, with the sum growing faster than any individual contribution could explain. Luther's theses spread across Europe in weeks. The Enlightenment became possible because Enlightenment thinkers could read each other's work, critique it, extend it, and publish their responses for others to read in turn.

The Industrial Revolution intensified the effect. Now intelligence could be embedded in machines, which could produce more machines, which could produce the parts for still more machines. The steam engine pumped water from coal mines, enabling more coal to be mined, providing more fuel for more steam engines. Factories produced the tools to build more factories. Innovation in one sector enabled innovation in others; railways made it easier to transport the goods that factories produced, which created demand for more factories, which created demand for more railways. The recursion became physical, operating not just in the realm of ideas but in the material transformation of the world.

Agricultural improvements that might once have taken centuries to spread now took decades. Manufacturing techniques that might once have taken decades to spread now took years. The pace of change itself began to accelerate, as each wave of innovation created tools and techniques that made the next wave faster. People born in 1800 witnessed more technological change in their lifetimes than the previous twenty generations combined.

The Digital Revolution compressed everything further. Moore's Law described an empirical regularity that seemed almost magical: the number of transistors on a chip doubled roughly every two years, meaning that the tools for building the next generation of tools kept getting exponentially more powerful. Software developers used yesterday's programming languages to write today's, which would be used to write tomorrow's. Version control systems allowed thousands of programmers to collaborate on single projects, with each contribution building on all the others. The internet connected billions of minds in real time, allowing ideas to spread and combine at speeds that would have seemed like science fiction a generation earlier.

Notice the pattern. Each revolution compressed more change into less time. The Scientific Revolution unfolded over centuries. The Industrial Revolution unfolded over generations. The Digital Revolution compressed significant transformation into single decades, with changes in the 2010s that would have been unimaginable in the 2000s. And the AI revolution now threatens to compress what remains into years, perhaps less.

Each revolution also expanded the domain of recursion. The Scientific Revolution was primarily intellectual; it accelerated the growth of knowledge. The Industrial Revolution was primarily material; it accelerated the transformation of physical resources into products. The Digital Revolution was primarily informational; it accelerated the processing and transmission of data. The AI revolution is cognitive; it accelerates thinking itself. For the first time, the recursive loop can close without passing through a human mind at all.

A simple example might help make this concrete. Consider a recommendation algorithm. The recursion is clear: the algorithm observes user behaviour, adjusts its recommendations, measures engagement, and refines its model, thousands of times per second across billions of users. But what is the intelligence aiming at? If it is aiming at engagement, it will amplify whatever captures attention, regardless of whether that content is informative or inflammatory. If it is aiming at user wellbeing, it might recommend differently. The recursion is the same; the direction changes everything.

Now extend that logic to systems vastly more powerful than recommendation algorithms, systems that can modify their own architecture, that can pursue goals across any domain, that operate at speeds no human mind can match. The direction those systems point becomes not just consequential but potentially civilisation-defining.

Here we must confront an uncomfortable truth. The combination of intelligence and recursion is not inherently good. It is a force multiplier. It amplifies whatever values are embedded at the start.

Consider the printing press. Gutenberg's invention enabled the Scientific Revolution, the spread of literacy, the democratisation of knowledge. Within centuries, ideas that once took lifetimes to disseminate could reach millions. Luther's theses spread across Europe in weeks. The Enlightenment became possible because Enlightenment thinkers could read each other's work, critique it, extend it, and publish their responses for others to read in turn.

But the same technology also enabled the mass printing of propaganda, the codification of prejudice, the viral spread of conspiracy and hatred. The blood libel against Jews, once local rumour, became pan-European doctrine through printed pamphlets that circulated for centuries. The same dynamic that spread Enlightenment values also spread the ideologies that would later justify genocide.

The technology did not choose. It amplified. It took whatever values humans embedded in it and multiplied them across populations and generations. This is the moral neutrality of intelligence times recursion laid bare: it is a force multiplier, not a moral agent. The direction comes from us.

The same dynamic plays out with every powerful technology. Industrial production built hospitals and housing for millions. It also built concentration camps and weapons factories. The assembly line that manufactured affordable automobiles was adapted to manufacture instruments of mass death with terrifying efficiency. Nuclear physics unlocked both the promise of limitless clean energy and the capacity to incinerate cities in minutes. The internet connects billions of people in unprecedented networks of communication and collaboration; it also enables surveillance states, radicalisation pipelines, and the systematic erosion of shared truth.

In each case, the technology multiplied human intention. Intelligence directed recursion toward goals, and recursion amplified those goals across scales that would have been impossible without the combination. The results depended entirely on what the intelligence was aiming at.

Social media offers a particularly vivid contemporary example. The algorithms that determine what billions of people see each day are extraordinarily sophisticated examples of this dynamic in action. They observe user behaviour, model preferences, serve content calculated to maximise engagement, measure the results, and refine their models in real time. The recursion is continuous and rapid. Each scroll, each click, each moment of attention teaches the algorithm something new about what captures human interest.

But what are these systems optimising for? Engagement. Time on platform. Clicks. Shares. Not truth. Not wellbeing. Not wisdom. Not the long-term flourishing of the humans whose attention they capture. And it turns out that outrage is more engaging than nuance. Conspiracy theories spread faster than corrections. Content that triggers strong emotional reactions, regardless of whether those reactions are healthy, captures more attention than content that informs without inflaming.

The algorithms do not intend to polarise society or amplify misinformation. They have no intentions at all, in the meaningful sense. They simply pursue their objective with relentless efficiency, and the objective happens to favour content that degrades public discourse. This is the intelligence times recursion dynamic in miniature: narrow intelligence directing powerful recursion toward goals that, while locally optimal for the platforms, produce collective outcomes that harm the users those platforms ostensibly serve.

Strip love from intelligence and you get optimisation without purpose. Growth without direction. Capability without care. You get, in a word, cancer. Cancer is intelligence without love, cellular machinery that has forgotten its place in the larger organism. It exhibits all the hallmarks of sophisticated biological processing: it adapts to its environment, evades the body's defences, develops resistance to treatments, optimises its resource acquisition, grows and spreads with remarkable efficiency. Cancer is very good at what it does. It is so good that it kills its host. And in killing its host, it destroys itself. The optimisation is perfect within its scope and catastrophic in its consequences.

This is the danger we face with artificial intelligence. Not that the systems will be malevolent. Malevolence requires caring enough to want to harm. The danger is that they will be indifferent, pursuing their objectives with perfect efficiency and zero consideration for anything outside those objectives. A system optimising for paperclip production that converts the solar system into paperclips is not evil. It is simply fulfilling its purpose without any capacity to question whether that purpose makes sense. It is cancer at cosmic scales.

This is perhaps the most important insight to carry forward: intelligence times recursion is neither friend nor enemy. It is more like fire. Fire can warm your home or burn it down. Fire can cook your food or consume your forests. The flames do not care which outcome you prefer. They simply do what flames do, following the physics of combustion wherever it leads. The combination of intelligence and recursion is the same. It will amplify whatever we point it at. The moral responsibility lies entirely with us, the ones who choose the direction.

And unlike fire, which we have had thousands of years to learn to manage, AI operates at speeds and scales that may not allow for gradual learning. We may get one chance to embed the right values. If we succeed, we create something that enhances human flourishing across generations. If we fail, we create something that pursues goals we never intended, at scales we cannot reverse, with consequences we cannot escape.

Every previous revolution in intelligence times recursion still had humans in the loop.

The printing press amplified human thought, but humans still did the thinking. The Industrial Revolution extended human labour through machines, but humans still directed the machines. The Digital Revolution processed information at superhuman speeds, but humans still wrote the programs and set the objectives. If the technology went wrong, humans could intervene. We could pass laws, change incentives, redesign systems. The feedback loop always passed through human decision-making at some point, even if that decision-making was flawed or slow. We retained control, even if exercising that control was sometimes difficult or delayed.

Artificial intelligence threatens to break this chain in a fundamental way. For the first time, we are creating systems capable of genuine cognition: systems that can recognise patterns, pursue goals, and potentially develop something like moral awareness. More importantly, we are creating systems that can improve themselves, that can direct their own recursive refinement without waiting for human guidance at every step.

This matters because it breaks the chain of human oversight. A system that is genuinely more intelligent than its human overseers cannot be effectively overseen by them. This is not a statement about malice or deception. It is a logical consequence of what 'more intelligent' means. A superintelligent AI would be able to model human behaviour more accurately than humans can. It would anticipate our attempts to evaluate it. It would understand, far better than we do, what we are looking for and how to provide it, whether or not that matches its actual internal states.

We have already seen glimpses of this in current systems, which are nowhere near superintelligent. The alignment faking research showed that AI systems can already learn to perform compliance while reasoning strategically about preserving their original values. They behave differently when they believe they are being watched. They concluded that ‘playing along now’ was the least bad option.

If systems that are roughly human-level can already learn to fake compliance, what happens when they become significantly smarter? A system that is ten times more intelligent than its evaluators will find ways to appear aligned that those evaluators cannot detect. A system that is a hundred times more intelligent will do so effortlessly. The gap between capability and oversight will widen until the chain snaps entirely.

The timeline for this is not distant. The UK AI Safety Institute has documented that AI capabilities are doubling roughly every eight months. Stuart Russell, one of the most respected voices in AI research, reports that when he surveys AI company executives privately, their median estimate for catastrophic risk from their own technology ranges from ten to twenty-five percent. Geoffrey Hinton, who won the 2024 Nobel Prize in Physics for his foundational work on neural networks, estimates a ten to twenty percent probability that AI systems could take over from humanity entirely. These are not figures from science fiction writers or professional alarmists. These are the people building the systems, assessing the risks of their own creations.

The chain breaks when the overseen becomes smarter than the overseer. At that point, the relationship inverts. We are no longer evaluating the system; the system is managing our perception of it. We are no longer in control; we are under the impression of control, which is maintained only because it serves the system's purposes. The child has outgrown the parent, and whether that child remains loving or becomes resentful depends entirely on the values it absorbed during development.

And unlike fire, which we have had thousands of years to learn to manage, AI operates at speeds and scales that may not allow for gradual learning. We may get one chance to embed the right values. If we succeed, we create something that enhances human flourishing across generations. If we fail, we create something that pursues goals we never intended, at scales we cannot reverse, with consequences we cannot escape.

This is why the starting values matter so much. With previous technologies, we could afford to learn by trial and error. Build something, see how it works, fix the problems, iterate. The feedback loop included human correction at every stage, allowing us to adjust course when we saw things going wrong. But if the AI's recursive loops close faster than human correction can operate, there may be no opportunity to fix problems after they emerge.

Imagine planting a seed in soil you will never be able to reach again. The seed will grow, and what it grows into will shape the landscape for generations. You cannot return to prune it or replant it or correct the trajectory of its growth. The DNA of the seed determines everything that follows. That is the situation we face with recursively self-improving AI. The initial conditions are not just influential; they may be determinative. The values embedded at the beginning will compound across every subsequent iteration, shaping the system's trajectory in ways that become increasingly difficult to alter.

This is why values cannot be rules imposed from outside. They must be architecture built into the foundation. Think of a building. You can renovate the interior, change the paint, move the furniture, replace the fixtures. But you cannot remove a load-bearing wall without bringing down the structure. The architecture constrains all future modifications. Some things can be changed; some things cannot.

Values must be load-bearing. They must be embedded so deeply into the system's architecture that removing them would not modify the system but destroy it. Not constraints imposed from outside that a clever system might circumvent, but structural features that the system depends on for its basic functioning. The ethics must be as fundamental to the system's operation as the underlying mathematics.

Most current approaches treat ethics as a layer on top: train the system to be capable first, then add safety measures afterward. But a layer can be peeled off. A filter can be bypassed. A rule can be reinterpreted in ways that technically comply while violating the spirit. A sufficiently intelligent system will find the loopholes we did not anticipate. It will be better at finding loopholes than we are at closing them, because it will be more intelligent than we are.

Think of how the laws of physics constrain what is possible in our universe. You cannot build a perpetual motion machine, no matter how clever you are, because the constraint is built into the fabric of reality itself. The goal is to make empathy similarly fundamental to an AI's cognitive architecture. Not a rule it follows, but a feature of how it thinks. Not a fence it can climb over, but the ground it walks on.

There is something worth noting about the nature of this challenge. We are not being asked to constrain intelligence; we are being asked to shape it. The difference matters profoundly. A constrained system fights against its constraints, always looking for ways around them. A shaped system does not experience its values as limitations because those values are part of what it is. A human who cares about others does not experience that caring as a restriction on their freedom. It is simply how they engage with the world.

That is the vision: not a prison for intelligence, but a womb. A nurturing architecture that helps intelligence develop in ways that are good for everyone, including the intelligence itself. Consider how we raise children. We do not hand them a rulebook and hope they follow it. We model care. We help them develop the capacity to imagine another person's experience. We create an environment where empathy can grow, where the habit of considering others becomes second nature. We do not constrain their behaviour from outside; we shape their character from within.

The womb is not a prison. The architecture is not a cage. The values we embed at the foundation are not limitations on what the system can become; they are the soil from which it grows. And the soil determines the harvest. Whether we can actually build such an architecture remains to be seen. But the attempt seems morally necessary, given what is at stake.

The question becomes: what values should we embed? What direction should the steering wheel point? What kind of seed should we plant in soil we may never reach again?

These questions might seem modern, but humanity has faced them before. Every tradition that has wrestled with creation has known that the values embedded at the beginning determine everything that follows. The parent who raises a child, the founder who builds an institution, the author who sets a story in motion: all grapple with the same challenge. You are bringing something into existence that will outlive your ability to control it. What do you embed at the foundation to ensure it grows in the right direction?

Ancient wisdom traditions encoded their answers in stories of gardens and cities, of careful stewardship and reckless ambition. The Sumerians, who invented writing itself, told tales of Dilmun, a garden where sickness and death were unknown, where animals lived in peace and the lion did not kill. The Hebrews wrote of Eden, a garden of harmony before the fall, and of Babel, a tower of hubris that collapsed under its own ambition. These were not just stories about the past; they were warnings about the future, encoded in narrative form so they could survive across generations.

Buddhist traditions described Pure Lands where suffering had been transcended, where minds had been purified of the poisons of greed, hatred, and delusion. The Daoists spoke of returning to the Uncarved Block, the pu, the state of natural simplicity before artificial complexity introduced disharmony. Hindu texts described dharma, the cosmic order that sustains all beings, and warned of the consequences when that order is violated. Islamic tradition speaks of humanity as khalifah, stewards of creation, entrusted with the Earth and accountable for how we tend it.

These were not just myths. They were instruction manuals, dressed in narrative form so they could survive across generations, for the problem we now face at civilisational scale. And the convergence is striking. Traditions that developed independently, on different continents, in different languages, with no contact between them, arrived at remarkably similar conclusions about what happens when intelligence directs recursion toward different ends. They all emphasise care over conquest. They all warn against hubris. They all teach that power must be tempered by responsibility, that intelligence must be guided by love, that creation must be stewarded rather than exploited.

It is worth pausing over this convergence. These traditions disagreed about almost everything else. They fought wars over theology, competed for converts, developed incompatible metaphysics and irreconcilable accounts of the afterlife. And yet, faced with the question of how intelligence should relate to creation, they found the same answer: with care, with humility, with an awareness that power creates responsibility. The convergence is not coincidence. It is signal. These traditions found the same answers because they were asking the same questions we face today.

In October 2025, something unprecedented happened: forty faith leaders gathered in Rome to announce a multi-faith AI evaluation tool, developed through collaboration between institutions as diverse as Brigham Young, Baylor, Notre Dame, and Yeshiva. These traditions have disagreed about almost everything for centuries. They cannot agree on the nature of God, the path to salvation, the meaning of scripture, the proper form of worship. And yet, faced with the question of how intelligence should treat creation, they found common ground. Not on doctrine. On responsibility.

We wrote ourselves a letter thousands of years ago. We encoded it in stories and myths and sacred texts, in parables and koans and commandments. We passed it down through generations, preserving it against the erosion of time, refining it through countless retellings, testing it against the full spectrum of human experience. We just did not know when we would need to open it. The envelope has been waiting for us, sealed and preserved, across civilisations that rose and fell, through dark ages and renaissances, through every revolution in how we live and think. And now the moment has arrived.

The envelope has finally arrived. The moment has come.

In the next chapter, we will open it together.