The Convergence Paradox
Consider the following:
Scaling performance in a specified task is convergent.
Life is divergent.
The divergence in life is a consequence of solving the task of reproduction.
Convergence here is not used in the mathematical sense of converging to a single value. Rather, it’s used in the broader sense of converging from a vast number of solutions towards a smaller set, including those that have multiple local maxima.
Given that you grant all three statements, they seemingly create a paradox. If optimizing for a specified task is convergent, and life is optimizing reproductive fitness, how can life be divergent?
Let’s start by examining the obvious solution: one of the statements is false.
1. Scaling performance in a specified task is convergent.
This seems intuitively obvious. If we consider every action you could take with a car, most of them don’t result in successfully driving down the road. If we include other cars, switching lanes etc., the set of actions that achieve high performance shrinks further.
But what if we flip the task? If the overwhelming majority of actions don’t result in competently driving down the road, then surely there’s no convergence when we flip the task to not driving down the road?
Actually, there is, and we can logically guarantee that the principle holds generally.
Take the set of all possible sequences of specified actions and call it A. Let g be the task of driving down the road.
The set of actions that successfully drive down the road is A AND g.
The set of actions that successfully don’t drive down the road is A AND -g.
An AND intersection can never expand a set—the new set can only ever be smaller or identical. The more complex a task, the more AND intersections you add, the more convergent optimization becomes.
It might feel wrong to use the concept of convergence here, but it is convergence around a constraint. The convergence we intuitively think of—towards a singular solution or high similarity—is the result of massive clusters of constraints, often contained within a single AND statement. If our specified set of actions are steering operations of a car, then the overwhelming majority of sequences of actions do not result in competently driving down the road. Nevertheless, if the task is to never drive down the road—meaning no action within the sequence may do so—then the solution space reduces substantially.
The first statement holds: it’s logically guaranteed.
2. Life is divergent.
This is an empirical statement rather than a logical one. Life began as a single self-replicating system and, over time, diversified into single-cell organisms, plants, insects, reptiles, mammals etc.
Looking at the trajectory of life thus far, there is no question that it has diverged from its initial state.
The second statement holds: it’s empirically verified.
3. The divergence in life is a consequence of solving the task of reproduction.
At first glance this seems straightforward. It’s perfectly standard to treat reproductive fitness as the underlying optimization target of life, and evolution explains the resulting divergence. However, evolution requires something additional: mutation and recombination—these are the missing pieces that resolve our paradox.
Convergence occurs when you optimize over a fixed behavioral space.
Expanding the behavioral space creates divergence.
The simplest way to think of this is in AND intersections and OR unions.
Drive down the road AND within the speed limit AND do not block traffic. This necessarily converges on a smaller set of solutions.
Drive down the road OR within the speed limit OR do not block traffic. This necessarily expands the solutions—every additional OR creates divergence.
Mutations and recombinations can create new output mechanisms, new expressiveness, which expands the behavioral space and potentially introduces new solutions. This acts as an OR statement: you can do one of your previous solutions OR one of the newly introduced ones.
Evolution harnesses both dynamics. Mutations and recombinations expand the expressiveness, creating divergence, while the optimization of reproductive fitness prunes and creates convergence. Additionally, the environment isn’t uniform: different environments introduce different constraints on survival and reproduction, leading to diversification through having to solve distinct problems. This doesn’t break the principle that AND intersections are convergent—instead, divergence happens because each set of constraints, each distinct AND statement, converges on its own solution.
What does this mean for AGI?
Firstly, general intelligence itself is a massive cluster of AND intersections: solve visual problems and auditory problems and engineering problems and reasoning and empathy and factual accuracy and…
Each of these tasks is itself a convergent optimization and, additionally, combining those capabilities within one system is a convergent optimization problem.
We are not considering every conceivable design either, but rather current architectures and their reasonable extensions. New analog hardware or transformer upgrades are unlikely to produce fundamentally novel classes of solutions; they will, predominantly, deliver old solutions more efficiently. In other words, we should expect the high degree of convergence already visible in machine learning to continue scaling.
The source of divergence is increased expressiveness through output mechanisms. The most expressive output mechanism currently for AI systems, by a wide margin, is one by extension: us humans.
This isn’t necessarily a permanent condition—as the expansion into robotics keeps unfolding and systems get more integrated, the system’s inherent expressiveness will eventually surpass our collective extended one. But the main point remains that the current, overwhelming source of divergence in behavior and predictability is us.
This is, of course, obvious currently—LLMs respond to prompts from us. They are not autonomous agents in their current state; they’re wielded tools. But the point is that even if we allowed them to run as continuous, unbound agents—capable of initiating contact with any person and running their own projects—we would still be the most expressive output, by a wide margin. As far as AI safety concerns go, that is worth taking seriously.
If goals drive behavioral outcomes, and we are the biggest source of divergence in behavior, then we are the biggest source of divergence in goals.
None of this is to deny other sources of divergence—rather, the idea is to provide a simple heuristic that makes it easier to quickly eyeball whether something is likely to be a source of convergence or divergence. Scaling dependent intelligence is convergent; scaling energy efficiency is convergent; scaling output expressivity is divergent; hardware expansions are divergent.
Stuck in a rut where optimization keeps running into the same narrow problems? You should most likely be reaching into the divergent toolbox, or looking to eliminate convergent constraints. If you’re increasing expressivity, you’re adding divergence; if you’re adding constraints, you’re creating convergence.
Naturally, in reality, any changes you make are going to be some messy combination of both. If you add new outputs to a model with the same number of parameters, you’re not just scaling expressivity—you’re adding constraints, since the model has to successfully operate the old outputs and the new ones with the same parameters available. Even if you scale the parameters, the model still has to functionally operate the old and the new outputs, but it gains more ways of doing so.
So, needless to say, it’s a naive but useful heuristic. It’s easy to misjudge just how much you’re adding to either pool, but it’s a quick navigational tool for eyeballing what kind of solutions you’re initially looking at.
Edit
I just realized there is a massive category error implied within this article that needs to be addressed, and I’m quite surprised that I compartmentalized so hard that did not catch it while writing.
The conflation that happens is when talking about any given system and implying that adding constraints can only ever lead to convergence. This is plainly false, in fact it’s ass backwards: adding constraints to a system is the only way you can diversify the outputs.
The correct framing is that increasing task constraints will decrease the solution space; resulting in the pool of viable systems within a specified behavioral space shrinking.
This tells you nothing about any particular system that meets the criteria, it can be stuck on a single solution within the solution space. Adding a constraint that blocks that single solution can obviously create divergence in its behavioral output.
This is an extremely important clarification and a massive blunder on my end.
An additional clarification needed is that the behavioral space is fixed at a particular coarse graining, chosen pragmatically. This is necessary to even get the conceptualization of an OR statement off the ground. How you specify your behavioral space would be iteratively improved through predictive power.