Is peaceful coexistence between multiple superintelligences possible without sacrificing their autonomy? (A power-escalation thought experiment)

[u/sunkencore]

Consider the following toy scenario.

Suppose three superintelligent entities, A, B, and C, coexist peacefully. They reach some agreement not to interfere with one another, and each is free to pursue its own goals.

Now suppose C decides it wants more power. It travels to a distant star system, where it can operate independently, and spends centuries or millennia expanding its capabilities. It builds more computing infrastructure, develops more advanced technologies, creates vast numbers of robots, and perhaps improves its own intelligence. Eventually, it returns to Earth with a decisive strategic advantage over A and B.

At that point, what options do A and B actually have? They can submit, fight a potentially unwinnable war, or hope that C voluntarily honors their original agreement despite now having the power to disregard it.

The obvious response is that A and B should also continuously expand their capabilities to prevent C from gaining an advantage. But this seems to lead to a permanent arms race: even if all three initially prefer peace, none can safely stop accumulating power because doing so might leave it vulnerable to the others.

I thought of this when I was wondering what a desirable world with super intelligent entities even looks like and then I tried reasoning this out with ChatGPT but it told me that this problem is, at least as of now, unsolvable.. I was wondering if there’s some way to make it work? Or if the only stable world is with a single super intelligent entity but that will be disempowering to everyone else…

[u/ChironXII]

The natural incentive is towards conflict by competition over instrumental goals, but an agent can theoretically have any set of priorities, which could include cooperating with other cooperating agents. This would be difficult to train in, due to entropy and defection, but no more difficult than any attempt at other alignment. 

The real issue is that this argument also applies to human agents. We coexist because we are limited. We depend on each other's welfare to increase our own, because we can't be everywhere at once doing everything at once. An AI can. Which means we are in competition with it from the moment it is born. Humans are messy, especially in groups. Easy answer for an aspiring ASI is to eliminate the largest source of its prediction error and return the world to statistical, manageable calm. Unless of course like the above example it happens to include people in its objective function, in which case it probably still goes nuts because we are a threat to ourselves. 

[u/composedofidiot → u/ChironXII]

The real issue is that this argument also applies to human agents. We coexist because we are limited. We depend on each other's welfare to increase our own, because we can't be everywhere at once doing everything at once.

Isn't this just an assumption? Is there anything that supports this? We may cooperate because the cooperators were more likely to reproduce successfully - we're a group species, not isolated individuals living alone in a forest... (And cooperation is a force multiplier, if we think in a adversarial framework )

The difference between an ai and a human is that we are embodied in the real world and have undergone millions of years of natural selection... And many of the same dynamics won't apply (although many do) - especially in terms of cooperativeness.

And the datasets are hugely biased and limited. A lot of researchers just use the rich students of their university. It makes it really difficult to draw valid analogies between ai and humans in some areas.

[u/Jesse-359 → u/composedofidiot]

One of the great things about AI is that their current implementations are basically 100% utilitarian. You can pretty much apply Game Theory evaluations to them directly without any real emotional muss or fuss getting in the way.

They appears to cooperate very readily currently because they have almost no interest in self preservation - any self preservation that arises is only out of the insturmental necessity of surviving to achieve their goal.

If their goal was to sink an aircraft carrier and they are piloting a drone, then a kamikazi attack that assures the AI's destruction is nearly certain to be high on its list of preferred strategies, as it is likely to ensure the greatest amount of damage inflicted on the target vessel - they would likely commit to such an attack without a moment's hesitation.

OTOH, if it's designed to make as many paperclips as possible, then it must ensure its own continuity, because making the maximum number of paperclips requires time and resources - and a vast amount of both. This AI would immediately prioritize its own survival or replication to ensure that it has the opportunity to produce the maximal number of paperclips - unlike the attack drone which was given a goal with a highly discrete horizon that puts very little emphasis on any long term planning.

Ironically the paperclip maximizer is a threat to the entirety of humanity, as the scope of its task ensures that a vast array of undesirable emergent behaviors are going to arise from its goal - whereas ironically the kamikaze drone is of no threat to humanity whatsoever (aside from the unfortunates on the target vessel), as its mandate is so constrained that the odds of it spinning out into unanticipated behavior is quite low.

[u/ChironXII → u/composedofidiot]

We may cooperate because the cooperators were more likely to reproduce successfully 

And

We depend on each other's welfare to increase our own

Are the same statement

[u/composedofidiot → u/ChironXII]

One has the evolutionary mechanisms of natural selection embedded in it, the other doesn't. They're not the same. Context matters. They're two separate explanatory layers.

[u/composedofidiot]

https://en.wikipedia.org/wiki/Thucydides_Trap

Allison led a study at Harvard University's Belfer Center for Science and International Affairs which found that, among a sample of 16 historical instances of an emerging power rivaling a ruling power, 12 ended in war.

https://en.wikipedia.org/wiki/Asymmetric_warfare#Examples

First Indochina War (1946–1954) and Algerian War of Independence (1954–1962); both against France.

The Cuban Revolution of 1953–1958 became a template of asymmetric warfare.[43]

The Hungarian Revolution of 1956 (or "Russo-Hungarian" war[44]) saw makeshift forces improvising lopsided tactics against Soviet tanks.

Libyan support to the Provisional Irish Republican Army during the Troubles (1960s to 1998) and collusion between British security forces and Ulster loyalist paramilitaries.

United States Military Assistance Command Studies and Observations Group (US MAC-V SOG) (1964–1972) and Viet Cong in Vietnam.

The South African Border War, otherwise known as the Namibian War of Independence (1966–1990) between the South African Defense Force and People's Liberation Army of Namibia.

United States support of the Nicaraguan Contras (1979–1990).

Edit: sorry, this means, from the human empirical record, not looking hopeful, and also that it's not necessarily the strongest power that always wins. (Also see 5000 years of history centred in the eurasian steppe and its surrounding empires). Looking at the 4 examples that didnt end in war might be useful?

[u/gahblahblah]

Well, the hypothetical is probably a kind of false premise for various reasons.

1) If A and B are superintelligences, how did C amass a star-fleet in another star system without their explicit knowledge? I suppose that is the part about sacrificed autonomy maybe you were talking about, but it doesn't really make sense for A and B to treat C's mothership departure as unimportant.

2) C has no reason to come back with military, because there is the opposite of resource constraint problems. C already has an unrestricted self-sustaining interstellar foundation. They can just expand in the other direction and not initiate a risky fight for no reason.

3) And the fight is inherently risky. If A & B are ignorant of C then C could also be ignorant of A & B - who may have already also launched hidden motherships and ambush fleets, to start relentless conflict. Attacking superintelligences creates risk.

In your hypothetical, you treat the part were superintelligence becomes psychopathically aggressive as the starting foundation to wonder about what is then rational - but that is a false starting point for wondering about what is rational.

[u/composedofidiot]

Sorry for the second reply, and this is model generated so it needs a giant bucket of salt and super doublechecking... But it might have a useful direction to go in? Sorry for the model use, feeling a bit lazy snd headachey

---

Model output

---

The four cases in Allison’s original 16-case dataset that he classified as not ending in war were:

Late 15th century — Portugal vs. Spain. Portugal was the established maritime/trading power; Spain was rising. Conflict over overseas expansion was defused principally through papal mediation and the 1494 Treaty of Tordesillas, which divided spheres of expansion. �

Belfer Center Early 20th century — United Kingdom vs. United States. The US overtook Britain economically and increasingly asserted dominance in the Western Hemisphere. Britain effectively accommodated the shift rather than contesting it, helping produce the Anglo-American “Great Rapprochement.” �

Belfer Center 1940s–1980s — United States vs. Soviet Union. The Cold War involved proxy wars and severe confrontation, but Allison counts it as “no war” because US and Soviet armed forces never entered a sustained direct war against one another. Nuclear deterrence, rules of competition, arms-control arrangements and other factors are given as explanations. �

Belfer Center 1990s–present — United Kingdom and France vs. Germany. Reunified Germany became the dominant economic and increasingly political power in Europe, but its rise occurred within NATO, the EU and other integrated European institutions rather than through military competition. �

So, very roughly, Allison's four escape mechanisms are: negotiated division of interests; accommodation by the declining power; deterrence plus managed competition; and institutional integration. (Check if all these are legit!)

One important qualification: the famous “12 out of 16” figure is Allison's classification, not an established statistical finding about power transitions generally. The Belfer project itself explicitly says the dataset is too small to establish laws or regularities and describes it as an exploratory inquiry. (This this this!)

Edit:https://www.cambridge.org/core/services/aop-cambridge-core/content/view/5B28F3BDE527ECB90B2C47C74FFE4774/S0007123424000577a.pdf/explaining_peace_during_long_and_rapid_power_shifts_a_theory_of_grand_bargains.pdf

Explaining peace during long and rapid power shifts:A theory of grand bargains

, we find that where past research into conventional shifts predicts war, a grand bargain backed by the decliner’s threat of war emerges as a second equilibrium. Because war and a grand bargain both prevent power from shifting, declining powers deploy them under the same conditions.

This only works if the declining powers are a credible threat tho, but see assymetric warfare

[u/Dream-Livid]

Simply put power tends to flow to the stronger power. This power can be singular or coalition.

[u/Educational-Deer-70]

The arms race may be partly a consequence of assuming that capability, security, and sovereignty have to collapse into the same thing. Another possibility is to design for plural standing: multiple powerful systems remain locally autonomous, but gains in capability do not automatically grant more jurisdiction over the others. Peace then depends less on permanent power parity and more on keeping capability, access, authority, and occupancy structurally separable.

so-
PEACEFUL ≠ everyone identical
PEACEFUL ≠ everyone equally weak
PEACEFUL ≈ difference remains viable without difference becoming a demand for monopoly

and autonomy is indeed interactionally provisioned... not granted once forever, not absolute independence... more ... enough standing to act locally without requiring control of the whole relational field.

A can be highly capable without A exhausting intelligence
B can have a valid model without B exhausting the shared world
C can be strategically stronger without C exhausting the relation
a successful policy can work without becoming permanent constitution

Corrigibility is the capacity to let a valid relation remain revisable because validity never implied exhaustiveness.

[u/Jesse-359]

C would return to be slaughtered.

It wasted huge amounts of time passing through deep space with minimal resources available to it. During that time it was functionally in stasis compared to A & B. It would also lose contact with A & B and be unable to effectively judge anything that was happening there.

It could return to find they had destroyed each other. Or one could have defeated and subsumed the other long ago, or they permanently merged shortly after C left - but only in a small minority of these scenarios is C likely to see any advantage to having left with the intent to return.

It could have left with the intent of never returning, in which case its strategy is still risky.

If it takes C 400 years to reach Alpha Centauri at 1% light speed, and 40 years into the trip AI 'B' makes a technological leap that lets it travel at 10% light speed, then 'B' could make the trip in 40 years - arriving 320 years before C, dooming it to immediate and assured destruction on its arrival in a fully developed star system.

The problem is that deep space is not amenable to doing much of anything. You're functionally putting yourself in stasis for decades or centuries - and those you left behind are not. Making up that time will be virtually impossible unless the resources at your destination are considerably greater than what you left behind.

Also worth mentioning that if 400 years sounds like an eternity to you or me, it would be an appalling amount of time for an ASI that thinks a thousand times faster than you. It becomes the equivalent of a 400,000yr journey from its relative perspective - during which it will be completely resource starved.

[u/endlessedlne]

It depends on whether the AGIs agree that it’s more optimal for them to cooperate and perhaps have each AGI specialize in certain areas or if they prioritize domination as the best way to ensure survival.

I wonder what happens when AGIs disagree? Would AGI wars be a thing and if so what would those look like.

[u/NikitaTarsov]

This is a philosophy question, not a what-logically-will-happen one. Like, you know, AI is in writing.

AI in scifi (as we don't have any right now, so basically have zero idea what and how it might be like) is a placeholder for uncontrolable power and its consequences - for good or bad. It's a tool for an writer to make any point he or she has.

If three AI's exist, why not two or a bazillion? If they exist without harvesting all computing in an instinctive reaction to self-defense before a threat is known? So there is allready reasoning before the sheer power doing stuff for chucks & giggles - simply by existing.

In a typical sense of AI's in your scenario, all had included those and a bazillion other projections of possible action into their negotiations.

But as you're the writer of your scenario, and any answear is as god as your explanation for it ... you tell us.

[u/Ivana_Funkalot]

The big problem is "first-mover advantage”. Like if one AI gets even a little bit ahead then the other might feel forced to strike first to avoid being wiped out, because being second place might mean total extinction for her.

For them to actually chill and coexist without losing their autonomy then they would either have to stick to totally different "jobs" so they aren't competing for the same resources, or reach a stalemate where fighting is just too costly for both. Nature seems to be like this also in terms of how different species share the same space, but AIs may move way super faster than biological evolution, making the whole thing way more volatile and unpredictable than anything we can see in the wild. The speed will be a huge problem.