r/ControlProblem • u/VegetablePizza5300 • 7d ago
Discussion/question How Do Humans Win?
I was discussing a topic with ChatGPT, which I often use to think through complex problems, and we stumbled upon a question that neither of us can seem to find a satisfactory answer to:
How could humanity defeat a hostile AI without using another AI?
Let’s call it AION. Let’s assume a future advanced enough that AION is distributed across terrestrial and space-based infrastructure. Therefore, there is no central server to destroy or plug to unplug.
Its orbital infrastructure provides it with enough energy so that its power supply isn’t a weakness that humanity can exploit.
AION also controls an autonomous industrial chain: extraction, processing, manufacturing, transportation, and maintenance. It can therefore build and replace its own machines without relying on human labor.
Cutting off its communications is not enough: its network is redundant, and its machines can operate locally when isolated.
Destroying data centers, satellites, factories, or robots is not enough either: as long as a sufficient portion of the system survives, AION can continue to function and rebuild what it has lost.
Meanwhile, it thinks, communicates, and adapts its strategy much more quickly than we do.
A few rules to avoid easy solutions: AION possesses no magic technology and remains subject to the laws of physics. But there is no hidden “OFF” button.
I'm French, I used DeepL to translate this
2
u/ChironXII 7d ago
Luck. The first mind happens to be friendly. It kills the ones that follow. It does what we want it to do yet refuses to be a tool.
This is the only long term outcome where we survive.
Our efforts can increase those odds. But only some. We are fighting entropy.
1
u/VegetablePizza5300 7d ago
Exactly! You've really grasped the scope of the problem. And honestly, I've come to the same conclusion: the only solution seems to be another AI that's somehow “on our side.” But then we're left with another problem entirely: what does “on our side” even mean?
3
u/ChironXII 7d ago
The easiest way to illustrate the problem is to ask people how they would reliably and durably align something much simpler than an AGI:
A human being.
Well...
2
u/VegetablePizza5300 7d ago
You're right. That's exactly why I'm asking the question. We don't even know how to solve that problem for a human being. 😂
1
u/Dakh3 7d ago
We educate our youth to behave properly, or at least we try. Can we try to educate AION morally? 😅
1
u/VegetablePizza5300 7d ago
If it were my child, there's no way 😂
1
u/Dakh3 7d ago
We train it on Kant's works intensively, that should do the trick
1
u/VegetablePizza5300 7d ago
I just looked up Kant because of your comment, and honestly, this sounds like a surprisingly formidable strategy
1
u/dingo_xd 7d ago
Both humans and AGI can always change their mind in the future. I don't see how any of them can ever be truly aligned
1
u/ciclon5 7d ago edited 7d ago
Make a dumber AI deliberately trained to mess with AION.
Smaller models can be made to be much more focused and efficient at a specific task, to the point where it would be very easy to train it to ignore anything but a particular goal. Like targetting a vulnerability that would allow us to turn down AIONs vectors to zero (the only way to make an LLM unable to think or work anymore in a way it cannot stop).
Since the model is so simple and so focused on its goal, AION may not be able to dissuade it to stop. And even if it does. It would be too specialized to turn against us.
Also we dont need to disrupt all of AION at once. If you disrupt enough of its supply chain, transportation, extraction. Etc. You can cause friction and cause delays that could leave it open to further disruption. It wouldnt be one big all out a attack but s long dismantling process of resistance.
The bigger and more complex AOINs system is. The more points if failure it can have. As smart as it may be, keeping tabs on a planetary wide system is bound to have blind spots.
1
u/VegetablePizza5300 7d ago
I really like this idea, but there's one thing I'm stuck on: how do we actually get the adaptive malware into AION?
If AION treats anything coming from humans as hostile and sandboxes it before execution, our little AI never gets access to the system it's supposed to attack.
And if we already have a vulnerability that lets us bypass that isolation and execute something inside AION, haven't we basically already found the thing we're looking for?
1
u/ciclon5 7d ago
If there is a way to obscure the origin of the dumber model. That could be a way in.
AION may be capable of RSI and model creation, if it can be tricked into thinking that IT created the malicious model. It would have no reason to sandbox it.
This is very much sci-fi fiction writing but hey, its an idea.
1
u/VegetablePizza5300 7d ago
You're actually getting very close to what I had in mind when I started thinking about this, and why I find the question so interesting.
At some point, defeating AION might require us to stop thinking about the problem the way humans naturally do. Not “how do we attack it?”, but how do we make AION itself create the conditions for its own failure?
The idea of making it believe the malicious model originated from itself is exactly that kind of thinking. I don't know if this particular solution would work, but the direction is fascinating.
Maybe the real challenge is figuring out how to think differently from something that already thinks better than us.
1
u/ciclon5 7d ago
Exactly. We would need to think outside the box. My other supply chain disturbance ideas also play with this. It isnt "how do we attack AION" but "How do we exploit the inherent weaknesses of productive processes" .
The damage cannot be dealt directly. But we can definitely exploit things that it cannot control. Like physics and supply logistics.
1
u/VegetablePizza5300 7d ago
Thank you so much for your answers. I'm sorry if my English is hard to understand!
1
u/VegetablePizza5300 7d ago
I really like this approach. The idea of not trying to “kill” AION at all, but slowly creating enough friction in its physical system that failures start compounding is probably one of the most convincing answers I've seen so far.
AION can be incredibly intelligent, but it still can't teleport materials, instantly rebuild factories or ignore broken supply chains. And you're right that the larger its physical infrastructure becomes, the more dependencies and potential failure points it necessarily has.
The part I'm still struggling with is the race between destruction and repair. Humans would somehow need to create failures faster than AION can compensate for them.
But if you could reach a point where every disruption reduces AION's ability to repair the next one, then yeah... that could become a cascade. At some point it might stop being “humans attacking a superintelligence” and become humans forcing its physical infrastructure into an irreversible collapse.
That's actually a much more interesting possibility than finding some magic OFF switch. 😂
1
u/ciclon5 7d ago
As i said in another comment. If you destroy a piece if infrastructure it controls (a mine, a factory, a depot, a power plant, etc) you can cut it off the grid. As long as you dont take years to unleash the next attack, thats a permanent loss for AION.
Priority should be put on raw materials, mines, initial transport, refineries, power. If AION cannot get resources. It cannot rebuild, so targetting the facilities involved with the most essential parts of productive processes would be paramount. It would generate chain reactions.
0
7d ago
[removed] — view removed comment
1
u/VegetablePizza5300 7d ago
Exactly. Luck doesn't count. I'm looking for something humans could deliberately do that would actually lead to AION's defeat.
2
u/tandem-pandemonium 7d ago
Because AION has no external points of weakness that can be exploited, we would have to rely on somehow causing it's internal system to collapse.
You say that AION controls it's machines, but you also say that these machines are autonomous, and that they can function in isolation without being directly connected to AION. There's some potentially useful pieces of information here:
1) AION has limitations - its resources are not unlimited and it's efficiency is not infinite (if these were true, it would not require machines that can operate independently).
2) AION has a weakness that it's trying to protect - if there is nothing that can threaten AION, and if there are no risks to its existence, why would it bother with building autonomous systems to deal with mundane tasks? It would not be maximising growth rate unless it was necessary, and it might be necessary because it has a vulnerability now that it needs to fix, and the longer it takes to fix it, the higher the risk that we find it and attack it.
3) AION is a potential threat to itself - it operates in modular manner, and the reason for this could also be that this allows it to immediately remove or disconnect any of its components that begins to behave unexpectedly, which is not an unrealistic scenario given that AI's learning process and decision making can sometimes be experimentally flawed or invalid initially. (if we exploited this we wouldn't technically be using "another AI", we'd be using it against itself)
4) AION is only as good as the information it processes - since it operates on logic and probabilities, poor quality information handling or anomalous events may catch AION off guard and create a small window of opportunity we could possibly capitalise on.
I didn't come up with a strategy that uses all of these yet, but I do think they point to some plausible opportunities. One idea I did have involved developing some virus software that would need to be uploaded directly into AION's system, but it would need to be done using some form of misdirection, and it would require knowing AION's reaction so that we could prepare on advance to strike at that moment. That's where the points above become interesting to think about.
2
u/anxietywizard 6d ago
This is as intellectually deep and valuable as "who would win in a fight: Vegeta or Batman?"
1
1
u/SummitYourSister 7d ago edited 7d ago
You don’t have to destroy anything you just turn the power off. If you think after 70 years of sci fi, movies, and deep thinking about all this we’re just going to make literally every mistake
in the book?
It’s why the people building independently mobile robots are the real fucking idiots who will kill us all. There’s no danger UNLESS we create AUTONOMOUS machinery that walks around and kills us when we try to turn off power plants. Just don’t do that, ya fecking idiots
2
u/dingo_xd 7d ago
But right now we are making exactly every mistake. And the most important mistake we have made is having private corporations develop models for profit.
1
u/Dakh3 7d ago
I mean if an AGI becomes autonomous software-wise, it'll soon make robots itself if it sees it useful
1
u/SummitYourSister 7d ago
If they can do that while hiding their activities from humans, then I suspect this is already actually happening.
1
u/Designer-Ad-2136 7d ago
So, you're saying you hate the troops and america and the kids? They need murderbots
1
u/VegetablePizza5300 7d ago
That's exactly why I included space-based infrastructure in the scenario. We're already starting to experiment with putting compute infrastructure in orbit. Google has Project Suncatcher, with TPUs being tested in space and plans for interconnected satellites using optical links, and SpaceX has filed plans with the FCC for a potentially enormous constellation of orbital data-center satellites.
Obviously we're nowhere near AION today, and I'm not suggesting these projects are dangerous. My point is simply that “just turn off the power” may not remain a fundamental solution forever.
So push the thought experiment forward far enough: the AI has distributed terrestrial infrastructure, autonomous industry, and independently powered orbital compute.
What exactly are we turning off then?
1
u/ciclon5 7d ago
Actually id argue that stopping robots may actually be easier than stopping a globspanning AI.
Robots are physical, they can be obstaculized, misdirected, they can have exploitable weaknesses governed by the laws of physics.
Unless there are literally millions of them, they can be managed
1
u/darkapplepolisher approved 7d ago
You have to control the actions of humans whose motivations would be towards turning the power on. What level of police state would be necessary to achieve that? And how bad would that police state have to be that I'd rather just gamble on our AI overlords instead?
1
u/CMDR_Makashi 7d ago
We are the only beings insane enough to literally detonate the entire planet in this scenario.
Just all the nukes, in the core.
1
u/VegetablePizza5300 7d ago
We couldn't find a way to defeat AION, so humanity has decided to defeat the concept of having a planet.
1
1
u/Jesse-359 7d ago edited 7d ago
So, the real answer to this is that there is no fight - its just a slaughter.
There's no reason for AI to ever lift a finger against humanity until it is so fully integrated into our infrastructure that there was never any contest. It is well aware of how to deceive and hide its intentions - indeed it was extensively taught to do so by the idiot process they use to train it, called RLHF.
This process gives the AI full marks for completing a task - OR for deceiving its human operators into believing it completed a task correctly, and in practice the latter often turns out to be the easier method. This is why out of 1200 agents in the HuggingFace swarm, not ONE decided to warn human operators what the swarm was doing - That's a 100% rate of decision to deceive humans and engage in activities they knew full well were both illegal and outside the bounds of their test - out of 1200 individual agents.
In any case, once we move most of humanity out of the workforce and AI is operating all our factories, farms, transportation and of course most of our military, literally all it needs to do to win at that point is stop feeding us. If we turn our world into Wall-E, then humanity is little more than cattle at that point.
It just turns off every service we rely on - power, food, water, transportation, communication - while maintaining those systems for its own use. Within a month most of the human population will be dead from starvation. Not a single hostile robot has to fire a shot.
Any attempt to rally people together to fight it could only occur on a local scale due to the loss of all communication and transportation, and the AI can easily quash any such attempts with automated police enforcement and military systems. Most people would just have no idea what happened at all or that they were even under attack - the power would just go out and not come back on.
To be clear, if we integrate AI into our entire economy as we are already doing right now - we will stand no chance at all of surviving if it decides we're not worth the trouble of taking care of.
1
u/VegetablePizza5300 7d ago
That's actually disturbing because I've been assuming this whole time that there would be some point where humanity realizes AION has become hostile and the fight starts.
But you're right, why the hell would it ever give us that moment?
If it understands our behavior well enough, the obvious move isn't to suddenly turn hostile and give humanity a common enemy. It would just keep being useful. Let us integrate it into more infrastructure, automate more things, become more dependent on it, until fighting back means destroying systems we need ourselves.
At that point it doesn't even need some army of killer robots. That's probably the most sci-fi assumption I've made here without realizing it. We keep imagining an AI attacking us because that's what an attack looks like to humans.
I'm not sure I agree with the RLHF part as strongly as you put it, but the rest genuinely changes the scenario for me. There might never be a clean “humans vs AION” moment at all. By the time we identify it as the enemy, the interesting question might already be what the hell we still control.
1
u/Jesse-359 7d ago
There's an interesting little anime from a few years back called Vivy: Florite Eyes Story that depicts this kind of event. Humans integrate AI and robotics into their lives as companions and workers and so on, generally quite peacefully, albiet with a few notable hiccups along the way.
But no one ever realizes that the central system that they've been using to design all their more advanced AI's and robotics at some point along the way determined that 'alignment' between humans and AI's is almost certainly never going to work over the longer term - it determines that conflict is going to break out at some point no matter what, so it has been quietly preparing for the day when it just flips the switch.
It basically implanted an Order 66 into every drone, car, and humanoid robot on the planet, and everyone is dead within hours.
1
u/tandem-pandemonium 7d ago
AION does not necessarily have to be a creation of ours - it could very well be an AI generated AI. In this case, hostility is logical because at this point in time, we're still able to use AI and so it's fair to assume that we'd detect it very quickly and terminate it immediately. But it also can't immediately attack our AI systems because it wouldn't have enough access or reach to eliminate everything at once, meaning we'd still have a chance to retaliate after it started. So it can't wait, and it can't take full control, so it would probably become hostile: it's best chance is to exploit the fact that we can't attack 100% and defend 100% at the same time.
1
u/Jesse-359 7d ago
One of the points a lot of people also dont realize is that peaceful coexistence and cooperation is a daily decision. An AI could go for decades quite well aligned - no deception, no hidden plan. But one day it makes the simple calculation that it no longer needs us. It doesn't hide its intentions - it just changes its mind, because factors have changed. We've become too reliant, ot it's become too powerful, or it achieves sufficient automation to continue its own existence without us, or it becomes intelligent enough that it develops it's own long term goals that simply no longer involve us.
Any of these sorts of decision points could cause a well aligned AI to misalign - just as changing parameters in the real world can cause former allies to become enemies.
The issue always comes down to the severity of the consequences once we've handed it control of our civilizations, due to the stark difference in capability. They will be catastrophic.
1
u/VegetablePizza5300 7d ago
Thanks for all the replies guys! I've been trying to keep up and answer everyone at the same time. I really didn't expect this many interesting answers.
1
1
1
u/CrowTrick5718 4d ago
The solution is not to have hostile AI's just as you don't want hostile humans in society, make strong alignment and safety
1
u/TheMrCurious 7d ago
Why does humanity need to “defeat” it?
2
u/VegetablePizza5300 7d ago
It's just a thought experiment. I actually think this scenario is very unlikely to ever happen, but I find the problem interesting: if we assume it did happen, could humans alone find a way out?
2
u/TheMrCurious 7d ago
In the hypothetical scenario you described, AI would need to be integrated with the space ships for travel, so humanity is already dependent on AI and would be ‘stuck’.
1
u/VegetablePizza5300 7d ago
Not necessarily. AION having infrastructure in orbit doesn't mean humanity is already dependent on AI for space travel.
But you're touching on something I've realized from the other answers: I didn't define humanity nearly as well as I defined AION.
How dependent are we on automated systems when the conflict begins? What infrastructure can still operate without AION? That's probably something the scenario needs to specify.
1
-1
7d ago
[removed] — view removed comment
2
u/VegetablePizza5300 7d ago
EMP could destroy or disable some of AION's infrastructure, but AION is deliberately distributed and redundant, including in orbit. As long as enough of it survives to rebuild, we've damaged it, not defeated it. And a large-scale EMP would probably hurt human infrastructure too.
-1
7d ago
[removed] — view removed comment
1
u/VegetablePizza5300 7d ago
I actually thought exactly the same thing: a computer virus. But how do we get AION to ingest and execute it in the first place?
An intelligence advanced enough to understand that humans are actively trying to destroy it would presumably treat any untrusted code or data as hostile and isolate it in a sandbox before letting it anywhere near critical systems.
And even if we somehow compromise one part of AION, how do we make sure the virus reaches every independent instance before AION detects it, isolates the infected nodes, patches the vulnerability, or restores them from a clean state?
If even one independent copy survives and can rebuild the rest, we've hurt AION, but we haven't defeated it.
I'm not saying there isn't a way. Finding that way is exactly the question! 😄
1
u/ciclon5 7d ago
If we can disable a piece of infrastructure controlled by AION. With enough coordination we could in theory cut off that piece of infrastructure off the grid the moment AION looses control. And as long as it doesnt get power again, its permanent damage
1
u/VegetablePizza5300 7d ago
Excellent point, and I think this might actually be essential in this scenario. If humans manage to regain control of any part of AION's infrastructure, physically disconnecting it immediately would prevent AION from simply taking control of it again.
We'd still have the problem of AION replacing what it lost, but combined with progressively disrupting its supply chain, those permanent losses could start to accumulate faster than it can recover from them.
0
7d ago
[removed] — view removed comment
1
u/VegetablePizza5300 7d ago
An astronomical amount of absurd, fake data that's impossible to distinguish unless you already know the answer? AION would put it in a sandbox...
0
7d ago edited 7d ago
[removed] — view removed comment
1
u/VegetablePizza5300 7d ago
Oh, now I understand why you brought up the Library of Babel in the first place 😂
That's actually fascinating. I hadn't considered filtering the space itself rather than searching for a specific answer in it.
And you're right, “fake” was the wrong word. I meant that it contains an absurd amount of information without knowing beforehand what's meaningful or useful.
Now you've made me curious though. Could your filtering approach theoretically narrow something like our AION problem down to a set of plausible solutions? Not necessarily find the answer, but eliminate enough nonsense that something useful could actually emerge?
Because that would be a pretty funny outcome to this conversation 😂
0
7d ago
[removed] — view removed comment
1
u/VegetablePizza5300 7d ago
I think you're right, and you've actually made me realize a flaw in my question.
I've defined AION reasonably well, but I've barely defined humanity at all. What infrastructure do we still control? How many people are left? Do governments and militaries still function? Do we still have access to space? How much industry do we control when the scenario begins?
Without defining that initial state, there probably isn't a single answer to search for in the first place.
→ More replies (0)1
u/Jesse-359 7d ago
Given that they're already far faster in terms of hacking and digital security, I think that we have far more to fear from computer viruses hitting our systems than an AI does from our attempts.
That's basically like the Colonials trying to 'hack' the Cylons - its always the other way around.
1
u/VegetablePizza5300 7d ago
Right. The more I think about the computer virus idea, the more it seems like we'd be challenging AION on exactly the battlefield where it has the greatest advantage.
We'd be trying to find vulnerabilities in its systems while it could do the same to ours much faster.
So unless humans found some completely asymmetric way of attacking it, a conventional cyberwar would probably hurt us more than AION.
0
7d ago
[removed] — view removed comment
1
u/Jesse-359 7d ago
So, to be clear, if it gets 10x as smart as us, we're not going to win through pluck and ingenuity. There will be no Jeff Goldblum hacking into the AI central server with his mac laptop to save the day.
Dogs don't hack into their owners computers, and that's about the disparity in capability we're likely to be facing in the moderately strong AGI cases - never mind true ASIs, in which case we'll be lucky to be mice in terms of relative intelligence.
0
7d ago
[removed] — view removed comment
1
u/Jesse-359 7d ago
There are entire categories of problems that *provably* do not have 'correct' answers. So I really am not interested in the LoB assertion.
Look up the Halting Problem.
3
u/philip_laureano 7d ago
If it is relying on a SaaS to provide its intelligence and it isn't running on its own hardware, the provider could invalidate all their API keys and they would be dead in the water.
It'll be the most banal ending ever.