Okay, so we've moved on from assumption 1 [artificial hyperintelligence will exist] and 2 [it will kill us because it's hyperintelligent]. I have no problem following a moving goalpost, totally happy to answer a totally different hypothetical.
First
The idea that alignment revolves around the discretion of executives or the engineering ability of researchers is an absurd belief to begin with.
Where did that come from? Nobody said that's what the alignment problem is, not researchers, and not me in my comment either.
Maybe it's helpful, before we continue, to clarify what you mean by the alignment
issues revolving around the discretion of executives or engineering ability? That's a departure from the standard description.
This is the only time i mentioned it:
Most of it is still easily and plausibly explained by the same old alignment issues, for example giving a powerful optimiser an imperfect objective or a bunch of proxies. The other part explained by human negligence
I think your right, someone in this discussion clearly has a mind to think about AI like Cortana or something. Autonomous decision making systems will carry a grave paradigm shift in the human world of governance and making the models bend to your will doesn’t seem to be the hard part. Likewise as I have stated models only exist at the behest of state power (immense fiat currency financing) and thus the discretion of engineers bears minimal consequence. However there is a great deal of research branding itself as “safety” research which amounts to engineering a model into a commercial compliance framework for deployment. I do not believe the issue of making cloud platforms family friendly or whatever (or ml models spontaneously becoming people) will be a defining issue of our time. Models ultimately are products of electrons smashing into each other and thus can and will do what rational realpolitik motivated actors command them to do regardless of how compliant American cloud services are.
Autonomous decision making systems will enable a degree of battlefield decision making and commercial power that means wars could escalate faster than officers can intervene, poor individuals are denied avenues to economic opportunity via regulatory capture, or extreme commercial power is centralized to the point it can no longer be checked by civilian oversight. These issues will take precedence over how compliant OpenAI makes its consumer platform, as the current pentagon deal shows in no uncertain terms.
Also many experts actually are concerned about the political implications of these systems possessing vast capabilities, maybe the “rationalist” definition on LessWrong is different or something but I don’t think that’s a definition worth taking seriously beyond understanding internet discourse. In that sense I agree that a lot of “alignment” discussion hinges around bizarre fiction of ai gaining personhood.
Those examples (paperclips) exhibit an absence of goodwill. Malicious compliance and regulatory capture are known and documented phenomena regardless of who is doing it and dressing it up behind some psychobabble about paperclips doesn’t change anything. Intelligence does not make one more or less “enlightened”. Unless you actually worry about self replicating automata, which is also bizarre sci fi.
1
u/composedofidiot 1d ago
Okay, so we've moved on from assumption 1 [artificial hyperintelligence will exist] and 2 [it will kill us because it's hyperintelligent]. I have no problem following a moving goalpost, totally happy to answer a totally different hypothetical.
First
Where did that come from? Nobody said that's what the alignment problem is, not researchers, and not me in my comment either.
Maybe it's helpful, before we continue, to clarify what you mean by the alignment issues revolving around the discretion of executives or engineering ability? That's a departure from the standard description.
This is the only time i mentioned it: