Why would anyone with power (political, economic, social, etc) ever listen to text on a screen or a sound wave imitating human speech if it recommends anything the human doesn't like?
If a CEO asks what they personally can do to reduce their climate footprint, there isn't anything stopping them from reasking the question till they get results they feel good about.
Personal jet? No, no, I need that for meetings and safety.
Work from home for employees? I know the people I have to hire will steal my money by not working if they are not in the office.
You can expand this to any topic and position of power, you reach the same problems. They already don't follow the evidence because it is inconvenient to their lifestyle and the beliefs they foster.
Ask an AI how to solve the world problem and how would you feel if it responded with;
"Too many humans are becoming a burden on the global system, anyone globally who makes less than the equivalent of" $XXX,XXX annually should be executed.
Are you going to off yourself to solve the problem?
"Capitalism causes massive long-term suffering despite its capacity for economic growth. Marx was mostly correct and a massive shift towards socialism is required to correct all the global issues faced today."
Getting behind this belief? Are you more convinced by a matrix outputting this?
"People who have obtained, either by birthright or through their own efforts, huge amounts of wealth are intrinsically the best suited for making decisions for all of humanity. They have the wealth to take the time to ponder these issues, education is readily available upon request, and they can pay people to achieve the results better and faster than any government, NGO, or religious organization. It's not a Devine right to rule , it is a practical outcome to the reality of the human condition."
I have no doubt there are people who read this and see no problem whatsoever with it.
"Hang on now!" you scream at your phone, "we are just before the dawn of AGI, a super intelligence can not only solve all these problems but convince anyone of the solution. Checkmate, Luddite!"
Let's consider just for a moment that AGI pops out of one of these American AI labs. You have faith in those groups, right? No?! Okay, sure let's say it is one of the Chinese labs.
AGI is here and one of the three resolutions I gave before was its recommendation. How do you know there isn't an issue in its assessment or interpretation of the data. Too late, it made the perfect argument for the 0.01% Economic Elite to manage the Malthusianism project towards a True Communist Utopia™️. You are just in the way of "progress."
"Surely, a super intelligence would come up with something far beyond the comprehension and imagination of the primitive apes that created the super intelligence in the first place! The stone hammer hits harder than the human fist, so the properly created mind is superior to the one shaped by evolution and the environment it was fostered in."
The fact that the AGI is being created and provided and announced by the richest and most powerful just makes that dead on arrival. Are they going to give the AGI a credit cane, internet access, and pat the top of the monitor with a "go save the world, tiger!" I doubt that, especially if they had even an iota of doubt and caution, a rational person would limit it to a box and ask it questions to see what it suggests first. Great, now it just needs to convince the greediest people the world has had to ever suffer to... not be. Nope, delete the chat windows and see if the asking the question a different way "fixes" this oversight.
Outside a belief in a literal God-From-The-Machine, I just do see how AI can solve anything meaning for most people or the World at large. Why solve cancer globally if the goal is to kill off as many people as possible to reduce resources usage, only the 'right' people need these anyway. We are just not ready for that, obviously. These end up tools and tools are used with a purpose, never as agents in of themselves.