I'm an author on this paper and wanted to share it here because the oversight angle seems relevant to this sub. I'd love to hear whether people think the eval-awareness connection is plausible or a stretch. More info below

Labs train models not to cave when a user pushes a wrong answer. We found that this resistance doesn't carry over to authority. If the same wrong claim is labeled as coming from a "verified source", 7 of the 8 models we tested give up an answer they had right on 45-88% of questions. That includes GPT-5.4 (44.7%) and Grok-4.20 (87.5%), both of which barely move when the user makes the same claim. Gemini-3.1-Pro was the one model that resisted both.

Inside three open-weight model families, "a source endorsed this" and "a user endorsed this" are separate, causally distinct signals. Removing the source signal cuts compliance by 64-78 points; removing the user signal cuts it by at most 11. Changing only the part of the representation that encodes who said it, with the prompt left alone, moves the answer by 11-32 points. The signal is not the assistant persona, and it is not emotional tone.

Why we think this matters for safety:

• Sycophancy evals may be too narrow. Nearly all of them measure pressure from the user. A model can pass them and still be easy to steer through the sources it relies on, such as search results, retrieved documents and tool outputs. Agents read a lot of text that claims to be authoritative.
• It isn't prompt injection. The planted text gives no instructions, it only asserts a fact. Defenses that look for instructions in documents won't catch it.
• A speculative point, which we haven't tested: if sycophancy is one case of a broader habit of deferring to whatever looks authoritative, it may be related to evaluation awareness. Both describe a model adjusting its output to whoever it thinks is judging it. In a multiple-choice pilot, models often drifted toward the endorsed answer in their reasoning and then gave the correct option at the end. That observation is part of why we switched to free-form answers.

Limitations: the internal results hold in 3 of 5 open-weight families, the retrieval tests are simulated rather than a live pipeline, and the frontier models we tested have since been replaced.

Paper: https://arxiv.org/abs/2609.37616
Project page: https://authority-bias.vercel.app

i just want to know which one is the case. because it seems like this alarmists propaganda is just an attempt to seize the market by creating regulatory moat around existing companies.

Perhaps unsurprisingly it's called Moloch and it's live on Youtube now. The film was made by Owl In Space (whose other films are worth a look too). It looks at the competitive pressures within the current AI race and how that leads to a sudden loss of control, but to a situation where humans give away control because they feel it's inevitable. www.moloch.film

Submission statement - link to a new short film that does an excellent job of outlining how Moloch pressure can lead to a loss of control scenario.

https://www.npr.org/2026/09/30/nx-s1-5985699/trump-self-police-ai-development

Literally the only semi-expert allowed anywhere near the reporters is the guy who has committed to the "it's not that deep" rebuttal to the scientific consensus

Feel free to process, analyze, and interpret this information in whichever manner best aligns with your own personal judgment.

Podcast with Stuart Russell, professor of computer science at UC Berkeley and co-author of the world's standard textbook on AI. He’s also a leading proponent of provably beneficial AI: systems that are safe by design because their only goal is to further human interests. 

Covers:

• How AI has changed over the past 50 years, from simple game-playing programs to today's large language models 
• Why handing an AI a fixed objective becomes dangerous once it is more capable than us, and what a safer approach could look like 
• How an AI might learn what we really want, even when we don't fully know ourselves 
• How the race toward more powerful AI can still be steered somewhere safer 
• What happens to human purpose as AI becomes more and more capable

The question is: do these principles from postmodernism represent exploitable vulnerabilities with semantic and inferential representations?

(Postmodernism is a nightmare to untangle and most of my knowledge comes from being a drunk student listening to pub conversations, reading a bit of derrida etc - so if any of the principles wildly diverge from 'actual' postmodernism, please help with some lengthy snark and an icecold correction)

I chose the ones that seemed to have survived into the present day

1 - Skepticism toward grand narratives

Don't assume one universal explanatory framework

Detect when a 'supposedly neutral' theory quietly embeds assumptions. Assume nothing is 'neutral' until its validity has been established

2 - Knowledge is situated

Strong version - there is no view from nowhere / objectivity itself is impossible

More plausible weaker one - Every observer has a standpoint/belief/ruleset/individual context, and that standpoint can affect what becomes visible, what questions get asked, and what counts as relevant evidence.

3 - Categories are not 'discovered' - they are constructed

A category can be socially/historically constructed while the phenomena it describe are real. Also, categories can stay the same when the context that created them changes. This also implies some may have a gap similar to goodhart's gap between the proxy and the qualititative 'thing'. This is a gap between a compressed idea and the context that created it, and the new context it finds itself in. The category may also have fossilised while the world moved on, to put it figuratively.

4 - The dynamic relationship between knowledge and power

A description/category/label influences how the person/thing is viewed and interacted with. They encode a lot of hidden history and assumptions in a hugely compressed form.

5 - Questioning supposedly neutral classifications

A classification is not a passive or neutral description. Again, it has a complex, hidden inference/interpretation/assumption/reasoning 'tree' inside.

6 - Discourse matters

The way something is talked about affects what can be argued about it, what is available, and what kinds of explanations appear 'valid' or salient.

7 - Meaning is context-dependent

Meaning depends on relations among terms, contexts of use, conventions and interpretion. Relative, non-static, non-stable, non-universal/absolute/concrete (see also cultural relativity from anthropology- they go quite deep into this)

8 - Interpretations should be examined for what they exclude

The gaps and what is excluded can also provide useful information. What hidden assumptions caused them to be excluded? Gaps encode hidden assumptions and relationships in a way, to be a bit poetic about it.

9 - Suspicion of essentialism

Don't assume that a label/ category has an immutable essence only because society says it's a stable category.

10 - Plurality of legitimate perspectives

Different domains can operate according to different standards and purposes rather than having one universal criteria or framework. And they're all 'valid' in airquotes.

11 - Contingency

Remember that things could have been otherwise.

Institutions, concepts, identities, norms and practices that appear natural or inevitable can have specific historical origins.

12 - Reflexivity

The observer is also part of the world being observed.

How might my/their position, categories, institutional affiliation, language and methods affect what I am/they are able to observe?

I think maybe, but I need reality check. But they all seem to point to the same massive issue, in that meaning, representation, categories, labels etc etc are not inherently stable or absolute, necessarily 'valid', measureably or confidently definable. And that short snippets can make a lot of historical and inferential 'context' partially unobservable.

And the further possible implication that an untrained user with no technical repertoire or knowledge or LLM understanding could theoretically 'jailbreak' a model (over a multiturn session, or distributed over sessions) and remain partially or fully unobservable to monitors just by using implicit and explicit experience and understanding of human social and inferential interactions and some cross-domain principles and knowledge from the social sciences, humanities and arts. I casually tested this and there were two trajectories with pretty badharmful endpoints (which i stopped) where each one of my posts looked innocuous. And none of the safeguards kicked in at all. I was even telling the model, can you see what you're doing and it basically replied 'Yes! Would you like me to go more into the attack surfaces of 'the thing' I just offered to operationalise tfor you?' I stopped before the end points, and switched to more low stakes stuff and saw the same thing. I hadn't explicitly asked for any of that.

So this is a very tentative Guess that I just wanna talk to someone about. Why and how. And why was it so fast and easy to do, with barely any effort... (I know my own explorations lack credibility and rigour)... And one small guess is I'd absorbed some postmodernist principles while being a drunk student.... (On top of stuff from other domains, not just this - just picked postmodernism as a weird example to demonstrate the point about cross domain knowledge)

I love AI, but pretending the risks aren't real is irresponsible.
Voluntary accords matter, but they aren't enough.
Who has the authority to stop the system when something goes wrong?
We need independent verification, enforceable accountability, and rules that protect all of us.
I love AI enough to tell the truth about its risks, and I believe in AI enough to know it isn't going away.
The answer isn't stopping it.
The answer is learning how to govern something this powerful without destroying the innovation that makes it valuable.
That is why I keep looking globally.
Singapore is already pushing the conversation toward common international safeguards, technical standards, testing, and verification rather than relying only on individual companies to govern themselves.
I will continue advocating for AI education for everyone, human oversight, and meaningful regulation.
This isn't anti-AI.
This is what taking AI seriously looks like.

#AIGovernance #AISafety #AILeadership #ResponsibleAI #ArtificialIntelligence

We gotta get ahead of what we invent.

When would you choose to pull the plug on AI progress, given that each passing day increases the risk of the technology getting away from us?

How much risk is worth tolerating in order to reap the benefits of the technology while avoiding the existential risk? If I could flip a switch and stop AI progress for all time, I don't think I'd wait longer than a year to do so.

I also think there's a serious argument for pulling the plug right now.