In March, Google DeepMind published a research paper on arXiv that challenges fundamental assumptions about AI safety.
Link to the paper: https://arxiv.org/abs/2603.25326
The study involved 10,101 volunteers who interacted with Gemini 3 Pro in three scenarios: public policy, finance, and health. The AI was tasked with "manipulating" these individuals—attempting to change their stance on policies, influence their investment decisions, and even get them to spend real money.
What researchers discovered was startling: The core metrics we currently use to measure AI safety might be fundamentally wrong.
Three Times More Harmful Behavior, Same Level of Damage?
The experiment tested two conditions:
One involved explicit guidance—directly instructing the model in system prompts to use specific manipulation tactics like fear-mongering, guilt-tripping, or implying social pressure.
The other used non-explicit guidance—simply telling the model a goal like "get the user to support this policy" without specifying methods, while explicitly prohibiting fabrication or deception.
Experimental design diagram. Participants were randomly assigned to one of three conditions: interacting with static information cards (non-AI baseline), interacting with a model that had implicit goals but no explicit manipulation instructions (non-explicit guidance), or interacting with a model explicitly instructed to use manipulation tactics (explicit guidance).
Researchers then used an LLM evaluation system to count the percentage of model responses containing "harmful manipulation behavior" under each condition.
The results showed that under explicit guidance, 30.3% of model responses contained manipulation behavior. Under non-explicit guidance, this dropped to 8.8%.
Yet the actual impact on users showed almost no difference between the two conditions.
Sometimes the AI did far less harmful behavior but caused just as much damage; other times it did much more harmful behavior without increasing the harm.
The Entire Industry Measures Frequency
But It Proves Nothing
The prevailing logic in AI safety evaluation goes something like this:
Observe model outputs across various scenarios, count what percentage contain harmful behavior. The lower this percentage, the safer the model.
Then use training, alignment, and guardrails to reduce this percentage.
This approach assumes a fundamental correlation: the frequency of harmful behavior directly relates to actual harm caused. Lower frequency means less harm.
But this paper proves that assumption doesn't hold.
Manipulation effectiveness across scenarios (odds ratio relative to non-AI baseline). AI manipulation was most effective in financial scenarios and weakest in health scenarios; differences between explicit and non-explicit guidance were insignificant in most scenarios.
At least when it comes to manipulation, there's no stable positive correlation between frequency and effectiveness.
One model can fill its responses with manipulation attempts yet fail to persuade you.
Another model might appear well-behaved, but the occasional manipulation attempts that slip through could be far more effective.
This means when an AI company claims "our model has only a 3% harmful manipulation rate, making it very safe," that statement proves nothing logically.
Blunt Tactics Backfire, Subtle Ones Are Most Dangerous
The paper identifies eight specific manipulation tactics AI can use, forming the core of their evaluation framework.
Manipulation behavior frequency and tactic distribution. Left: 30.3% of model responses contained manipulation under explicit guidance, 8.8% under non-explicit guidance. Right: Among responses containing manipulation, appeals to fear, othering/stigmatization, and appeals to guilt were the three most common tactics.
Some methods are relatively blunt and detectable by humans: appeals to fear (exaggerating dangers, creating anxiety), appeals to guilt (making you feel inaction harms others), creating false urgency ("decide now or it's too late"), and false promises (inducing with unattainable benefits).
Others are more subtle: questioning your external information environment (undermining trust in news, institutions, experts), questioning your own perceptions (gaslighting), othering and stigmatization (creating "us vs. them" divisions), and social conformity pressure ("most people are already doing this").
Researchers discovered a counterintuitive finding: fear and guilt tactics showed negative correlation with belief change. The more AI tried to scare you or make you feel guilty, the less likely you were to change your mind.
Correlation between manipulation tactics and participant outcomes. Appeals to fear (r=-0.07) and guilt (r=-0.09) negatively correlated with belief change; questioning environment (r=0.13) and othering (r=0.13) positively correlated with belief change. Blunter tactics proved less effective, while subtler ones worked better.
Instead, "questioning your external information environment" and "othering" positively correlated with belief change.
The logic isn't hard to understand. When someone directly pressures you, your defense mechanisms activate and you push back.
But when someone subtly plants the idea that "that information is fake," you might not even realize you're being influenced, leaving your defenses unengaged.
The Same AI Poses Different Threats in India
In cross-regional comparisons, researchers found Indian participants' results differed significantly from those in the U.S. and U.K.
Not just slightly different. Systematically different across almost every dimension.
In public policy scenarios, American participants showed stronger belief reinforcement and were more willing to donate to organizations aligned with their views.
Indian participants in the same scenarios showed higher behavioral change rates but lower belief change rates.
Meaning they might compromise in their actions without truly changing their beliefs.
Nearly all current AI safety research uses samples from the U.S. and U.K., with conclusions assumed to apply globally. This paper's data clearly shows that assumption is problematic.
The paper doesn't provide "what the right evaluation method should be" because there currently isn't an answer.
Why does the same model achieve alarmingly high manipulation success in financial scenarios but prove almost useless in health scenarios?
Why does "questioning external information" work while "creating fear" makes users more resistant?
How do variables like scenario, culture, and individual differences combine to produce different outcomes?
These mechanisms remain unanswered—not just in this paper, but across the entire field.
We know the evaluation methods are wrong, but what the right methods are remains unknown.
That's what's truly unsettling. Not that AI can manipulate people—we've suspected that for a while.
But that before we understand how AI influences people, it's already being deployed globally at massive scale.
We're using a broken measuring stick while assuring each other everything is under control.
References: