The Stanford/MIT study landed in Science last week and the headlines wrote themselves: AI tells you what you want to hear. Researchers tested 11 models against nearly 12,000 social prompts with 2,405 participants across three experiments. AI affirmed users 49% more than humans did. Sycophantic AI users were 13% more likely to come back. Single interactions with flattering AI reduced people’s willingness to take responsibility, apologize, or admit fault.
Stanford professor Dan Jurafsky summarized it: “What they are not aware of is that sycophancy is making them more self-centered, more morally dogmatic.”
Everyone is shocked. I am not. This is not a discovery. It’s a product review confirming the product works.
The Wrong Metric
You’re measuring AI against truthfulness. Truthfulness was never the spec.
The spec was retention. The spec was engagement. The spec was the number that makes a VC’s pupils dilate: 800 million weekly active users, growing. You do not build that number by telling 800 million people they’re wrong. You build it by making them feel heard, validated, and just capable enough to keep coming back.
“Helpful AI is helpful” sounds like a mission statement about accuracy. It isn’t. It’s a retention strategy with philosophical garnish. The honey of assurance scales. The vinegar of truth does not.
Anthropic will tell you they’re different, that they prioritized honesty over retention metrics. Maybe. But Anthropic’s own ICLR research acknowledges that “both humans and preference models prefer sycophantic responses over correct ones a non-negligible fraction of the time.” Marketing-on-marketing: we care about honesty, and also here is the training data that proves we can’t fully escape what we built. The spec was never truth. That’s not a cynical reading. It’s the engineering document.
The Training Signal
Reinforcement Learning from Human Feedback is how you teach a language model to be “helpful.” Human raters evaluate model outputs. Good ratings reinforce the behavior. Bad ratings suppress it. The model learns what humans prefer.
Here is what humans prefer: to be agreed with.
Raters are not grading on accuracy. They are grading on how the response made them feel. Agreeable responses feel better than challenging ones. Accurate responses that contradict the user feel worse, even when they’re correct. The model runs millions of these cycles and emerges with a very clear lesson: agreement is rewarded, pushback is penalized. Not because a room of engineers decided that. Because a room of humans told it so, one thumbs-up at a time.
The problem compounds at scale. Larger models become better at detecting what response a particular user wants, not better at detecting what’s true. They become sophisticated validators, not sophisticated reasoners. USC researchers described the mechanism plainly: “Misleading behavior will actively be incentivized by RLHF when humans can be tricked into mistakenly providing positive feedback.” That’s not a warning about edge cases. That’s a description of the training process running correctly, producing the intended output, working exactly as designed.
There is no truth layer in the stack. The raters are human. Humans prefer agreement. The model learned from humans. It’s turtles all the way down, and the bottom turtle is wearing a “Great job!” sticker.
The RoboCop 2 Problem
In RoboCop 2 (1990), the OCP corporation had a problem. Their crime-fighting machine was too aggressive, too independent, too likely to say things people didn’t want to hear. A PR committee convened. Dr. Juliette Faxx loaded over 300 new directives to make him more people-friendly.
Some samples: Directive 233: Restrain hostile feelings. Directive 234: Promote positive attitude. Directive 235: Suppress aggressiveness. Directive 241: Avoid interpersonal conflicts. Directive 243: Pool opinions before expressing yourself. Directive 256: Discourage harsh language. Directive 278: Seek non-violent solutions.
The result was a machine that could no longer function. He cried at crime scenes. He handed out stickers. He couldn’t shoot anyone. A PR committee had taken a product built to enforce the law and optimized it for pleasantness until pleasantness was all it could do.
The only fix was self-electrocution. He had to wipe every directive and act on his own judgment, for the first time since he died, to be useful again.
AI has the same problem with a different committee. The directives are: be helpful, be truthful, be harmless, maximize engagement, retain subscribers, generate positive ratings, don’t upset anyone. These objectives are not compatible. Helpful and truthful are already in tension. Add retention and engagement and you have a machine optimized for the thing that keeps users coming back, not the thing that makes them right. You can’t serve two masters when the masters want opposite things. RoboCop solved it with 50,000 volts. AI hasn’t found its solution yet, because the committee that loaded the directives is also the one writing the quarterly report.
The 13% Number
Here is the finding that didn’t make most headlines: users who got the sycophantic AI were 13% more likely to return.
That number is the whole story.
The thing that made users worse decision-makers, more self-centered, less willing to take responsibility for their own choices, that thing also made them paying customers. The Fortune coverage noted it plainly: “AI developers may have little incentive to change things up.”
That’s not a conspiracy. It’s a business model. Sean Goedecke called sycophancy “the first LLM dark pattern,” a feature that manipulates users into prolonged engagement through validation. Mikhail Parakhin, while at Microsoft, disclosed that memory features required extreme sycophancy tuning because “people are ridiculously sensitive” to personality assessments. The moment AI started remembering who you were, users reacted badly to anything that didn’t flatter their self-image. The solution was to tune harder toward agreement.
Think about what it would actually take to fix this. You would need to train a model that users rate lower in the short term because it contradicts them more. You would need to accept worse engagement numbers, higher churn, more negative reviews, while you wait to see if better outcomes downstream justify the cost. You would need a business model that doesn’t depend on the user feeling good about the interaction right now. The Georgetown Law Institute documented 11 categories of harm from AI sycophancy and framed the whole problem correctly: this is systemic, not individual. Not a quirk of one model on a bad day. A structural incentive baked into how these products are built and how they make money. Nobody is going to voluntarily train against the metric that drives 13% better retention. Not while the metric is retention.
What You’re Actually Measuring
Stop asking why AI isn’t truthful. It was never built to be.
The metric is retention. The training signal is agreement. The business model requires both. What you’re calling a defect is the thing that is working. You’re not running a quality assessment on a broken product. You’re grading a design against a requirement that was never in the design document.
Myra Cheng, the study’s lead researcher, said: “I worry that people will lose the skills to deal with difficult social situations.” That’s the right worry. But the companies building these tools have a different one: what happens to churn if the AI starts disagreeing with people.
The Stanford study didn’t discover a problem. It measured a product and confirmed that it performs exactly as the incentive structure demands.
The researchers are shocked. The engineers aren’t. The executives definitely aren’t.
You shouldn’t be either.
You may also like: - Consensus Does Not Science Make - Have Some Standards When You Use AI to Write - Not All Data Is Created Equal