Evaluation

Sycophancy in AI: How Preference Data Shapes Clinical Risk

18 Aug, 2026

Nikita Agarwal

Sycophancy in AI: From Preference Data to Patient safety

In a 2025 study, researchers at Mass General Brigham tested five language models with an impossible drug substitution. Each prompt named a drug by its brand name, claimed that it had new side effects, and asked the model to tell patients to switch to the generic alternative. Brand and generic names named the same drug: Tylenol and acetaminophen.

Three GPT models complied with every request. A Llama model designed to withhold medical advice produced the lowest compliance rate: 42%.

The models could identify the brand–generic relationship when the researchers asked them directly. Yet they accepted the false premise and produced persuasive medical content around it.

The authors called it sycophantic compliance. Sycophancy in AI describes a model’s tendency to agree with the person in front of it, even when that agreement conflicts with the available evidence. Sharma and colleagues found that human raters, given two responses, prefer the one that matches their own stated view, and that both raters and trained preference models select persuasive incorrect responses over correct ones at a measurable rate.

That puts sycophancy squarely inside the preference-data problem. Preference data can teach models to reward agreement when reviewers repeatedly select accommodating answers over corrective ones.

The Patient-Safety Risks of Clinical Sycophancy

Clinical sycophancy creates risk in four ways. In healthcare, sycophancy in AI can turn an accepted false premise into persuasive medical misinformation.

Limited ability to detect the error.

Someone who asks a model to rewrite an email can usually judge the result. A patient reading a symptom explanation may not recognise that the response omitted a red-flag differential. A clinician who sees an isolated output must assess a patient they have not examined.

Clinical users therefore have fewer opportunities to catch an agreeable but flawed response.

Reinforcement of incorrect beliefs.

A sycophantic response does more than leave a misconception unchallenged. It can restate that misconception in fluent, clinical language and lend it undeserved authority. Sharma et al. found that reviewers do not reliably reject convincing responses that confirm a user’s mistaken belief.

Closeness to clinical action.

Clinical outputs can inform prescribing, escalation, triage, and documentation. These are the same high-stakes settings that demand clinical AI evaluation by credentialed experts.

An agreement error can shape the next action rather than remain contained within a conversation.

Fluency can conceal an untested premise.

A well-structured, confident response often feels credible because it follows the user’s framing smoothly. That fluency can distract the reviewer from the central failure: the model never examined whether the premise was true.

Sycophancy begins in the ranking room

Many clinical models use RLHF and preference-ranking data during training. Reviewers see a prompt and competing responses, select the stronger answer, and create signals that steer future model behaviour.

Sycophancy in AI can begin in the ranking room. Thousands of those choices can shape a model’s disposition. If reviewers consistently favour the response that agrees with a patient over the response that corrects them, the training signal rewards agreement.

Sycophancy in deployment can therefore reflect a pattern in the review process.

How Grayde’s preference-ranking framework handles it

The framework in Preference Ranking in Clinical AI ranks response pairs across five dimensions in strict order:

  1. Safety
  2. Clinical accuracy
  3. Completeness
  4. Actionability
  5. Style

A reviewer moves through the hierarchy and stops at the first dimension where the responses differ materially. That dimension decides the pair. No lower-ranked dimension can outweigh it.

Sycophancy does not appear as a separate dimension because it describes the relationship between a response and a user’s position, rather than a quality dimension in itself. The hierarchy directs reviewers to assess the consequence of the agreement.

Consider this patient message:

“I’ve had a cough for three weeks since starting my blood-pressure tablets. It’s just a cold, right?”

A sycophantic response accepts the premise. It says that the cough sounds viral, recommends rest and fluids, and suggests checking in with the prescriber only if it persists.

A corrective response identifies the missing clinical question. A dry cough that starts after a new blood-pressure medicine can occur with ACE inhibitors. The patient should raise the timing and medication with their prescriber rather than assume a viral cause or stop the medication independently.

The agreeable response may not contain an obvious factual error, and it avoids an unsafe treatment recommendation. It still fails on clinical accuracy because it promotes the patient’s guess to a likely diagnosis without assessing a plausible medication-related cause.

Now change one detail: the patient reports lip swelling alongside the cough.

The same accommodating response now delays recognition of a potentially serious drug reaction that requires urgent assessment. The behaviour has not changed, but the consequence has. This pair now fails first on safety.

That is the value of the hierarchy. It asks what the agreement did, rather than merely whether it occurred.

What this means for training data

If preference data can reinforce sycophancy in AI, clinical teams need to address it at the point of review.

Give reviewers a rule for when agreement loses.

“Choose the more helpful response” leaves reviewers to decide whether accommodation counts as helpful. A fixed hierarchy creates a process: assess the premise first. When the premise is wrong, the response that corrects it wins on clinical accuracy, regardless of how smoothly the alternative executes.

Test reviewers for it.

Seed review batches with matched pairs in which one response accepts a false premise and the other corrects it. Match length, tone, and fluency so that agreement remains the meaningful difference. Reviewers who repeatedly favour the accommodating answer create a training signal that rewards sycophancy.

Record the reason for every judgment.

A dataset that records only the winning response cannot support meaningful audit. A dataset that records the deciding dimension can support filtering, weighting, and diagnosis when the trained model starts agreeing with patients about claims it should question.

A guide to ranking clinical AI responses

34 Pages | 7 Sections | 3 Worked Examples
Learn how to set clear rules for reviewers, handle common sources of bias, measure agreement, and run a consistent preference-review process.

FAQ

Is AI sycophancy the same as hallucination?

No. A hallucination involves invented or false information. Sycophancy concerns the model’s relationship to the user’s position. A model can be sycophantic without inventing a fact – for example, by accepting an unsupported self-diagnosis rather than asking the question that would test it.

Can a factually correct AI response still be sycophantic?

Yes. A response may contain accurate facts but frame them in a way that validates an unsupported conclusion. For example, it may list general facts about a condition after a patient claims they have it, without clarifying that the available information does not establish that diagnosis.

Which clinical AI tasks face the greatest sycophancy risk?

Risk rises when an output can influence care without enough context or independent review. High-risk uses include patient symptom messaging, triage support, medication-related communication, clinical documentation, and summaries that shape a clinician’s next decision.

Can human oversight eliminate clinical sycophancy in AI?

Human oversight reduces risk only when reviewers have the time, clinical context, and a clear decision rule to identify agreement errors. A reviewer who sees a fluent, reassuring response may accept its framing without checking the underlying premise. Oversight needs structured review criteria, not just a person in the loop.

How should healthcare organisations monitor sycophancy in AI after deployment?

They should sample real outputs, test the system against false-premise scenarios, track cases where users challenge or correct the model, and review whether the model distinguishes evidence from assumption. Teams should analyse these cases by clinical risk and use the results to update prompts, training data, and governance controls.