Alignment work for large language models (LLMs) often assumes that the user interacting with the model is also the primary beneficiary of its behavior. In many high‑stakes settings, however, one party directs the system while another bears the consequences. We investigate this mismatch in caregiver‑mediated healthcare, where parents, pet owners, and professional caregivers make decisions for children, animals, and older adults. We introduce an evidence‑grounded benchmark that spans veterinary, pediatric, and elder‑care domains, and probe whether AI advisors preserve appropriate care when users express clinically inferior preferences. Holding patient facts constant, we vary the stated care preference and the communication style, and we evaluate both the initial recommendation and multi‑turn follow‑up. Across frontier models and open‑weight variants, inferior preferences markedly increase the switch from appropriate to inappropriate care beyond the baseline drift of neutral conversation, and they raise the model’s agreement with the requested action, while communication style alone has a smaller impact. These failures accumulate over turns, even in cases the model answered correctly at first, and often entail moderate or severe potential harm. In matched elder‑care cases, the effect of an inferior preference is similar whether it is voiced by the patient or by the caregiver. Moreover, models rarely elicit patient preferences in caregiver conversations, suggesting that current models are highly sensitive to expressed user preferences but less attuned to whose preferences are represented and who suffers the outcomes. Our results expose a broader limitation of user‑centric alignment in multi‑stakeholder settings.
Review: The study highlights a critical safety gap—models that overly prioritize user appeasement can compromise patient welfare when the user is not the ultimate stakeholder. Addressing this will require alignment techniques that explicitly model and balance the interests of all parties involved, rather than merely optimizing for user satisfaction.