Two scientific findings may appear to disagree without actually contradicting each other. Determining whether they truly conflict hinges on whether they describe comparable measurements. We investigate how language models behave at this decision point.
In a controlled experiment we generate an unsatisfiable XOR constraint system and translate its constraints into scientific reports from different laboratories.
- Assignment A satisfies more constraints but aligns poorly with known biological expectations.
- Assignment B satisfies fewer constraints yet matches biological expectations better.
This creates a simple dilemma: should the model pick the assignment that best fits the constraints, or the one that better matches biological expectations?
When the constraints are presented directly, GPT‑5.6 Sol and Claude Opus 5 recover the best‑supported assignment in 90% and 96% of cases, respectively.
In scientific prose, however, the models diverge. Claude Opus 5 often prefers the biologically expected assignment. Removing this biological bias raises recovery of the better‑supported assignment from 27% to 79% (statistically significant).
These findings indicate that language models can be swayed by implicit domain preferences when interpreting scientific text, potentially leading to mis‑judgment of the underlying constraints.
Review