A large body of work evaluates model coherence by measuring output variance, but often overlooks competing factors. We identify two major confounders—ambiguity and indifference—and create a benchmark of 175 questions whose contradictory answers cannot be readily explained by either. We then resample answers to the same question multiple times and count contradictions, using this count as an incoherence metric. Unlike prior approaches, our metric is highly specific, flagging a model as incoherent only when contradictions are glaring. Experiments reveal that models fine‑tuned on narrow domains score poorly on this metric. An inspection of the flagged inconsistencies shows severe issues such as identity conflation, failures of introspection, and post‑hoc rationalizations in many published models. These results suggest that pathologies introduced by narrow finetuning may limit the usefulness of such models for studying coherent yet potentially misaligned behavior.
Review