Users increasingly rely on large language models (LLM) to analyse collections of independent inputs such as product reviews or social media posts and return a single aggregated answer. This requires the model to combine information from multiple instances in one prediction, a setting known as multi‑instance processing (MIP).
We study a task where the model must infer a binary label for each instance and identify the label that appears most frequently in the whole set. The correct answer should be invariant to the order of instances.
Fifteen LLMs are evaluated on five datasets under seven instance‑order patterns, keeping each instance set fixed across patterns. The results reveal strong order sensitivity: accuracy can differ by about 90 percentage points between the best and worst patterns. Models tend to be more accurate when majority‑label instances appear at the start, and less accurate when they appear in the middle or at the end.
For a subset of models, analysis of reasoning traces and gradient‑based attribution shows uneven emphasis across positions. Traces disproportionately mention early instances, while attribution scores are generally higher near the beginning and end than in the middle.
These findings demonstrate that instance order can systematically bias aggregated predictions, highlighting the need to evaluate MIP across diverse input arrangements.
Review: The observed order effects expose a fundamental limitation of current LLMs in aggregating information, suggesting future work should develop order‑invariant architectures or adversarial training to improve robustness.