Open‑ended large language models (LLMs) often exhibit apparent plurality in ensembles, yet underlying homogeneity can produce false diversity: different systems seem to offer independent perspectives while returning the same familiar default. Single‑turn answers obscure whether agreement stems from a tightly constrained answer space, prompt‑vocabulary echo, or broader spaces with stable alternatives beneath the surface. To address this, we introduce CHOIR (Collective Hierarchically‑Ordered Inquiry Responses), a framework that adapts free‑list elicitation from cognitive anthropology for LLM ensembles. CHOIR repeatedly gathers ranked lists, clusters items into prompt‑level concepts, and measures concept salience across models, prompt variants, and persona conditions. We evaluate CHOIR on Infinity‑Chat 100 (an external prompt bank) and on a 27‑question targeted diagnostic set designed to isolate mechanism‑level contrasts. CHOIR reproduces high surface agreement (93 % of prompts above chance) while separating narrow prompts from broad prompts with recoverable depth. Across both banks, base‑model identity remains the strongest recoverable signature, and persona prompts shift surfaced concepts within a given base model. A source‑blind ranking module prioritises rare‑but‑stable candidates for later inspection. CHOIR turns open‑ended homogeneity into a diagnostic measurement problem by asking where models converge, why they converge, and what remains reachable under structured depth probing.
Review