Selecting a suitable large language model (LLM) from hundreds of thousands of candidates requires a deep understanding of many model properties. Conventional practice measures each property with a dedicated benchmark suite, incurring high evaluation costs. Model embeddings compress a model into a fixed‑length vector, enabling simultaneous prediction of multiple properties. Existing approaches such as EmbedLLM build these vectors from correctness patterns and question features, often needing a large probe battery. However, behaviors important for model choice—verbosity, safety, style—manifest more in the answer text than in binary correctness. We therefore introduce BULLPEN (Behavioral Understanding of Large Language models via Profile ENcodings), extending EmbedLLM in three ways: (1) we reorganize prior downstream tasks and add 20 new benchmarks focused on behavior, style, and provenance; (2) we jointly read a model’s answer text, its correctness label, and the question text during encoding; (3) we encode models from a few hundred questions instead of the full 24,118‑question pool. Across 260 open‑weight models, our answer‑text encoder achieves 0.48 ± 0.10 on the new tasks, outperforming the best prior correctness‑only encoder at 0.35 ± 0.10. With only 400 questions (1.3% of the full pool compute), the answer‑text encoder retains at least 91% of its full‑pool skill, while a correctness‑only baseline retains 83% under the same budget. Because these embeddings require only model outputs, they support behavioral auditing and low‑cost profiling, and can extend to API‑only models. Code and data are released at https://github.com/wcquesera/bullpen
Review: BULLPEN’s integration of answer‑text information and behavior‑centric benchmarks delivers high‑quality model profiling at a fraction of the computational cost, offering a practical solution for real‑world model selection.