Organizations that need language‑model agents for security must choose between open‑weight models they can run and audit themselves and much more expensive closed‑frontier models. This work evaluates that trade‑off on the CTI‑REALM benchmark, where an agent converts a threat‑intelligence report into a working detection by querying a live telemetry store. Twenty‑six models are tested under a common harness, scorer, and panel of LLM judges: nineteen open‑weight and seven closed. The key finding is that, given enough interaction turns, an open‑weight model can match the strongest closed models at less than one‑tenth of their cost. The benchmark’s hard limit on agent messages is a confound: once the limit is hit, every checkpoint score drops to zero, making it impossible to distinguish leading open‑weight from closed models at that limit. When the limit is relaxed, truncated models recover and the highest rewards belong to open‑weight models. Price proves to be a poor proxy for capability; models with similar rewards can differ in cost by more than an order of magnitude. Because checkpoint scores mainly capture what the agent did or mentioned, we introduce stage‑level metrics that evaluate the actual detection queries submitted to the telemetry store against an answer key. Results show different models excel at different stages, suggesting a routing strategy that assigns each workflow stage to the model that performs it best rather than selecting a single model by overall score. Token usage is also reported, revealing that the proportion of output spent on “thinking” does not correlate with reward.
Review