As large language models shift from pure question‑answering to general‑purpose agents, evaluation must go beyond static answer correctness to cover multimodal perception, multi‑step execution, tool usage, and artifact delivery. Existing benchmarks are often tied to specific task types, execution environments, or scoring protocols, which hampers comparability, interpretability, and reliability for deployment decisions. To address this, we introduce DAREBench (Deployment‑Aware and Reliable Evaluation of Models as Agents), a benchmark designed to capture workload variation and provide reliable agent evaluation.
DAREBench is built on a shared OpenClaw execution environment and organizes 233 tasks—selected and adapted from 22 source benchmarks—into a $2\times3$ workload matrix defined by input modality (text vs. multimodal) and execution form (single‑step, multi‑step, tool‑calling). All tasks follow a unified contract‑based interaction protocol, and scores are audited with evidence‑based verification to ensure reproducibility.
We evaluated 23 commercial API models and 12 locally deployed open‑weight models, completing 7,587 model‑task runs. Evaluation metrics include accuracy, token consumption, and reference cost for API models. Key findings:
- No single model dominates across all workload groups;
- Text and multimodal tasks exhibit distinct accuracy‑cost trade‑offs;
- Open‑weight local models are competitive in several groups but still lag behind frontier commercial models overall.
These results suggest that model deployment and selection should consider workload profiles, deployment mode, and accuracy‑cost trade‑offs rather than relying on a single aggregate score.
Review: DAREBench offers a systematic, comparable, and reliable framework for evaluating model agents in realistic scenarios, enabling developers to make more informed decisions balancing cost and performance.