Assessing whether an agent can strictly satisfy diverse and procedurally complex requirements is crucial for its practical reliability. Such an assessment demands tasks that balance broad domain coverage with deep procedural execution.
Existing benchmarks face a trade‑off: expert‑curated suites achieve procedural depth but suffer from limited domain diversity, while crowdsourced collections span many scenarios yet lack rigorous execution checks.
SkillForgeBench resolves this tension by using agent skills as a unifying execution blueprint across three co‑designed stages: resource preparation, skill‑driven task synthesis, and holistic evaluation design.
In the resource preparation stage, domain‑balanced skills guide artifact retrieval, establishing wide workflow coverage.
During task synthesis, skills serve as procedural anchors; artifact roles are partitioned, natural‑language task instructions are generated, and sandboxed scenarios are packaged to enable deep procedural execution.
The evaluation design stage translates skill‑embedded requirements into fine‑grained completion and process rubrics, allowing strict requirement‑level scoring.
The benchmark comprises 842 sandboxed tasks across five major professional domains. Experiments reveal a steep capability hierarchy among eight mainstream agents, with Opus 4.8 achieving the highest strict completion rate of $40.9\%$.
When all constraints must be satisfied, severe bottlenecks appear across all model tiers: individual rubric pass rates range from $70.0\%$ to $85.7\%$, whereas strict completion drops to $12.6\%$–$40.9\%$.
External guidance from skills yields larger gains for top‑tier agents; Opus 4.8 improves by $19.2\%$ compared to a $5.8\%$ gain for Haiku 4.5.
Agents meet process rubrics far less often than completion rubrics—Haiku 4.5’s process pass rate is under $3\%$—demonstrating that completion‑only checks overestimate real‑world reliability.
Review: SkillForgeBench provides a benchmark that simultaneously captures breadth and depth of agent execution, exposing systematic shortcomings in current models’ adherence to procedural requirements.