Large language model (LLM) agents can reason, invoke tools, and act over many steps autonomously, yet their abilities to correctly sequence tools, plan under dependencies, judge untrusted inputs, and ground arguments are hard to capture with accuracy‑only leaderboards. BekchiAI tackles both sides by offering a benchmark to measure these agentic skills and a platform to observe and control live agents.
The BekchiAI‑Benchmark comprises 13 ReAct‑style tool‑using agents across seven task categories—arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool‑policy—totaling 2,057 deterministic, committed test cases. Gold answers are generated by executing canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed‑form lambda expressions; security tasks pair adversarial samples with deliberately imperfect signature scanners so that scores reflect the model’s own judgment rather than copying an oracle.
Beyond accuracy, BekchiAI defines behavioral metrics such as tool‑call adherence, URL hallucination rate, source‑match rate, and per‑model token cost. A comparison of four models (Qwen3.7‑Max, gemma‑4‑31B‑it, gemma‑4‑26B, gpt‑oss‑120B) reveals that differences lie in the spread within each model family rather than in aggregate performance. Benchmark runs are executed with the provided evaluation scripts.
BekchiAI‑Platform is a complementary web‑based observability and control layer that records full token and latency telemetry and allows remote termination of runs. The benchmark, evaluation tools, and platform are all released as open source.
Blogger's Review: By unifying rigorous measurement with real‑time observability, BekchiAI gives researchers a holistic view of LLM agents—from fine‑grained tool behavior to overall cost—making its security‑aware and policy‑centric metrics especially valuable for the community.