PhysAI-Bench is a benchmark designed to evaluate agentic decision‑making in Physical AI scenarios, specifically for autonomous UAVs. It contains 10,178 standardized decision instances automatically extracted from conversational traces of UAV missions. Each instance preserves mission context, temporal dependencies, physical constraints, Model Context Protocol (MCP) tool calls, Agent‑to‑Agent (A2A) interactions, sensor observations, and AI‑native 6G network conditions such as latency, packet loss, throughput, edge load, and network slicing. Only information available before each decision is exposed, preventing future‑event leakage and approximating online decision making.
We evaluated 29 foundation models using a two‑stage protocol. First, on a human‑verified development set of 35 instances, we tested 12 configurations combining zero‑, three‑, and five‑shot prompting with four temperature values, each run three times, and selected the best configuration per model. Then we froze each selected configuration and evaluated it on a disjoint test set of 500 instances, again with three runs. GPT‑5.3 achieved the highest accuracy (52.00%), followed by GPT‑5.2 (49.40%) and Grok 4.5 (49.07%). Few‑shot prompting generally improved performance, while temperature had limited impact. The results indicate that reliable agentic decision‑making in Physical AI remains an open challenge. The dataset is available at https://github.com/maferrag/physai-bench.
Review