CraftBenchUE is an evaluation harness that runs coding agents inside an isolated Unreal Engine environment. It reconstructs each agent's saved submission in a fresh project and performs deterministic build, asset, and runtime checks without relying on an LLM judge. Using this harness we assembled a benchmark of 70 tasks covering C++ source files, Blueprint assets, and editor scripting. Seven models were evaluated under two editor‑tool configurations, with a file‑and‑shell baseline added for the C++ tasks. To compare language differences we paired tasks that specify the same gameplay and share identical runtime tests but require either C++ or Blueprint as the deliverable. Across the 10 paired tasks, C++ completion rates exceed Blueprint by 30.0% and 42.9% respectively. Among on‑time Blueprint submissions that pass asset checks, 42.2% and 50.0% fail explicit runtime assertions, indicating they meet asset requirements yet do not satisfy the required gameplay behavior. The harness, task benchmark, and our trajectory findings will be released with the report.
Review