Most code‑generation benchmarks for large language models focus on well‑resourced languages such as Python or Java, where abundant training data gives models an advantage. Consequently they offer little insight into reasoning over unfamiliar computational models—for example deriving arithmetic from a single subtract instruction, coordinating parallel programs across communicating nodes, or wiring logic gates into circuits. ISA‑Bench addresses this gap by offering a suite of programming games built on constrained instruction sets. For each game we supply a complete execution stack—parser, virtual machine, and verifier—enabling fully automated evaluation with structured feedback for iterative refinement. Experiments show that reasoning‑oriented models achieve higher average solve rates than code‑specialized and general‑purpose models, yet unfamiliar syntax remains a dominant source of failure. Providing iterative feedback improves solve counts, though gains vary widely across architectures. We introduce a reasoning‑execution gap (REG) analysis that highlights a recurring disconnect between identifying a plausible computational strategy and expressing it correctly in the target ISA. The codebase is open‑source.
Review: ISA‑Bench offers a reproducible framework to probe LLMs’ ability to reason and synthesize programs on non‑standard ISAs, exposing syntax adaptation and program synthesis as key bottlenecks.