This paper empirically investigates whether contemporary large language models can faithfully translate informal testing goals written by video‑game testers into well‑formed PDDL specifications suitable for classical planning. A prompt template, refined through iterative experimentation, is introduced to maximize both accuracy and response coherence. Six state‑of‑the‑art models are evaluated on real‑world, domain‑specific benchmarks using three metrics: correctness, latency, and error patterns. All models achieve correctness above 92%. Gemini 2.5 Flash attains the highest accuracy at 96% and the lowest false‑positive rate, while GPT‑4.1 leads in response speed. Despite these advances, occasional failures arise from linguistic ambiguity and limited domain representation. The analysis highlights significant progress in using LLMs as bridges between natural‑language objectives and automated planning pipelines, yet underscores remaining gaps in robustness and domain adaptation.
Review