NeFut Logo NeFut
Admin Login

[CS.AI] Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

Published at: 2026-09-11 22:00 Last updated: 2026-09-12 06:35
#AI #LLM #Planning

Large language models (LLMs) have shown promise for translating natural language (NL) planning descriptions into PDDL problem instances. Yet, evaluating only syntactic validity or planner success often overestimates fidelity: a generated problem may be parsable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target.\ \ This work investigates an end‑to‑end NL‑to‑PDDL pipeline that integrates LLM generation, PDDL parsing checks, planning execution, problem validation, domain‑conformance checking, an LLM critic, and iterative repair. Fine‑grained repair feedback is derived from the domain description, the generated problem, the NL description, and operational diagnostics.\ \ For evaluation, reference‑based comparisons against curated benchmark PDDL descriptions are performed offline. The comparison uses renaming‑invariant structural matching and, when domain support is available, semantic equivalence checking. Experiments span Planetarium, AutoPlanBench, and curated PDDL 2.1 problems. Results reveal a substantial gap between operational success and benchmark‑reference reconstruction; structured repair proves beneficial; and even with improved operational success, reference reconstruction for PDDL 2.1 remains challenging.\ \ Review: The paper highlights the shortcomings of success‑only metrics and proposes a domain‑aware repair loop, offering a practical route toward more faithful NL‑to‑PDDL translation.

Original Source: https://arxiv.org/abs/2609.09898

[h] Back to Home