LLM‑based agents are increasingly credited with practical reasoning abilities such as planning, monitoring, and self‑correction. Their success is often attributed to prompting techniques—chain‑of‑thought, search‑based prompting, self‑refinement, reflection, critics, and tool‑using loops—and to specific agent architectures. However, these claims are usually supported only by end‑to‑end success rates (indicating whether a task was completed) or by LLM judges, which are not true verifiers.\ \ We evaluate LLM planning against formal semantics: each natural‑language task is grounded in a hidden PDDL model that determines action applicability, goal achievement, and the existence of a plan. Experiments span five domains (two newly introduced and released with the paper), four language models, six prompting strategies, and three agent architectures. Offline, the best configuration solves 78% of solvable tasks; online—where agents act with minimal state inspection—drops to 40%. Unsolvable tasks are recognized in fewer than 10% of episodes. Heavier prompting consumes 5–22× more tokens than direct prompting without clear performance gains. Planning performance varies more across models than across prompting strategies or architectures.\ \ Conclusion: Claims that agents can plan must be backed by validated action traces rather than mere success rates or LLM judges.\ \ Review