Large Language Models (LLMs) have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs' ability to rectify their own code through execution feedback.
Using real-world programming problems across four models and two major programming languages, the study evaluates performance using an iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. Metrics are introduced to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning versus non-reasoning models, providing actionable insights into the understanding and practical application of feedback loops in LLM-driven code generation systems.
Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.
Blogger's Review: This paper delves into the iterative correction capabilities of LLMs in code generation, highlighting the significance of feedback mechanisms. The findings offer fresh perspectives on enhancing practical utility in code generation, particularly as reasoning abilities become crucial when tackling complex programming challenges.