Large language models often emit natural‑language comments while coding, and those comments become part of the context for the next code generation step. It is still unclear which comment properties actually affect generation performance. We first performed observational analyses on LiveCodeBench and found that neither how often comments appear nor their broad intent reliably predicts pass@1. We then prefixed weaker recipient models with comment blocks written by stronger source models, allowing us to separate the comment’s surface form from the solution content it conveys. Comments extracted from source solutions that pass the tests raise the recipient’s pass@1 by about 17.2% on average. In contrast, comments describing failed solutions provide no reliable gain, and comments written for a different problem reduce pass@1 by roughly 20.8%. Across many models and prompt variants, most recipients fail to recover the external‑comment benefit, with the best case recovering only about 24%. These results show that comments help code generation not merely because they are comments, but because they can supply correct solution content that prompting alone cannot reliably elicit.
Review