Abstract
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude.
Methodology
Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments.
Results
Our results show that:
- English prompts do not consistently produce the best functional correctness or code quality;
- The impact of prompt language depends on both the programming language and the LLM;
- Generated code frequently mixes English with the prompt language in comments and string literals.
These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.
Blogger's Review: This article reveals the profound impact of multilingual prompts on code generation quality, highlighting the importance of language selection in programming contexts. The findings suggest that the functionality and structural quality of code do not always correlate with the choice of prompt language, which is a crucial consideration for developers and researchers.