NeFut Logo NeFut
中 Admin Login

[CS.AI] ProgramGym: Scalable Training Environments for Long-Horizon Program Reproduction Agents

Published at: 2026-10-13 22:00 Last updated: 2026-10-14 02:30
#AI #Machine Learning #LLM

ProgramGym is a training framework for long‑horizon program reproduction agents, consisting of two main components: Program‑Env and Program‑Agent.\ \ Program‑Env builds verifiable training environments from real repositories via a feedback‑driven loop. The loop automatically repairs runtime sandbox build failures and improves the quality of black‑box tests, ensuring that each environment is executable and that evaluation is reliable. Using this pipeline we generated 1,648 environments covering C, C++, Go and Rust.\ \ Program‑Agent follows a role‑isolated workflow that separates analysis, implementation and review into distinct stages, and introduces revision management for compilation checks and failure recovery. After code generation, the implementation stage triggers an immediate compilation check; on error the agent rolls back to the last successful revision and retries, enabling continuous iteration over long development horizons.\ \ Experiments on the ProgramBench benchmark show that both Program‑Env and Program‑Agent substantially improve program reproduction success rates. Fine‑tuning Qwen3.8‑27B on trajectories from the constructed environments raises accuracy from 34.87% to 42.84%. Further fine‑tuning with Program‑Agent lifts the base and SFT models to 55.71% and 56.25%, surpassing several flagship models.\ \ We will release the ProgramGym code and datasets to support research on coding agents that build programs from scratch.\ \ Review: ProgramGym offers a systematic approach to training and evaluating long‑horizon code generation, notably advancing environment verifiability and workflow robustness.

Original Source: https://arxiv.org/abs/2610.13395

[h] Back to Home