Abstract
Infrastructure-as-Code (IaC) generation from natural language involves satisfying provider schemas, dependency planning, and organizational policy constraints, not merely producing syntactically plausible configurations.
We present a verifier-first empirical study of seven agentic strategies for Terraform generation evaluated on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 intent policies.
Our evaluation separates failures into three verifier stages (terraform validate, terraform plan, opa eval) and applies McNemar's test with Wilson confidence intervals on all pairwise comparisons (n=186, alpha=0.05). We report five principal findings.
- Active retrieval via ReAct agents with MCP or ChromaDB-backed RAG raises Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p < 0.05).
- ... (subsequent findings can be added as necessary)
Conclusion
The verifier-first evaluation method significantly enhances the generation of infrastructure-as-code, showcasing the potential of agentic models to meet complex constraints.
Blogger's Review: This study highlights the application potential of generative models in complex environments through a structured verification process, especially in meeting specific policies and dependencies. Future research could further explore optimizing different strategies to adapt to evolving infrastructure needs.