Validating autonomous driving systems requires diverse, regulation-compliant test scenarios. In simulation-based testing, scenarios are defined as executable scripts. However, automatically generating such scripts from regulatory descriptions remains an open challenge, with existing approaches facing fundamental trade-offs.
Retrieval-assemble methods achieve reasonable compilation rates but lack scalability, while retrieval-based full-script generation suffers from low compilation success rates.
We present Chat2Scenic, the first iterative retrieval-augmented framework for generating scenario scripts in Domain Specific Language (DSL). Specifically, Chat2Scenic offers a chatbot interface for interactive scenario refinement and integrates Retrieval-augmented Generation (RAG) to ground scenario generation in regulatory knowledge and DSL syntax.
Furthermore, we propose an open benchmark for scenario generation comprising 123 scenarios from various regulations, including NHTSA and United Nations Vehicle Regulations, among others.
Extensive evaluation with State-of-the-Art (SOTA) Large Language Models (LLMs) demonstrates that Chat2Scenic achieves a 76.42% Compilation Success Rate (CSR) and a 58.17% Framework Accuracy (FA), outperforming existing methods (Retrieval Assemble with 30.08% CSR, 11.03% FA and Retrieval full script generation with 16.26% CSR, 10.86% FA).
To facilitate future research, we release our code as open source at GitHub.
Blogger's Review: Chat2Scenic significantly enhances the efficiency and accuracy of scenario script generation for autonomous driving by leveraging retrieval-augmented generation technology. This tool and benchmark provide substantial support for industry compliance and reliability in autonomous driving systems. Its open-source code lays a solid foundation for future research and exploration.