In the field of hardware design, Large Language Models (LLMs) have demonstrated remarkable capabilities in Register Transfer Level (RTL) code generation, particularly in the application of Verilog. However, the evaluation of their performance concerning other Hardware Description Languages (HDLs), especially VHDL, remains relatively limited. This is primarily due to VHDL's characteristics, such as stricter semantic rules, which introduce evaluation considerations that differ from those of Verilog. This gap restricts our comprehensive understanding of how well current models generalize across hardware design languages with differing structures and semantics.
To address this issue, we introduce VHDLSuite, a benchmark-centered infrastructure for scalable VHDL generation evaluation that integrates automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. First, we propose a data pipeline that automatically converts Verilog designs and their accompanying testbenches into executable VHDL benchmark instances, followed by VUnit/GHDL-based validation to ensure each released task is compilable, runnable, and consistently checkable in the VHDL environment.
Second, we introduce VHDLBench, a benchmark containing over 200 VHDL problems with complete and validated testbenches across a wide range of complexity levels. Finally, we extensively evaluate cutting-edge LLMs and uncover key challenges specific to LLM-aided VHDL generation. Our findings provide important insights and support future work in multi-language hardware design automation.
Our data pipeline, benchmark, and evaluation framework will be open-sourced for public use.
Blogger's Review: The introduction of VHDLSuite provides a significant foundation for VHDL generation evaluation, especially amidst the growing application of LLMs. With automated benchmark synthesis and validation, developers can assess model performance more efficiently, which will promote broader research and application of hardware design languages. The open-source initiative will also foster further community development.