We investigate whether AI agents powered by locally deployed large language models (LLMs) can reliably automate expert‑defined hardware design workflows in an industry‑realistic tool‑calling setting. In such environments engineers perform repetitive, dependency‑ordered operations—creating components, adding ports, wiring connections—through specialized tools. Confidentiality constraints on component specifications and naming conventions often preclude the use of hosted proprietary APIs, motivating the adoption of local models.
To study this, we implemented a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development. Based on this server we constructed a benchmark covering single‑operation edits, multi‑step dependency chains, invalid requests, misspelled prompts, and multi‑server tool contexts.
We evaluated seven open‑source models, comparing pipeline choices such as system prompts, tool‑description detail, context scope, and single‑agent versus multi‑agent architectures. Results show that strong models can achieve near‑complete expected‑call coverage on the benchmarked workflows, yet reliability depends heavily on task structure and agent configuration. Key observations include:
- Comprehensive tool descriptions consistently reduce failures;
- Few‑shot prompting can cause severe inaction for some models;
- Cumulative context harms constrained models;
- Multi‑agent decomposition helps weak workers or long sessions at the cost of additional calls.
These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.
Blogger's Review: The paper presents a well‑designed MCP benchmark and a thorough evaluation of local LLMs for hardware design automation. The methodology is rigorous, and the insights—especially regarding tool description granularity and multi‑agent strategies—offer actionable directions for industry adoption.