Addressing structural engineering requests requires more than a single answer; it necessitates a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report. Traditional evaluations centered on question answering or script generation often fail to verify this complete evidence chain, potentially rewarding fluent outputs even when the underlying engineering workflow is incomplete, inconsistent, or non-executable.
To tackle this limitation, we present StructureClaw, an artifact-centered workbench where LLM agents operate through governed engineering skills, typed tools, shared artifact state, and local analysis backends. We also introduce StructureClaw-Bench, an executable benchmark comprising 150 controlled scenarios covering standard workflow execution, interactive robustness, and multimodal structural-model reconstruction. A scenario succeeds only when all required artifact- and execution-level assertions pass in a single run.
Across ten agent-model configurations, each evaluated on the same 50 standard cases, the average Success Rate rises from 56.8% with the generic-skill baseline to 88.6% with the full automatic workflow. Interactive and multimodal evaluations identify two prominent challenges: safe handling of invalid numerical inputs and fixture-consistent reconstruction of structural models. These findings demonstrate that artifact-centered evaluation can expose workflow-level failures that are difficult to identify from final responses alone, providing a more rigorous basis for evaluating and improving structural engineering agents.
The code and benchmark are available at GitHub.
Blogger's Review: By introducing structured documentation and executable benchmarks, StructureClaw offers a novel perspective on assessing LLM applications in structural engineering, emphasizing the importance of complete workflows. This approach not only enhances success rates but also paves the way for future research, particularly in tackling complex engineering challenges.