The control of scientific characterization tools, such as microscopes and synchrotron beamlines, using large language models (LLMs) is an increasingly active area of research. When designing and optimizing an agentic microscope controller, researchers want to ensure that the agent can not only perform known tasks but also generalize to new tasks. To address this, we developed a benchmark and trace-logging framework to evaluate the impact of different agent architectures on microscopy tasks and the limitations of benchmarks for predicting agent performance on unseen tasks. We evaluated multiple agent configurations, including one-, two-, and three-agent graph topologies, five LLMs, RAG and context parameters, and operational constraints. Our results show that while benchmarks are useful for qualification, regression testing, diagnosis, and direct comparison, they do not reliably predict an agent's performance on new tasks. Blogger's Review: The development of benchmarks and evaluation frameworks is crucial for ensuring the performance and reliability of intelligent microscope controllers. Although the current benchmark set has its limitations, it remains a valuable tool for evaluating and comparing different agent configurations.