For Internet of Things (IoT) devices, a secure algorithm alone cannot stop an attacker who has physical access and can target the implementation directly; such flaws are hard to fix after deployment. Large language models (LLMs) are now being used to build and analyze these implementations, yet existing cryptography or general cybersecurity benchmarks do not cover cryptographic engineering comprehensively. This paper introduces CESBench, a benchmark containing 380 expert‑written items across six sub‑domains of cryptographic engineering security for IoT devices: side‑channel, fault injection, implementation details, countermeasures, evaluation, and integration. The items are organized into four task types to probe different competencies: 209 multiple‑choice questions test recall, 67 judgment items require a security verdict with justification, 63 scenario items demand an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open‑weight and proprietary LLMs answered every item. Multiple‑choice and code responses were scored automatically, while judgment and scenario responses were evaluated by an LLM judge, whose scores were cross‑checked with a second judge from another model family and with human re‑scoring. Composite scores range from 54.4% to 83.6%. The best scores per task type are 98.6% for multiple choice, 95.1% for code, 88.4% for scenario diagnosis, and only 58.8% for judgment. Across models, 88.5% of verdicts are correct, but their justifications receive only 53.4% of the rubric marks. Multiple‑choice performance is near its ceiling and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per‑item results are publicly available.
Review