Evaluating parallelization strategies through distributed training runs is prohibitively expensive at scale, while performance simulation offers a cheaper alternative. Existing simulators differ in the training stages they model and the workloads they use, making fair comparison difficult.
LM-SimBench introduces a unified framework that decomposes simulation into functional stages from model compilation to schedule‑level aggregation, linking execution representations, cost estimators, and measurable evaluation targets.
Following this framework, we profile 23 model settings from 10 dense and Mixture‑of‑Experts LLM families on an Ascend NPU cluster. The resulting dataset contains 348 feasible configurations that jointly vary five parallelism dimensions and activation recomputation, and provides six views spanning configuration, rank, operator, event, and kernel granularity.
Evaluations with representative training simulators demonstrate how LM‑SimBench supports comparison of prediction accuracy, assessment of configuration selection, and diagnosis of prediction errors.
Matched GPU‑NPU experiments validate cross‑platform applicability and show that NPU‑guided GPU configuration selection is effective.
Review