Abstract
Federated Continual Learning (FCL) evaluates how distributed clients learn from changing data streams while retaining previously learned knowledge. Existing evaluations are challenging to compare due to simultaneous changes in datasets, task splits, client data splits, task orders, backbones, memory assumptions, and reporting rules.
We introduce HERO, a heterogeneity-aware benchmark library for FCL. HERO builds benchmark streams by separating three often-coupled choices: task split, client data split, and client task sequence. In HERO-Core, the main comparable benchmark, $\eta$ controls client data skew and $\rho$ controls task-order mismatch.
We evaluate representative FCL methods on CIFAR-100 and TinyImageNet using final average accuracy, average forgetting, and bottom-10% client accuracy. Additionally, we include a graph-based Domain-IL portability case study on OGB-MolPCBA, where scaffold-domain granularity changes the input distribution while the prediction task remains fixed. Our results show that method behavior varies across easy and heterogeneous settings, that average accuracy can obscure weak bottom-client performance, and that task-order mismatch favors different strategies from synchronized evaluation. The same HERO interface can expose domain-shift difficulty beyond image-based FCIL.
HERO releases benchmark streams, configurations, method implementations, and reporting scripts to support reproducible and setting-aware FCL evaluation.
Blogger's Review: HERO offers a flexible and comprehensive approach to evaluating federated continual learning methods, especially in heterogeneous data environments. By clearly separating various factors, researchers can better identify and optimize model performance, driving further advancement in this field.