Robust Counterfactual Explanations (CFE) promise that recourse remains valid after the underlying model changes. Whether this promise holds depends on the nature of the change. Small parameter perturbations, retraining on new data, and swapping to a new architecture are distinct events, yet existing methods are typically evaluated only against the specific change they were designed for, so reported robustness scores answer different questions and cannot be directly compared.
To address this, we propose a unified cross‑family evaluation protocol. The protocol fixes factual instances and generated counterfactuals, then tests every method against the same eight types of model change (including parameter perturbations, bootstrap retraining, architecture changes, etc.), enabling a fair comparison across methods.
We conduct experiments on four tabular datasets, comparing six robust CFE methods and two standard baselines. Each altered classifier is characterized by its output changes, and we report empirical robustness together with coverage, base validity, and proximity.
Results show that relative performance and failure modes vary across change families. Bounded parameter perturbations affect only $0.95\%$ of test predictions on average, whereas bootstrap retraining changes about $4.9\%$. Methods that guarantee robustness for parameter perturbations do not necessarily transfer to other changes.
RobX exhibits the most consistent transfer in our experiments, although achieving greater stability can require larger interventions.
We argue that robust CFE methods should be evaluated through a common protocol that specifies the model changes, measures their realized behavioral magnitude, and keeps generation performance separate from robustness.
Review