The notion of algorithmic fairness has been actively explored, particularly the relationship between counterfactual fairness (CF) and group fairness (GF). However, in image classification tasks, this relationship remains unclear due to the difficulty of collecting counterfactual samples related to sensitive attributes (e.g., a photo of the same person with different secondary sex characteristics).
In this paper, we construct new image datasets, utilizing high-quality image editing methods and human annotators for labeling. Our datasets, \oursceleb and \ourslfw, are built upon popular image GF benchmarks, allowing simultaneous evaluation of CF and GF.
We empirically observe that CF does not imply GF in image classification, contrasting with previous studies on tabular datasets. Theoretically, this discrepancy may arise from a latent attribute $G$ that is correlated with, but not caused by, the sensitive attribute (e.g., secondary sex characteristics highly correlate with hair length).
From this observation, we propose a simple baseline, Counterfactual Knowledge Distillation (CKD), to mitigate such correlation with sensitive attributes. Extensive experimental results on \oursceleb and \ourslfw demonstrate that models achieving CF can satisfy GF if we successfully reduce reliance on $G$ (e.g., using CKD).
Blogger's Review: This paper makes a significant contribution by constructing new datasets and proposing the CKD method, shedding light on the intricate relationship between counterfactual and group fairness in image classification. The findings offer valuable insights for understanding algorithmic fairness, advancing both theoretical and practical implications in the field.