Abstract
Text-to-Image (T2I) diffusion models often propagate harmful bias inherited from the training data. Existing bias mitigation techniques typically intervene only at the text encoder or provide inference-time guidance, often leading to semantically incoherent outputs. To address these limitations, we introduce CO-ALIGN (Concept Ontology Alignment), a novel bias mitigation approach based on concept-graph alignment that operates on the model's internal concept ontology.
By aligning concepts within the text encoder and denoiser, CO-ALIGN achieves substantial bias reduction while preserving generative integrity. We demonstrate the effectiveness of concept-graph alignment across three paradigms: text-encoders, denoisers, and joint text-denoiser ontology alignment. CO-ALIGN outperforms the state of the art, improving fairness by 30%, $\text{FID} = 11.4$ in image quality, 2.8% in image fidelity, all while reducing semantically incoherent outputs by 88%.
Beyond bias mitigation, we show that CO-ALIGN benefits other downstream tasks as well. In particular, our experiments demonstrate that better-aligned internal ontologies enhance concept unlearning robustness across multiple unlearning techniques.
Blogger's Review: The CO-ALIGN method effectively mitigates bias through concept graph alignment, surpassing existing techniques across multiple metrics. This not only enhances fairness and generative quality in T2I models but also opens new avenues for future research, making it a noteworthy advancement in the field.