In retrieval‑augmented generation (RAG) systems, group‑fairness metrics are usually computed by aggregating over all queries. This aggregation can hide disparities at the topic level: a system may favor women on human‑rights questions and men on actor‑related questions, appearing neutral overall while no topic is truly fair.
To address this, we define topic‑level fairness, requiring fairness to hold within each topic rather than only in aggregate. We reduce topic disparities by adapting a reranker for each topic using a Bradley‑Terry objective supervised directly by the generator’s output: preference pairs derived from the generator’s own responses encode the direction required by the topic.
With preference pairs built solely from RoMQA, the adaptation cuts the mean absolute per‑topic disparity by 51% on RoMQA and by 40% on TREC Fair Ranking, while preserving accuracy. No baseline achieves this reduction, even when modified to treat topics separately. A single reranker for all topics also fails, as it follows the majority direction.
Review