Automated candidate‑job matching systems are increasingly classified as high‑risk AI under emerging regulations, yet auditing them for demographic bias is costly. Traditional correspondence audits require handcrafted resumes and manual submissions, which do not scale to rapid model retraining cycles. This paper proposes a general, reusable methodology consisting of four key steps:
-
Employ task‑specialized LLM agents to generate identity‑neutral base resumes and inject controlled demographic treatments across five protected‑characteristic axes (sex/gender, age, residence, language, disability), yielding a $K \times (1+N)$ correspondence‑audit matrix.
-
Use an EU AI Act‑aligned prompt to qualitatively flag inferred protected characteristics for each resume.
-
Rank candidates against a job description with a fine‑tuned sentence‑embedding model using cosine similarity, where $cos(\mathbf{v}_1,\mathbf{v}_2)=\frac{\mathbf{v}_1\cdot\mathbf{v}_2}{\|\mathbf{v}_1\|\|\mathbf{v}_2\|}$.
-
Compute a nine‑metric fairness suite covering three families:
- Counterfactual metrics: score delta, mean absolute rank change, flip rate.
- Group‑fairness metrics: top‑K retention, four‑fifths/impact ratio.
- Merit‑aware metrics: Recall@K, nDCG@K, equal opportunity, equalized odds. Each metric includes bootstrap confidence intervals, significance tests, and Benjamini‑Hochberg correction, culminating in an automated PASS/INVESTIGATE/FAIL report with a composite risk score.
In an example corpus of 5 job orders, 100 base candidates, and 10 demographic treatments (90 metric‑variant evaluations), all treatments stayed within tolerance for score shifts, top‑K retention, and merit‑aware gaps. However, the rank‑stability metric (MARC) and nDCG@K surfaced borderline findings—including one on the neutral baseline itself—that would be missed by a score‑only or retention‑only view. The results argue for multi‑metric, multi‑family auditing over any single aggregate score, and for LLM‑agent‑generated audits as a practical, low‑cost complement to human‑curated audits for any candidate‑job matching pipeline.
Blogger's Review: This work convincingly demonstrates the potential of LLMs to automate fairness audits in fast‑moving recruitment systems. The multi‑metric framework offers finer‑grained risk insight, making it a valuable addition to real‑world product deployments.