Abstract
Machine learning-based Type 2 diabetes risk prediction models achieve good internal validation results but fail to perform effectively in real-world applications due to insufficient external testing and fairness assessment. We developed a multi-dimensional framework to evaluate discrimination, calibration, interpretability, and algorithmic fairness on nationally representative populations.
An XGBoost model was trained on NHANES 2015-2020 (n=15,685) using eight non-laboratory predictors: age, sex, race/ethnicity, BMI, smoking status, physical activity, history of heart attack, and history of stroke. External validation was performed on BRFSS 2020-2022 (n=1,285,783) under realistic distribution shift. Internal validation showed good discrimination (AUC=0.794, 95% CI 0.788-0.800), with performance loss in external validation (AUC=0.717, relative decrease: -9.7%, p<0.05), and AUC for young adults showed 0.607 vs 0.742 (difference=0.135, p<0.05).
Conclusion
This study highlights the importance of evaluating machine learning models across diverse populations, emphasizing the need for external validation and fairness analysis in practical applications.
Blogger's Review: The multi-dimensional framework proposed in this study provides crucial insights into the effectiveness of machine learning models in real-world applications, particularly in terms of fairness and external validation, which is noteworthy and worth referencing.