Deployed machine learning models often suffer performance loss when the distribution of their input data shifts. Such shifts stem from two distinct sources: changes in the underlying population and data‑integrity faults in the feature‑generation pipeline, each affecting the model through different mechanisms. Because true labels are frequently unavailable or delayed, practitioners typically monitor data drift as a proxy, yet a detected drift does not reveal its impact on model performance.
We formalize this challenge as online model testing under distribution shift. The test requires reproducing the observed drift on a controlled baseline where true labels are known. Since the same drift pattern may arise from either population change or data‑integrity error, reproducing it demands first identifying the responsible mechanism.
Our proposed methodology separately simulates population distribution changes and feature‑generation faults on the baseline, then combines them to estimate the model’s current performance and the potential gain from fixing data‑integrity issues.
Experiments on several real‑world datasets with injected faults show that disentangling the two sources generally yields more accurate performance estimates and gain predictions, supporting our formulation. A churn‑prediction case study demonstrates how these tests can aid monitoring, fault diagnosis, and repair‑priority decisions.
Review