NeFut Logo NeFut
中 Admin Login

[CS.AI] ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024

Published at: 2026-11-04 22:00 Last updated: 2026-11-05 00:45
#AI #Machine Learning #Data Structure

We introduce ReuScope, a protocol for auditing data reuse and evaluation sensitivity, demonstrated on Capsule Vision 2024. The method jointly measures two comparisons: (1) changing the evaluation population while keeping model predictions fixed, and (2) changing the training data while keeping the evaluation set fixed.\n\nByte‑identical files cross the public training‑validation boundary, and all 41 Kvasir‑Capsule recordings that contribute frames appear in both partitions, creating cross‑split overlap. Missing patient identifiers across recordings prevent counting unique shared patients.\n\nAcross ten paired random seeds, removing every training row attributed to Kvasir‑Capsule plus five rows flagged as near‑matches reduces public‑validation balanced accuracy from $82.5\%$ to $61.2\%$. Evaluating the same fitted models on validation rows from other sources reverses the comparison: removal improves balanced accuracy by $1.38$ percentage points. The drop therefore measures sensitivity to source removal, not leakage‑induced inflation.\n\nMatched‑control experiments show that sample size and class counts alone cannot explain the observed score differences. An exploratory experiment removes training neighbors of validation frames that have close training matches, or an equal number of distant training rows, yielding a difference‑in‑differences estimate of $-0.355$ percentage points in true‑class probability relative to matched unflagged frames; the paired‑$t$ interval for accuracy spans zero, indicating no significant effect.\n\nThe procedure distinguishes observed reuse from conditional performance sensitivity and explicitly specifies the population, class support, and uncertainty unit for each conclusion.\n\nReview: ReuScope offers a systematic framework that captures actual data reuse while quantifying its impact on evaluation outcomes, establishing a reproducible benchmark for data‑leakage auditing.

Original Source: https://arxiv.org/abs/2610.41835

[h] Back to Home