Selecting among robot demonstration corpora is fundamentally a ranking problem: an upstream diagnostic must preserve which candidates yield better downstream policies, not merely correlate with average performance.
This work studies paired instruction dependence, defined as the rate at which a demonstrator changes its first decision when only the goal changes.
Experiments were conducted on 136 independently seeded quadruped‑navigation corpora. Predictors were trained on 56 corpora and prospectively evaluated on the remaining 80.
The audit proved reliable and associated with downstream paired correctness. Adding it to a correctness‑aware baseline reduced mean absolute prediction error, yet the primary proper‑log‑score gain was only +0.059 nats with a 95% corpus‑bootstrap CI of [-0.027, 0.133]. The pre‑specified top‑20% comparison did not demonstrate an incremental selection benefit.
An exploratory response decomposition suggested a structural source of the ranking mismatch: under the tested predictor specification, separating both‑correct and both‑wrong responses yielded larger held‑out gains than the unsigned switch statistic.
Seed‑to‑seed agreement indicates that the held‑out endpoint still retains recoverable rank signal.
In summary, counterfactual response structure can be reliable and predictive, but when compressed into an unsigned statistic it provides limited incremental decision value in this setting. This motivates diagnostics that preserve outcome‑relevant structure and are validated against the decisions they are intended to support.
Review