This work examines whether passing a test in automated discovery systems truly guarantees the intended scientific claim. We introduce four executable operations for modifying physical descriptions: L1 identifies redundancies, L2 coarse‑grains dynamics, L3 transfers structure, and L4 infers missing quantities from conservation imbalances. The operations are evaluated under three interpretability settings: null acceptance (NAR), positive acceptance (PAR), and consequence agreement (CAR).
A reduction that maps every state to zero predicts its own future perfectly. It passes the closure test on 96% of 129 public chaotic systems despite retaining no dynamical variation. An independent observable‑preservation test rejects it throughout. On controlled benchmarks, the combined criteria accept all 24 genuine reductions and reject all 72 null reductions. The other operations distinguish symmetries from redundancies, validate transfers, and reject spurious conservation imbalances.
In language models and the ESM‑2 protein model, linear CKA remains unchanged when a proposed correspondence is randomly rotated, whereas localized interventions separate genuine mappings from tested nulls. In sparse autoencoders, enrichment rejects all 16 tested nulls, yet independent ablation supports 45% of evaluable enriched features compared with 0.5% of matched controls.
These findings reveal two acceptance gaps: descriptive tests can accept scientifically empty candidates, and accepted candidates may lack independently verified consequences. Reliable automated discovery therefore requires evaluating both what a test accepts and the scientific consequences it establishes.
Review