In many review pipelines the final verdict is the only information retained, while the text passages that justify it are rarely annotated because labeling costs far exceed simply recording the decision. This paper investigates whether a small language model can recover such evidence when it is post‑trained solely on verdict labels, without any human evidence annotations at any stage. Experiments are conducted on the ContractNLI dataset, where human evidence spans are held out until evaluation. Two post‑training strategies are compared:
- Label‑only training, where the model sees only the verdict labels;
- Rejection sampling, which keeps a generated trace only if its verdict matches the recorded one and then selects the best trace using an automatic source‑grounding score.
Results show that label‑only training already raises accuracy to 0.896 and span F1 to 0.564. Rejection sampling achieves 0.797 accuracy and 0.556 span F1, both improvements over the pre‑training scores of 0.747 and 0.493. Verbatim citation rates increase from 0.597 to 0.729 with label‑only training and to 0.701 with rejection sampling. Although a single seed on one corpus cannot definitively declare a superior method, both approaches substantially enhance evidence recovery without any human annotation.
Review: The study demonstrates that, in annotation‑scarce review settings, fine‑tuning on verdicts alone or applying rejection sampling can markedly improve the traceability of evidence while keeping labeling costs minimal, offering a practical low‑cost solution for real‑world systems.