Abstract
On-device research agents search a corpus, read sources, and write a cited brief on a personal laptop. This study examines the faithfulness of citations and their associated costs for a deployable small model, focusing on a 4B generator on a 24GB laptop.
It separates two quantities typically reported together: citation faithfulness, which assesses whether the cited source supports the claim, and trustworthy coverage, which evaluates whether the agent cites the correct sources. The study compares the exposure of the generator to different amounts of source content—400 characters versus 1500 characters—and the quality of sources, including gold papers versus retrieved papers.
The findings reveal that exposure influences faithfulness. Increasing exposure to each source raises faithfulness from 0.45 to 0.58 for retrieved sources and from 0.37 to 0.58 for gold sources, indicating that faithfulness is bounded by exposure rather than the correctness of the source. This exposure lift remains robust under a second independent judge, with tighter convergence under the primary judge and only approximate under the second.
Retrieval impacts coverage. Trustworthy coverage remains near 0.22 for retrieved sources regardless of exposure, as recall is held around 0.40, indicating that exposure cannot fix which sources are cited. The additional exposure costs about 235 output tokens. The practical recommendation is to first increase per source exposure cheaply, and then treat retrieval recall as the only remaining lever.
Blogger's Review: This research delves into the relationship between citation faithfulness and coverage in on-device generative models, highlighting the critical impact of exposure on outcomes, offering valuable practical guidance that warrants attention and consideration.