NeFut Logo NeFut
中 Admin Login

[CS.AI] What Git Does Not Record: Provenance Legibility in Two AI‑Assisted Research Repositories

Published at: 2026-10-23 22:00 Last updated: 2026-10-25 13:04
#AI #Machine Learning #Open Source

Publishers and funders are increasingly demanding that authors disclose how AI tools contributed to their research. In computational work the code repository is the primary place where an author or auditor looks for such evidence. We audited two single‑author repositories in the same field, each preserving a different class of agent residue: a prescriptive trace (standing rule files) that records static rule statements, and a narrative trace (running agent scratchpad) that logs the agent’s interactive notes. The repositories contain 466 and 103 commits respectively.

For the prescriptive trace, 85.6% of the lines ever committed remain visible in the current revision; the standing rule typically states a conclusion and discards the episode behind it. By contrast, the narrative trace shows only 21.8% of its lines at the tip. When measured by revision, the gap narrows: out of 41 revisions of the prescriptive trace, 15 leave no visible line, while 18 of the 41 revisions of the narrative trace are completely invisible.

We also examined the Co‑Authored‑By trailer, the field an automated audit reads. In Project A it appears in 66 of the 121 commits that carry any AI marker (≈ 54.5%). In Project B its sensitivity to an explicit in‑repository proxy is low: 39 commits modify a file meant for the agent’s working notes, and only 1 (2.6%) of those carries the trailer. Project A’s repository‑level rate is unstable over time, jumping from 2.7% to 17.7% across a boundary we did not pre‑define.

Different audit channels disagree, and neither rate can be read as a true involvement rate. We provide a field‑by‑field account of which provenance claims each trace can support, which it cannot, and which it answers confidently yet incorrectly. From this we construct an auditable evidence ledger and derive the requirements any system must meet to record research provenance as it happens.

Review: The paper exposes Git’s blind spots in capturing AI‑agent activity, highlights the diversity and limits of current audit methods, and offers empirical guidance for building more transparent research provenance systems.

Original Source: https://arxiv.org/abs/2610.28655

[h] Back to Home