Frontier AI evaluations are shifting toward open‑ended, agentic, long‑horizon tasks whose transcripts can span hundreds of pages and involve complex multi‑agent networks. Consequently, the observability envelope—the range of behaviours that evaluators can reliably infer—is shrinking. Language‑model assistants can classify and interpret agent behaviour, but they also give human evaluators substantial analytical freedom, endangering the reproducibility and auditability of transcript analysis based on language models.
Transect is an open‑source package built on Inspect Scout that helps evaluators understand how a long‑run agent unfolded, spot behaviours worth deeper investigation, and verify interpretations against the raw transcript. Users define task context and a behavioural vocabulary in a reusable evaluation‑family configuration, while judge models and analysis settings are supplied separately.
The navigable reports produced by Transect align recorded events, token usage, sub‑agent activity, and model‑generated behavioural labels on a common turn‑based timeline. Reviewers can grasp the narrative of a run within minutes, trace any label or event back to its source turns, and export the underlying data tables for cross‑run analysis.
We demonstrate the workflow on an AI R&D evaluation that generated almost 13 million tokens. By aligning behavioural phases, research‑skill classifications, sub‑agent delegations and interactions, and token consumption, we observed a focus on operational work and manuscript production, with little evidence of a sustained hypothesis‑generation stage—a component typically necessary for high‑quality scientific output.
Transect’s flexible, customizable transcript‑analysis pipeline enables evaluators to keep pace with longer, more complex, and more frequent AI evaluations while supporting scientific rigor, transparency, and reproducibility.
Review