NeFut Logo NeFut
中 Admin Login

[CS.AI] OmicsBench: A Long-Horizon Multi-Turn Benchmark for AI Agents on Research-Study-Scale Bioinformatics

Published at: 2026-10-16 22:00 Last updated: 2026-10-18 08:05
#algorithm #AI #Machine Learning

Computational biologists are increasingly offloading substantial analyses to AI agents, especially in research scenarios that require long-horizon, multi-turn interactions. Existing bioinformatics benchmarks, however, typically present a single short task with a one‑shot prompt. To address this gap we introduce OmicsBench, a benchmark specifically designed for long‑horizon, multi‑turn bioinformatics workflows.\ \ OmicsBench comprises 97 tasks that are linked via a directed acyclic graph (DAG), yielding a total of 662 underlying subtasks. Each subtask is presented as a single user turn and is graded deterministically against a reference answer key; optional in‑context feedback can be provided for incorrect responses.\ \ We evaluated 12 models across three harnesses. The slowest agent averages 3.6 hours per task, and no model approaches saturation: mean scores range from 0.43 to 0.57 out of 1 without feedback, and improve to 0.54–0.67 with feedback. Multi‑turn evaluation uncovers behaviors invisible to single‑shot prompts—subtasks that depend on earlier results receive lower scores, while in‑context feedback helps agents recover some of the loss.\ \ Because deterministic scoring offers no qualitative insight into why an agent fails, we also propose an unsupervised failure‑mode discovery pipeline that describes, embeds, and clusters failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline reveals qualitative shortcomings such as agents applying their own statistical filters instead of the prescribed ones, or generating whole answers from memory.\ \ Review

Original Source: https://arxiv.org/abs/2610.21074

[h] Back to Home