Reliable reinforcement learning (RL) agents must maintain operational integrity amidst sensor malfunctions, dynamic disturbances, and slow environmental shifts. Detecting out-of-distribution (OOD) conditions is crucial for determining when an agent's observations, transitions, or trajectory dynamics deviate from the assumptions underpinning its policy training. Current OOD detection benchmarks typically evaluate image classifiers or static low-dimensional datasets, failing to account for the complex, action-dependent temporal structure inherent in RL trajectories.
To address this gap, we present OOD-RL-Bench, a comprehensive and extensible framework designed to evaluate OOD detectors against categories of anomalies injected into RL trajectories. Detectors and anomaly injectors are integrated through shared interfaces and configuration, allowing new scoring methods and perturbation families to be evaluated without modifying the core benchmark loop.
We evaluate the utility of the framework using a Deep Q-Network policy within the LunarLander-v3 environment. We assess the performance of each detector across a suite of anomaly types using matched-time AUROC, matched-time AUPRC, matched-time false-positive rate, detection delay, and segmented-onset metrics.
Our analysis reveals significant performance variance across anomaly types: observation perturbations and regime switches are identified with high accuracy by several methods, while observation delay and action-conditioned dynamics remain difficult even when post-onset anomaly scores are compared against clean scores from the same timesteps. We make the framework, trained policy checkpoint, and complete results publicly available as a reproducible artefact.
Blogger's Review: OOD-RL-Bench provides a crucial tool for the reinforcement learning field, enabling researchers to better assess agent performance under anomalous conditions. This framework not only fills the gap in existing detection benchmarks but also supports testing of various anomaly types through its flexible interface design, offering high practicality and foresight. Its public availability of trained models and results also promotes transparency and reproducibility in research.