[CS.AI] Similarity as Reward Alignment in Robust Preference-based RL

Abstract

Preference-based Reinforcement Learning (PbRL) involves various approaches to align models with human intent, reducing the burden of reward engineering. However, most previous PbRL work has not explored robustness against labeler errors, which are common with non-expert labelers or those under time constraints. We introduce Similarity as Reward Alignment (SARA), a simple contrastive framework that is resilient to noisy labels and adaptable to diverse feedback formats. SARA learns a latent representation of preferred samples and computes rewards based on similarities to the learned latent.

On preference data with varying realistic noise rates, we demonstrate competitive and more stable performance on continuous control offline RL benchmarks, with statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.05).

Code Implementation

Here is a core implementation of SARA:

// Core SARA implementation example
void saraAlgorithm(vector<Sample> preferences) {
    // Learn latent representation
    LatentRepresentation latent = learnLatent(preferences);
    // Compute rewards
    for (Sample sample : preferences) {
        float reward = computeSimilarity(latent, sample);
        assignReward(sample, reward);
    }
}

Blogger's Review: The SARA framework exhibits robustness in the presence of label noise, making it particularly suitable for scenarios where human feedback is imperfect. By centering similarity as the core of rewards, SARA enhances model stability and offers new insights into the application of preference-based reinforcement learning. Its simple yet effective design performs well across various feedback formats, warranting further exploration and application.