Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options

The study of preference-based reinforcement learning (PbRL) has gained traction due to its success in aligning large language models (LLMs). However, most existing research has been limited to pairwise comparisons, which can hinder performance when feedback length increases. Recent works have attempted to explore multiple comparisons, but they often fail to improve performance guarantees. To overcome these challenges, the M-AUPO algorithm was proposed, utilizing the Plackett-Luce model to maximize average uncertainty in action selection. This approach demonstrates a suboptimality gap that suggests larger subsets can lead to improved performance, thus enhancing the efficiency of PbRL. The implications of this research are profound, as it not only addresses existing limitations but also opens new avenues for optimizing reinforcement learning frameworks, particularly in the context of LLMs.