MARBLE: Multi-Armed Restless Bandits in Latent Markovian Environment

MARBLE represents a significant advancement in decision-making models by augmenting Restless Multi-Armed Bandits (RMABs) with a latent Markov state, which introduces nonstationary behavior. This innovation is crucial as traditional RMABs often fail in dynamic environments where conditions change over time. The introduction of the Markov-Averaged Indexability (MAI) criterion further enhances the model's robustness, allowing synchronous Q-learning with Whittle Indices to converge to optimal solutions despite unobserved regime switches. Validation on a calibrated simulator-embedded recommender system shows that MARBLE can effectively adapt to shifting latent states, confirming its theoretical foundations and practical applicability in real-world scenarios.