Multi-agent Markov Entanglement

The paper titled 'Multi-agent Markov Entanglement' discusses the concept of value decomposition in multi-agent dynamic programming and reinforcement learning (RL). It highlights that the value function of a global state can be approximated as the sum of local functions. This method has historical roots in index policies related to restless multi-armed bandit problems and is widely applied in modern RL systems. However, the theoretical basis for the effectiveness of this decomposition has not been thoroughly examined. The authors reveal that a multi-agent Markov decision process (MDP) allows for value decomposition only if its transition matrix is not 'entangled,' a term borrowed from quantum physics. The study also suggests that Markov entanglement can help bound decomposition errors in multi-agent MDPs, with certain index policies exhibiting a sublinear scale of decomposition error for systems with multiple agents.