Transfer Learning for Multiagent Reinforcement Learning Systems (Synthesis Lectures on Artificial Intelligence and Machine Learning)

Learning to solve sequential decision-making tasks is difficult. Humans take years exploring the environment essentially in a random way until they are able to reason, solve difficult tasks, and collaborate with other humans towards a common goal. Artificial Intelligent agents are like humans in this aspect. Reinforcement Learning (RL) is a well-known technique to train autonomous agents through interactions with the environment. Unfortunately, the learning process has a high sample complexity to infer an effective actuation policy, especially when multiple agents are simultaneously actuating in the environment. However, previous knowledge can be leveraged to accelerate learning and enable solving harder tasks. In the same way humans build skills and reuse them by relating different tasks, RL agents might reuse knowledge from previously solved tasks and from the exchange of knowledge with other agents in the environment. In fact, virtually all of the most challenging tasks currently solved by RL rely on embedded knowledge reuse techniques, such as Imitation Learning, Learning from Demonstration, and Curriculum Learning. This book surveys the literature on knowledge reuse in multiagent RL. The authors define a unifying taxonomy of state-of-the-art solutions for reusing knowledge, providing a comprehensive discussion of recent progress in the area. In this book, readers will find a comprehensive discussion of the many ways in which knowledge can be reused in multiagent sequential decision-making tasks, as well as in which scenarios each of the approaches is more efficient. The authors also provide their view of the current low-hanging fruit developments of the area, as well as the still-open big questions that could result in breakthrough developments. Finally, the book provides resources to researchers who intend to join this area or leverage those techniques, including a list of conferences, journals, and implementation tools. This book will be useful for a wide audience; and will hopefully promote new dialogues across communities and novel developments in the area. Cover Copyright Page Title Page Contents Preface Acknowledgments Introduction Contribution and Scope Overview Background The Basics of Reinforcement Learning Deep Reinforcement Learning Multiagent Reinforcement Learning Transfer Learning Taxonomy Nomenclature Learning Algorithm (LA) Source Task Selection (ST) Mapping Autonomy (MA) Transferred Knowledge (TK) Allowed Differences (AD) Intra-Agent Transfer Methods Adapting to Other Agents Sparse Interaction Algorithms Relational Descriptions Source Task Selection Biases and Heuristics Curriculum Learning Deep Reinforcement Learning Transfer Others Inter-Agent Transfer Methods Action Advising Human-Focused Transfer Learning from Demonstrations Imitation Reward Shaping and Heuristics Inverse Reinforcement Learning Curriculum Learning Transfer in Deep Reinforcement Learning Scaling Learning to Complex Problems Experiment Domains and Applications Gridworld and Variations Simulated Robot Soccer Video Games Robotics Smart Grid Autonomous Driving Simulation Current Challenges Curriculum Learning in Multiagent Systems Benchmarks for Transfer in Multiagent Systems Knowledge Reuse for Ad Hoc Teams End-to-End Multiagent Transfer Frameworks Transfer for Deep Multiagent Reinforcement Learning Integrated Inter-Agent and Intra-Agent Transfer Human-Focused Multiagent Transfer Learning Cloud Knowledge Bases Mean-Field Knowledge Reuse Security Inverse Reinforcement Learning for Enforcing Cooperation Adversary-Aware Learning Approaches Resources Conferences Journals Libraries Conclusion Bibliography Authors' Biographies