Multi-Agent Reinforcement Learning - Christianos, Filippos

. .

Résumé :
Preface xi
Summary of Notation xv
List of Figures xvii
1 Introduction 1
1.1 Multi-Agent Systems 2
1.2 Multi-Agent Reinforcement Learning 6
1.3 Application Examples 8
1.3.1 Multi-Robot Warehouse Management 8
1.3.2 Competitive Play in Board Games and Video Games 10
1.3.3 Autonomous Driving 11
1.3.4 Automated Trading in Electronic Markets 11
1.4 Challenges of MARL 12
1.5 Agendas of MARL 13
1.6 Book Contents and Structure 15
I FOUNDATIONS OF MULTI-AGENT REINFORCEMENT LEARNING 17
2 Reinforcement Learning 19
2.1 General Definition 20
2.2 Markov Decision Processes 22
2.3 Expected Discounted Returns and Optimal Policies 24
2.4 Value Functions and Bellman Equation 26
2.5 Dynamic Programming 29
2.6 Temporal-Difference Learning 32
2.7 Evaluation with Learning Curves 36
2.8 Equivalence of R(s, a, s') and R(s, a) 39
2.9 Summary 40
3 Games: Models of Multi-Agent Interaction 43
3.1 Normal-Form Games 44
3.2 Repeated Normal-Form Games 46
3.3 Stochastic Games 47
3.4 Partially Observable Stochastic Games 49
3.4.1 Belief States and Filtering 53
3.5 Modelling Communication 55
3.6 Knowledge Assumptions in Games 56
3.7 Dictionary: Reinforcement Learning Game Theory 58
3.8 Summary 58
4 Solution Concepts for Games 61
4.1 Joint Policy and Expected Return 62
4.2 Best Response 65
4.3 Minimax 65
4.3.1 Minimax Solution via Linear Programming 67
4.4 Nash Equilibrium 68
4.5 -Nash Equilibrium 70
4.6 (Coarse) Correlated Equilibrium 71
4.6.1 Correlated Equilibrium via Linear Programming 74
4.7 Conceptual Limitations of Equilibrium Solutions 75
4.8 Pareto Optimality 76
4.9 Social Welfare and Fairness 78
4.10 No-Regret 81
4.11 The Complexity of Computing Equilibria 83
4.11.1 PPAD Complexity Class 84
4.11.2 Computing -Nash Equilibrium is PPAD-Complete 86
4.12 Summary 87
5 Multi-Agent Reinforcement Learning in Games: First Steps and Challenges 89
5.1 General Learning Process 90
5.2 Convergence Types 92
5.3 Single-Agent RL Reductions 95
5.3.1 Central Learning 95
5.3.2 Independent Learning 97
5.3.3 Example: Level-Based Foraging 99
5.4 Challenges of MARL 101
5.4.1 Non-Stationarity 102
5.4.2 Equilibrium Selection 104
5.4.3 Multi-Agent Credit Assignment 106
5.4.4 Scaling to Many Agents 108
5.5 What Algorithms Do Agents Use? 109
5.5.1 Self-Play 109
5.5.2 Mixed-Play 111
5.6 Summary 111
6 Multi-Agent Reinforcement Learning: Foundational Algorithms 115
6.1 Dynamic Programming for Games: Value Iteration 116
6.2 Temporal-Difference Learning for Games: Joint Action Learning 118
6.2.1 Minimax Q-Learning 121
6.2.2 Nash Q-Learning 123
6.2.3 Correlated Q-Learning 124
6.2.4 Limitations of Joint Action Learning 125
6.3 Agent Modelling 127
6.3.1 Fictitious Play 128
6.3.2 Joint Action Learning with Agent Modelling 131
6.3.3 Bayesian Learning and Value of Information 134
6.4 Policy-Based Learning 140
6.4.1 Gradient Ascent in Expected Reward 141
6.4.2 Learning Dynamics of Infinitesimal Gradient Ascent 142
6.4.3 Win or Learn Fast 145
6.4.4 Win or Learn Fast with Policy Hill Climbing 147
6.4.5 Generalised Infinitesimal Gradient Ascent 149
6.5 No-Regret Learning 151
6.5.1 Unconditional and Conditional Regret Matching 151
6.5.2 Convergence of Regret Matching 153
6.6 Summary 156
II MULTI-AGENT DEEP REINFORCEMENT LEARNING: ALGORITHMS AND PRACTICE 159
7 Deep Learning 161
7.1 Function Approximation for Reinforcement Learning 161
7.2 Linear Function Approximation 163
7.3 Feedforward Neural Networks 165
7.3.1 Neural Unit 166
7....

Biographie:
Stefano V. Albrecht is Associate Professor in the School of Informatics at the University of Edinburgh, where he leads the Autonomous Agents Research Group. His research focuses on the development of machine learning algorithms for autonomous systems control and decision making, with a particular focus on deep reinforcement learning and multi-agent interaction.

Filippos Christianos is a research scientist in multi-agent deep reinforcement learning focusing on how MARL algorithms can be used efficiently and the author of multiple popular MARL-focused code libraries. ...

Sommaire:
Exploiting Spatial Relationships in Data 175
7.5.2 Learning from Sequences with Memory 178
7.6 Summary 180
8 Deep Reinforcement Learning 183
8.1 Deep Value Function Approximation 184
8.1.1 Deep Q-Learning &ndash...