Deep Reinforcement Learning in Action

Summary Humans learn best from feedback—we are encouraged to take actions that lead to positive results while deterred by decisions with negative consequences. This reinforcement process can be applied to computer programs allowing them to solve more complex problems that classical programming cannot. Deep Reinforcement Learning in Action teaches you the fundamental concepts and terminology of deep reinforcement learning, along with the practical skills and techniques you'll need to implement it into your own projects. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology Deep reinforcement learning AI systems rapidly adapt to new environments, a vast improvement over standard neural networks. A DRL agent learns like people do, taking in raw data such as sensor input and refining its responses and predictions through trial and error. About the book Deep Reinforcement Learning in Action teaches you how to program AI agents that adapt and improve based on direct feedback from their environment. In this example-rich tutorial, you'll master foundational and advanced DRL techniques by taking on interesting challenges like navigating a maze and playing video games. Along the way, you'll work with core algorithms, including deep Q-networks and policy gradients, along with industry-standard tools like PyTorch and OpenAI Gym. What's inside Building and training DRL networks The most popular DRL algorithms for learning and problem solving Evolutionary algorithms for curiosity and multi-agent learning All examples available as Jupyter Notebooks About the reader For readers with intermediate skills in Python and deep learning. About the author Alexander Zai is a machine learning engineer at Amazon AI. Brandon Brown is a machine learning and data analysis blogger. Table of Contents PART 1 - FOUNDATIONS 1. What is reinforcement learning? 2. Modeling reinforcement learning problems: Markov decision processes 3. Predicting the best states and actions: Deep Q-networks 4. Learning to pick the best policy: Policy gradient methods 5. Tackling more complex problems with actor-critic methods PART 2 - ABOVE AND BEYOND 6. Alternative optimization methods: Evolutionary algorithms 7. Distributional DQN: Getting the full story 8.Curiosity-driven exploration 9. Multi-agent reinforcement learning 10. Interpretable reinforcement learning: Attention and relational models 11. In conclusion: A review and roadmap brief contents 7 contents 9 preface 15 acknowledgments 17 about this book 18 Who should read this book 18 How this book is organized: A roadmap 19 About the code 20 liveBook discussion forum 20 about the authors 21 about the cover illustration 22 Part 1—Foundations 23 1 What is reinforcement learning? 25 1.1 The “deep” in deep reinforcement learning 26 1.2 Reinforcement learning 28 1.3 Dynamic programming versus Monte Carlo 31 1.4 The reinforcement learning framework 32 1.5 What can I do with reinforcement learning? 36 1.6 Why deep reinforcement learning? 38 1.7 Our didactic tool: String diagrams 40 1.8 What’s next? 42 Summary 43 2 Modeling reinforcement learning problems: Markov decision processes 45 2.1 String diagrams and our teaching methods 45 2.2 Solving the multi-arm bandit 50 2.2.1 Exploration and exploitation 51 2.2.2 Epsilon-greedy strategy 52 2.2.3 Softmax selection policy 57 2.3 Applying bandits to optimize ad placements 59 2.3.1 Contextual bandits 60 2.3.2 States, actions, rewards 61 2.4 Building networks with PyTorch 62 2.4.1 Automatic differentiation 62 2.4.2 Building Models 63 2.5 Solving contextual bandits 64 2.6 The Markov property 69 2.7 Predicting future rewards: Value and policy functions 71 2.7.1 Policy functions 72 2.7.2 Optimal policy 73 2.7.3 Value functions 73 Summary 75 3 Predicting the best states and actions: Deep Q-networks 76 3.1 The Q function 77 3.2 Navigating with Q-learning 78 3.2.1 What is Q-learning? 78 3.2.2 Tackling Gridworld 79 3.2.3 Hyperparameters 81 3.2.4 Discount factor 82 3.2.5 Building the network 83 3.2.6 Introducing the Gridworld game engine 85 3.2.7 A neural network as the Q function 87 3.3 Preventing catastrophic forgetting: Experience replay 97 3.3.1 Catastrophic forgetting 97 3.3.2 Experience replay 98 3.4 Improving stability with a target network 102 3.4.1 Learning instability 103 3.5 Review 108 Summary 110 4 Learning to pick the best policy: Policy gradient methods 112 4.1 Policy function using neural networks 113 4.1.1 Neural network as the policy function 113 4.1.2 Stochastic policy gradient 114 4.1.3 Exploration 116 4.2 Reinforcing good actions: The policy gradient algorithm 117 4.2.1 Defining an objective 117 4.2.2 Action reinforcement 119 4.2.3 Log probability 120 4.2.4 Credit assignment 121 4.3 Working with OpenAI Gym 122 4.3.1 CartPole 124 4.3.2 The OpenAI Gym API 125 4.4 The REINFORCE algorithm 125 4.4.1 Creating the policy network 126 4.4.2 Having the agent interact with the environment 126 4.4.3 Training the model 127 4.4.4 The full training loop 129 4.4.5 Chapter conclusion 130 Summary 131 5 Tackling more complex problems with actor-critic methods 133 5.1 Combining the value and policy function 135 5.2 Distributed training 140 5.3 Advantage actor-critic 145 5.4 N-step actor-critic 154 Summary 159 Part 2—Above and beyond 161 6 Alternative optimization methods: Evolutionary algorithms 163 6.1 A different approach to reinforcement learning 164 6.2 Reinforcement learning with evolution strategies 165 6.2.1 Evolution in theory 165 6.2.2 Evolution in practice 169 6.3 A genetic algorithm for CartPole 173 6.4 Pros and cons of evolutionary algorithms 180 6.4.1 Evolutionary algorithms explore more 180 6.4.2 Evolutionary algorithms are incredibly sample intensive 180 6.4.3 Simulators 181 6.5 Evolutionary algorithms as a scalable alternative 181 6.5.1 Scaling evolutionary algorithms 182 6.5.2 Parallel vs. serial processing 183 6.5.3 Scaling efficiency 184 6.5.4 Communicating between nodes 185 6.5.5 Scaling linearly 187 6.5.6 Scaling gradient-based approaches 187 Summary 188 7 Distributional DQN: Getting the full story 189 7.1 What’s wrong with Q-learning? 190 7.2 Probability and statistics revisited 195 7.2.1 Priors and posteriors 197 7.2.2 Expectation and variance 198 7.3 The Bellman equation 202 7.3.1 The distributional Bellman equation 202 7.4 Distributional Q-learning 203 7.4.1 Representing a probability distribution in Python 204 7.4.2 Implementing the Dist-DQN 213 7.5 Comparing probability distributions 215 7.6 Dist-DQN on simulated data 220 7.7 Using distributional Q-learning to play Freeway 225 Summary 231 8 Curiosity-driven exploration 232 8.1 Tackling sparse rewards with predictive coding 234 8.2 Inverse dynamics prediction 237 8.3 Setting up Super Mario Bros. 240 8.4 Preprocessing and the Q-network 243 8.5 Setting up the Q-network and policy function 245 8.6 Intrinsic curiosity module 248 8.7 Alternative intrinsic reward mechanisms 261 Summary 263 9 Multi-agent reinforcement learning 265 9.1 From one to many agents 266 9.2 Neighborhood Q-learning 270 9.3 The 1D Ising model 274 9.4 Mean field Q-learning and the 2D Ising model 283 9.5 Mixed cooperative-competitive games 293 Summary 303 10 Interpretable reinforcement learning: Attention and relational models 305 10.1 Machine learning interpretability with attention and relational biases 306 10.1.1 Invariance and equivariance 308 10.2 Relational reasoning with attention 309 10.2.1 Attention models 310 10.2.2 Relational reasoning 312 10.2.3 Self-attention models 317 10.3 Implementing self-attention for MNIST 320 10.3.1 Transformed MNIST 320 10.3.2 The relational module 321 10.3.3 Tensor contractions and Einstein notation 325 10.3.4 Training the relational module 328 10.4 Multi-head attention and relational DQN 332 10.5 Double Q-learning 339 10.6 Training and attention visualization 341 10.6.1 Maximum entropy learning 345 10.6.2 Curriculum learning 345 10.6.3 Visualizing attention weights 345 Summary 349 11 In conclusion: A review and roadmap 351 11.1 What did we learn? 351 11.2 The uncharted topics in deep reinforcement learning 353 11.2.1 Prioritized experience replay 353 11.2.2 Proximal policy optimization (PPO) 354 11.2.3 Hierarchical reinforcement learning and the options framework 355 11.2.4 Model-based planning 355 11.2.5 Monte Carlo tree search (MCTS) 356 11.3 The end 357 Appendix—Mathematics, deep learning, PyTorch 358 A.1 Linear algebra 358 A.2 Calculus 360 A.3 Deep learning 365 A.4 PyTorch 366 Reference list 370 index 373 Symbols 373 Numerics 373 A 373 B 374 C 374 D 374 E 375 F 376 G 376 H 376 I 376 J 376 K 376 L 376 M 377 N 378 O 378 P 378 Q 379 R 379 S 380 T 380 U 381 V 381 W 381 X 381 Y 381 Humans learn best from feedback-we are encouraged to take actions that lead to positive results while deterred by decisions with negative consequences. This reinforcement process can be applied to computer programs allowing them to solve more complex problems that classical programming cannot. Deep Reinforcement Learning in Action teaches you the fundamental concepts and terminology of deep reinforcement learning, along with the practical skills and techniques you'll need to implement it into your own projects. Key features * Structuring problems as Markov Decision Processes * Popular algorithms such Deep Q-Networks, Policy Gradient method and Evolutionary Algorithms and the intuitions that drive them * Applying reinforcement learning algorithms to real-world problems Audience You'll need intermediate Python skills and a basic understanding of deep learning. About the technology Deep reinforcement learning is a form of machine learning in which AI agents learn optimal behavior from their own raw sensory input. The system perceives the environment, interprets the results of its past decisions, and uses this information to optimize its behavior for maximum long-term return. Deep reinforcement learning famously contributed to the success of AlphaGo but that's not all it can do! Alexander Zai is a Machine Learning Engineer at Amazon AI working on MXNet that powers a suite of AWS machine learning products. Brandon Brown is a Machine Learning and Data Analysis blogger at outlace.com committed to providing clear teaching on difficult topics for newcomers.