Deep Reinforcement Learning with Python : With PyTorch, TensorFlow and OpenAI Gym

Deep reinforcement learning is a fast-growing discipline that is making a significant impact in fields of autonomous vehicles, robotics, healthcare, finance, and many more. This book covers deep reinforcement learning using deep-q learning and policy gradient models with coding exercise. You'll begin by reviewing the Markov decision processes, Bellman equations, and dynamic programming that form the core concepts and foundation of deep reinforcement learning. Next, you'll study model-free learning followed by function approximation using neural networks and deep learning. This is followed by various deep reinforcement learning algorithms such as deep q-networks, various flavors of actor-critic methods, and other policy-based methods. You'll also look at exploration vs exploitation dilemma, a key consideration in reinforcement learning algorithms, along with Monte Carlo tree search (MCTS), which played a key role in the success of AlphaGo. The final chapters conclude with deep reinforcement learning implementation using popular deep learning frameworks such as TensorFlow and PyTorch. In the end, you'll understand deep reinforcement learning along with deep q networks and policy gradient models implementation with TensorFlow, PyTorch, and Open AI Gym. **What You'll Learn*** Examine deep reinforcement learning * Implement deep learning algorithms using OpenAI’s Gym environment * Code your own game playing agents for Atari using actor-critic algorithms * Apply best practices for model building and algorithm training **Who This Book Is For** Machine learning developers and architects who want to stay ahead of the curve in the field of AI and deep learning. Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: Introduction to Reinforcement Learning Reinforcement Learning Machine Learning Branches Supervised Learning Unsupervised Learning Reinforcement Learning Core Elements Deep Learning with Reinforcement Learning Examples and Case Studies Autonomous Vehicles Robots Recommendation Systems Finance and Trading Healthcare Game Playing Libraries and Environment Setup Alternate Way to Install Local Environment Summary Chapter 2: Markov Decision Processes Definition of Reinforcement Learning Agent and Environment Rewards Markov Processes Markov Chains Markov Reward Processes Markov Decision Processes Policies and Value Functions Bellman Equations Optimality Bellman Equations Types of Solution Approaches with a Mind-Map Summary Chapter 3: Model-Based Algorithms OpenAI Gym Dynamic Programming Policy Evaluation/Prediction Policy Improvement and Iterations Value Iteration Generalized Policy Iteration Asynchronous Backups Summary Chapter 4: Model-Free Approaches Estimation/Prediction with Monte Carlo Bias and Variance of MC Predication Methods Control with Monte Carlo Off-Policy MC Control Temporal Difference Learning Methods Temporal Difference Control On-Policy SARSA Q-Learning: An Off-Policy TD Control Maximization Bias and Double Learning Expected SARSA Control Replay Buffer and Off-Policy Learning Q-Learning for Continuous State Spaces n-Step Returns Eligibility Traces and TD(λ) Relationships Between DP, MC, and TD Summary Chapter 5: Function Approximation Introduction Theory of Approximation Coarse Coding Tile Encoding Challenges in Approximation Incremental Prediction: MC, TD, TD(λ) Incremental Control Semi-gradient N-step SARSA Control Semi-gradient SARSA(λ) Control Convergence in Functional Approximation Gradient Temporal Difference Learning Batch Methods (DQN) Linear Least Squares Method Deep Learning Libraries Summary Chapter 6: Deep Q-Learning Deep Q Networks Atari Game-Playing Agent Using DQN Prioritized Replay Double Q-Learning Dueling DQN NoisyNets DQN Categorical 51-Atom DQN (C51) Quantile Regression DQN Hindsight Experience Replay Summary Chapter 7: Policy Gradient Algorithms Introduction Pros and Cons of Policy-Based Methods Policy Representation Discrete Case Continuous Case Policy Gradient Derivation Objective Function Derivative Update Rule Intuition Behind the Update Rule REINFORCE Algorithm Variance Reduction with Reward to Go Further Variance Reduction with Baselines Actor-Critic Methods Defining Advantage Advantage Actor Critic Implementation of the A2C Algorithm Asynchronous Advantage Actor Critic Trust Region Policy Optimization Algorithm Proximal Policy Optimization Algorithm Summary Chapter 8: Combining Policy Gradient and Q-Learning Trade-Offs in Policy Gradient and Q-Learning General Framework to Combine Policy Gradient with Q-Learning Deep Deterministic Policy Gradient Q-Learning in DDPG (Critic) Policy Learning in DDPG (Actor) Pseudocode and Implementation Gym Environments Used in Code Code Listing Policy Network Actor (PyTorch) Policy Network Actor (TensorFlow) Q-Network Critic Implementation PyTorch TensorFlow Combined Model-Actor Critic Implementation Experience Replay Q-Loss Implementation PyTorch TensorFlow Policy Loss Implementation One Step Update Implementation DDPG: Main Loop Twin Delayed DDPG Target-Policy Smoothing Q-Loss (Critic) Policy Loss (Actor) Delayed Update Pseudocode and Implementation Code Implementation Combined Model-Actor Critic Implementation Q-Loss Implementation Policy-Loss Implementation One-Step Update Implementation TD3 Main Loop Reparameterization Trick Score/Reinforce Way Reparameterization Trick and Pathwise Derivatives Experiment Entropy Explained Soft Actor Critic SAC vs. TD3 Q-Loss with Entropy-Regularization Policy Loss with Reparameterization Trick Pseudocode and Implementation Code Implementation Policy Network-Actor Implementation Q-Network, Combined Model, and Experience Replay Q-Loss and Policy-Loss Implementation One-Step Update and SAC Main Loop Summary Chapter 9: Integrated Planning and Learning Model-Based Reinforcement Learning Planning with a Learned Model Integrating Learning and Planning (Dyna) Dyna Q and Changing Environments Dyna Q+ Expected vs. Sample Updates Exploration vs. Exploitation Multi-arm Bandit Regret: Measure of Quality of Exploration Epsilon Greedy Exploration Upper Confidence Bound Exploration Thompson Sampling Exploration Comparing Different Exploration Strategies Planning at Decision Time and Monte Carlo Tree Search AlphaGo Walk-Through Summary Chapter 10: Further Exploration and Next Steps Model-Based RL: Additional Approaches World Models Imagination-Augmented Agents (I2A) Model-Based RL with Model-Free Fine-Tuning (MBMF) Model-Based Value Expansion (MBVE) Imitation Learning and Inverse Reinforcement Learning Derivative-Free Methods Transfer Learning and Multitask Learning Meta-Learning Popular RL Libraries How to Continue Studying Summary Index Deep reinforcement learning is a fast-growing discipline that is making a significant impact in fields of autonomous vehicles, robotics, healthcare, finance, and many more. This book covers deep reinforcement learning using deep-q learning and policy gradient models with coding exercise. You'll begin by reviewing the Markov decision processes, Bellman equations, and dynamic programming that form the core concepts and foundation of deep reinforcement learning. Next, you'll study model-free learning followed by function approximation using neural networks and deep learning. This is followed by various deep reinforcement learning algorithms such as deep q-networks, various flavors of actor-critic methods, and other policy-based methods. You'll also look at exploration vs exploitation dilemma, a key consideration in reinforcement learning algorithms, along with Monte Carlo tree search (MCTS), which played a key role in the success of AlphaGo. The final chapters conclude with deep reinforcement learning implementation using popular deep learning frameworks such as TensorFlow and PyTorch. In the end, you'll understand deep reinforcement learning along with deep q networks and policy gradient models implementation with TensorFlow, PyTorch, and Open AI Gym. You will: Examine deep reinforcement learning ; Implement deep learning algorithms using OpenAIs Gym environment ; Code your own game playing agents for Atari using actor-critic algorithms ; Apply best practices for model building and algorithm training