Ensemble Methods for Machine Learning - MEAP Version 6

Chapters 1 to 9 of 10In Ensemble Methods for Machine Learning you'll learn to implement the most important ensemble machine learning methods from scratch. Each chapter contains a new case study, taking you hands-on with a fully functioning ensemble method for medical diagnosis, sentiment analysis, handwriting classification, and more. There's no complex math or theory—each method is taught in a practical and visuals-first manner. Best of all, all code is provided in Jupyter notebooks for your easy experimentation! By the time you're done, you'll know the benefits, limitations, and practical methods of applying ensemble machine learning to real-world data, and be ready to build more explainable ML systems. Ensemble Methods for Machine Learning MEAP V06 Copyright Welcome Brief contents Chapter 1: Ensemble Learning: Hype or Hallelujah? 1.1 Ensemble Methods: The Wisdom of the Crowds 1.2 Why You Should Care About Ensemble Learning 1.3 Fit vs. Complexity in Individual Models 1.3.1 Regression with Decision Trees 1.3.2 Regression with support vector machines 1.4 Our First Ensemble 1.5 Summary Chapter 2: Homogeneous Parallel Ensembles: Bagging and Random Forests 2.1 Parallel Ensembles 2.2 Bagging: Bootstrap Aggregating 2.2.1 Intuition: Resampling and Model Aggregation 2.2.2 Implementing Bagging 2.2.3 Bagging with scikit-learn 2.2.4 Faster Training with Parallelization 2.3 Random Forests 2.3.1 Randomized Decision Trees 2.3.2 Random Forests with scikit-learn 2.3.3 Feature Importances 2.4 More Homogeneous Parallel Ensembles 2.4.1 Pasting 2.4.2 Random Subspaces and Random Patches 2.4.3 ExtraTrees 2.5 Case Study: Breast Cancer Diagnosis 2.5.1 Loading and pre-processing 2.5.2 Bagging, Random Forests and ExtraTrees 2.5.3 Feature importances with Random Forests 2.6 Summary Chapter 3: Heterogeneous Parallel Ensembles: Combining Strong Learners 3.1 Base estimators for heterogeneous ensembles 3.1.1 Fitting base estimators 3.1.2 Individual predictions of base estimators 3.2 Combining predictions by weighting 3.2.1 Majority Vote 3.2.2 Accuracy weighting 3.2.3 Entropy weighting 3.2.4 Dempster-Shafer Combination 3.3 Combining predictions by meta-learning 3.3.1 Stacking 3.3.2 Stacking with cross validation 3.4 Case Study: Sentiment Analysis 3.4.1 Pre-processing 3.4.2 Dimensionality Reduction 3.4.3 Stacking classifiers 3.5 Summary Chapter 4: Sequential Ensembles: Boosting 4.1 Sequential Ensembles of Weak Learners 4.2 AdaBoost: ADAptive BOOSTing 4.2.1 Intuition: Learning with Weighted Examples 4.2.2 Implementing AdaBoost 4.2.3 AdaBoost with scikit-learn 4.3 AdaBoost in Practice 4.3.1 Learning Rate 4.3.2 Early Stopping and Pruning 4.4 Case Study: Handwritten Digit Classification 4.4.1 Dimensionality Reduction with t-SNE 4.4.2 Boosting 4.5 LogitBoost: Boosting with the Logistic Loss 4.6 Summary Chapter 5: Sequential Ensembles: Gradient Boosting 5.1 Gradient Descent for Minimization 5.1.1 Gradient Descent with an Illustrative Example 5.1.2 Gradient Descent over Loss Functions for Training 5.2 Gradient Boosting: Gradient Descent + Boosting 5.2.1 Intuition: Learning with Residuals 5.2.2 Implementing Gradient Boosting 5.2.3 Gradient Boosting with scikit-learn 5.2.4 Histogram-based Gradient Boosting 5.3 LightGBM: A Framework for Gradient Boosting 5.3.1 What Makes LightGBM “Light”? 5.3.2 Gradient Boosting with LightGBM 5.4 LightGBM in Practice 5.4.1 Learning Rate 5.4.2 Early Stopping 5.4.3 Custom Loss Functions 5.5 Case Study: Document Retrieval 5.5.1 The LETOR Data Set 5.5.2 Document Retrieval with LightGBM 5.6 Summary Chapter 6: Sequential Ensembles: Newton Boosting 6.1 Newton’s Method for Minimization 6.1.1 Newton’s Method with an Illustrative Example 6.1.2 Newton Descent over Loss Functions for Training 6.2 Newton Boosting: Newton’s Method + Boosting 6.2.1 Intuition: Learning with Weighted Residuals 6.2.2 Intuition: Learning with Regularized Loss Functions 6.2.3 Implementing Newton Boosting 6.3 XGBoost: A Framework for Newton Boosting 6.3.1 What Makes XGBoost “Extreme”? 6.3.2 Newton Boosting with XGBoost 6.4 XGBoost in Practice 6.4.1 Learning Rate 6.4.2 Early Stopping 6.5 Case Study Redux: Document Retrieval 6.5.1 The LETOR Data Set 6.5.2 Document Retrieval with XGBoost 6.6 Summary Chapter 7: Learning with Continuous and Count Labels 7.1 A Brief Review of Regression 7.1.1 Linear Regression for Continuous Labels 7.1.2 Poisson Regression for Count Labels 7.1.3 Logistic Regression for Classification Labels 7.1.4 Generalized Linear Models 7.1.5 Nonlinear Regression 7.2 Parallel Ensembles for Regression 7.2.1 Random Forest and ExtraTrees 7.2.2 Combining Regression Models 7.2.3 Stacking Regression Models 7.3 Sequential Ensembles for Regression 7.3.1 Loss and Likelihood Functions for Regression 7.3.2 Gradient Boosting with LightGBM and XGBoost 7.4 Case Study: Demand Forecasting 7.4.1 The UCI Bike Rental Data Set 7.4.2 Generalized Linear Models and Stacking 7.4.3 Random Forest and ExtraTrees 7.4.4 XGBoost and LightGBM 7.5 Summary Chapter 8: Learning with Categorical Features 8.1 Encoding Categorical Features 8.1.1 Types of Categorical Features 8.1.2 Ordinal and One-Hot Encoding Ordinal Encoding One-Hot Encoding 8.1.3 Encoding with Target Statistics Greedy Target Encoding Information Leakage and Distribution Shift Hold-out & Leave-One-Out Target Encoding 8.1.4 The category_encoders Package 8.2 CatBoost: A Framework for Ordered Boosting 8.2.1 Ordered Target Statistics and Ordered Boosting Ordered Target Statistics Ordered Boosting 8.2.2 Oblivious Decision Trees 8.2.3 CatBoost in Practice Cross Validation with CatBoost Early Stopping with CatBoost 8.3 Case Study: Income Prediction 8.3.1 The Adult Census Data Set 8.3.2 Creating Preprocessing and Modeling Pipelines 8.3.3 Category Encoding and Ensembling Random Forest LightGBM XGBoost 8.3.4 Ordered Encoding and Boosting with CatBoost 8.4 Encoding High-Cardinality String Features The dirty-cat package 8.5 Summary Chapter 9: Explaining Your Ensembles 9.1 What is Interpretability? 9.1.1 Black-Box vs. Glass-Box Models 9.1.2 Decision Trees (and Decision Rules) Interpreting Decision Trees in Practice Feature Importances 9.1.3 Generalized Linear Models Feature Weights Interpreting GLMs in Practice 9.2 Case Study: Data-driven Marketing 9.2.1 The Bank Telemarketing Data Set 9.2.2 Training Ensembles 9.2.3 Feature Importances in Tree Ensembles 9.3 Black-Box Methods for Global Explainability 9.3.1 Permutation Feature Importance Permutation Feature Importance in Practice 9.3.2 Partial Dependence Plots Partial Dependence Plots in Practice 9.3.3 Global Surrogate Models The Fidelity-Interpretability Tradeoff Training Global Surrogate Models in Practice 9.4 Black-Box Methods for Local Explainability 9.4.1 Local Surrogate Models with LIME The Fidelity-Interpretability Tradeoff Again Sampling Surrogate Examples for Local Explainability LIME in Practice 9.4.2 Local Interpretability with SHAP Understanding Shapley Values Shapley Values as Feature Importance SHapley Additive exPlanations (SHAP) SHAP in Practice 9.5 Glass-Box Ensembles: Training for Interpretability 9.5.1 Explainable Boosting Machines (EBMs) Generalized Additive Models with Feature Interactions Training EBMs 9.5.2 EBMs in Practice 9.6 Summary Ensemble machine learning combines the power of multiple machine learning approaches, working together to deliver models that are highly performant and highly accurate.Inside Ensemble Methods for Machine Learning you will find: Methods for classification, regression, and recommendations Sophisticated off-the-shelf ensemble implementations Random forests, boosting, and gradient boosting Feature engineering and ensemble diversity Interpretability and explainability for ensemble methods Ensemble machine learning trains a diverse group of machine learning models to work together, aggregating their output to deliver richer results than a single model. Now in Ensemble Methods for Machine Learning you'll discover core ensemble methods that have proven records in both data science competitions and real-world applications. Hands-on case studies show you how each algorithm works in production. By the time you're done, you'll know the benefits, limitations, and practical methods of applying ensemble machine learning to real-world data, and be ready to build more explainable ML systems. About the Technology Automatically compare, contrast, and blend the output from multiple models to squeeze the best results from your data. Ensemble machine learning applies a “wisdom of crowds” method that dodges the inaccuracies and limitations of a single model. By basing responses on multiple perspectives, this innovative approach can deliver robust predictions even without massive datasets. About the Book Ensemble Methods for Machine Learning teaches you practical techniques for applying multiple ML approaches simultaneously. Each chapter contains a unique case study that demonstrates a fully functional ensemble method, with examples including medical diagnosis, sentiment analysis, handwriting classification, and more. There's no complex math or theory—you'll learn in a visuals-first manner, with ample code for easy experimentation! What's Inside Bagging, boosting, and gradient boosting Methods for classification, regression, and retrieval Interpretability and explainability for ensemble methods Feature engineering and ensemble diversity About the Reader For Python programmers with machine learning experience. About the Author Gautam Kunapuli has over 15 years of experience in academia and the machine learning industry. Table of Contents PART 1 - THE BASICS OF ENSEMBLES 1 Ensemble methods: Hype or hallelujah? PART 2 - ESSENTIAL ENSEMBLE METHODS 2 Homogeneous parallel ensembles: Bagging and random forests 3 Heterogeneous parallel ensembles: Combining strong learners 4 Sequential ensembles: Adaptive boosting 5 Sequential ensembles: Gradient boosting 6 Sequential ensembles: Newton boosting PART 3 - ENSEMBLES IN THE WILD: ADAPTING ENSEMBLE METHODS TO YOUR DATA 7 Learning with continuous and count labels 8 Learning with categorical features 9 Explaining your ensembles Ensemble machine learning combines the power of multiple machine learning approaches, working together to deliver models that are highly performant and highly accurate. Inside Ensemble Methods for Machine Learning you will Ensemble machine learning trains a diverse group of machine learning models to work together, aggregating their output to deliver richer results than a single model. Now in Ensemble Methods for Machine Learning youll discover core ensemble methods that have proven records in both data science competitions and real-world applications. Hands-on case studies show you how each algorithm works in production. By the time you're done, you'll know the benefits, limitations, and practical methods of applying ensemble machine learning to real-world data, and be ready to build more explainable ML systems. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the Technology Automatically compare, contrast, and blend the output from multiple models to squeeze the best results from your data. Ensemble machine learning applies a wisdom of crowds method that dodges the inaccuracies and limitations of a single model. By basing responses on multiple perspectives, this innovative approach can deliver robust predictions even without massive datasets. About the Book Ensemble Methods for Machine Learning teaches you practical techniques for applying multiple ML approaches simultaneously. Each chapter contains a unique case study that demonstrates a fully functional ensemble method, with examples including medical diagnosis, sentiment analysis, handwriting classification, and more. Theres no complex math or theoryyoull learn in a visuals-first manner, with ample code for easy experimentation! Whats Inside About the Reader For Python programmers with machine learning experience. About the Author Gautam Kunapuli has over 15 years of experience in academia and the machine learning industry. Table of Contents PART 1 - THE BASICS OF ENSEMBLES 1 Ensemble Hype or hallelujah? PART 2 - ESSENTIAL ENSEMBLE METHODS 2 Homogeneous parallel Bagging and random forests 3 Heterogeneous parallel Combining strong learners 4 Sequential Adaptive boosting 5 Sequential Gradient boosting 6 Sequential Newton boosting PART 3 - ENSEMBLES IN THE ADAPTING ENSEMBLE METHODS TO YOUR DATA 7 Learning with continuous and count labels 8 Learning with categorical features 9 Explaining your ensembles