Fundamentals of Pattern Recognition and Machine Learning

This book is a concise but thorough introduction to the tools commonly used in pattern recognition and machine learning, including classification, dimensionality reduction, regression, and clustering, as well as recent popular topics such as deep neural networks and Gaussian process regression. The Second Edition is thoroughly revised, featuring a new chapter on the emerging topic of physics-informed machine learning and additional material on deep neural networks. Combining theory and practice, this book is suitable for the graduate or advanced undergraduate level classroom and self-study. It fills the need of a mathematically-rigorous text that is relevant to the practitioner as well, with datasets from applications in bioinformatics and materials informatics used throughout to illustrate the theory. These datasets are available from the book website to be used in end-of-chapter coding assignments based on python and Keras/Tensorflow. All plots in the text were generated using python scripts and jupyter notebooks, which can be downloaded from the book website. Preface to the Second Edition Preface to the First Edition Contents Chapter 1 Introduction 1.1 Pattern Recognition and Machine Learning 1.2 Basic Mathematical Setting 1.3 Prediction 1.4 Prediction Error 1.5 Supervised vs. Unsupervised Learning 1.6 Complexity Trade-Offs 1.7 The Design Cycle 1.8 Application Examples 1.8.1 Bioinformatics 1.8.2 Materials Informatics 1.9 Bibliographical Notes Chapter 2 Optimal Classification 2.1 Classification without Features 2.2 Classification with Features 2.3 The Bayes Classifier 2.4 The Bayes Error 2.5 Gaussian Model 2.5.1 Homoskedastic Case 2.5.2 Heteroskedastic Case 2.6 Additional Topics 2.6.1 Minimax Classification 2.6.2 F-errors 2.6.3 Bayes Decision Theory 2.6.4 Rigorous Formulation of the Classification Problem 2.7 Bibliographical Notes 2.8 Exercises 2.9 Python Assignments Chapter 3 Sample-Based Classification 3.1 Classification Rules 3.2 Classification Error Rates 3.3 Consistency 3.4 No-Free-Lunch Theorems 3.5 Additional Topics 3.5.1 Ensemble Classification 3.5.2 Mixture Sampling vs. Separate Sampling 3.6 Bibliographical Notes 3.7 Exercises 3.8 Python Assignments Chapter 4 Parametric Classification 4.1 Parametric Plug-in Rules 4.2 Gaussian Discriminant Analysis 4.2.1 Linear Discriminant Analysis 4.2.2 Quadratic Discriminant Analysis 4.3 Logistic Classification 4.4 Additional Topics 4.4.1 Regularized Discriminant Analysis 4.4.2 Consistency of Parametric Rules 4.4.3 Bayesian Parametric Rules 4.5 Bibliographical Notes 4.6 Exercises 4.7 Python Assignments Chapter 5 Nonparametric Classification 5.1 Nonparametric Plug-in Rules 5.2 Histogram Classification 5.3 Nearest-Neighbor Classification 5.4 Kernel Classification 5.5 Cover-Hart Theorem 5.6 Stone’s Theorem 5.7 Bibliographical Notes 5.8 Exercises 5.9 Python Assignments Chapter 6 Function-Approximation Classification 6.1 Support Vector Machines 6.1.1 Linear SVMs for Separable Data 6.1.2 General Linear SVMs 6.1.3 Nonlinear SVMs 6.2 Neural Networks 6.2.1 Fully-Connected Neural Networks 6.2.2 Neural Network Classifiers 6.2.3 Loss Functions 6.2.4 Backpropagation Algorithm 6.2.5 Training Deep Neural Networks 6.2.6 Convolutional Neural Networks 6.2.7 Universal Approximation Property of Neural Networks 6.2.8 Universal Consistency Theorems 6.3 Decision Trees 6.4 Rank-Based Classifiers 6.5 Bibliographical Notes 6.6 Exercises 6.7 Python Assignments Chapter 7 Error Estimation for Classification 7.1 Error Estimation Rules 7.2 Error Estimation Performance 7.2.1 Deviation Distribution 7.2.2 Bias, Variance, RMS, and Tail Probabilities 7.2.3 Consistency 7.3 Test-Set Error Estimation 7.4 Resubstitution 7.5 Cross-Validation 7.6 Bootstrap 7.7 Bolstered Error Estimation 7.8 Additional Topics 7.8.1 Convex Error Estimators 7.8.2 Smoothed Error Estimators 7.8.3 Bayesian Error Estimation 7.9 Bibliographical Notes 7.10 Exercises 7.11 Python Assignments Chapter 8 Model Selection for Classification 8.1 Classification Complexity 8.2 Vapnik-Chervonenkis Theory *8.2.1 Finite Model Selection 8.2.2 Shatter Coefficients and VC Dimension 8.2.3 VC Parameters of a Few Classification Rules 8.2.4 Vapnik-Chervonenkis Theorem 8.2.5 No-Free-Lunch Theorems 8.3 Model Selection Methods 8.3.1 Validation Error Minimization 8.3.2 Training Error Minimization 8.3.3 Structural Risk Minimization 8.4 Bibliographical Notes 8.5 Exercises Chapter 9 Dimensionality Reduction 9.1 Feature Extraction for Classification 9.2 Feature Selection 9.2.1 Exhaustive Search 9.2.2 Univariate Greedy Search 9.2.3 Multivariate Greedy Search 9.2.4 Feature Selection and Classification Complexity 9.2.5 Feature Selection and Error Estimation 9.3 Principal Component Analysis (PCA) 9.4 Multidimensional Scaling (MDS) 9.5 Factor Analysis 9.6 Bibliographical Notes 9.7 Exercises 9.8 Python Assignments Chapter 10 Clustering 10.1 K-Means Algorithm 10.2 Gaussian Mixture Modeling 10.2.1 Expectation-Maximization Approach 10.2.2 Relationship to 10.3 Hierarchical Clustering 10.4 Self-Organizing Maps (SOM) 10.5 Bibliographical Notes 10.6 Exercises 10.7 Python Assignments Chapter 11 Regression 11.1 Optimal Regression 11.2 Sample-Based Regression 11.3 Parametric Regression 11.3.1 Linear Regression 11.3.2 Gauss-Markov Theorem 11.3.3 Penalized Least Squares 11.4 Nonparametric Regression 11.4.1 Kernel Regression 11.4.2 Gaussian Process Regression 11.5 Function-Approximation Regression 11.6 Error Estimation 11.7 Variable Selection 11.7.1 Wrapper Search 11.7.2 Statistical Testing 11.7.3 LASSO and Elastic Net 11.8 Model Selection 11.9 Bibliographical Notes 11.10 Exercises 11.11 Python Assignments Chapter 12 Physics-Informed Machine Learning 12.1 The Relationship between Data and Theory 12.2 Partial Differential Equation Models 12.3 Physics-Informed Neural Networks 12.3.1 Forward Problem 12.3.2 Inverse Problem 12.3.3 Hard Constraints 12.4 Physics-Informed Gaussian Processes 12.4.1 Forward Problem 12.4.2 Inverse Problem 12.4.3 Hard Constraints 12.5 Bibliographical Notes 12.6 Exercises 12.7 Python Assignments Appendix A1 Probability Theory A1.1 Sample Space and Events A1.2 Probability Measure A1.3 Conditional Probability and Independence A1.4 Random Variables A1.5 Joint and Conditional Distributions A1.6 Expectation A1.7 Vector Random Variables A1.8 Convergence of Random Sequences A1.9 Asymptotic Theorems A2 Basic Matrix Theory A3 Basic Lagrange-Multiplier Optimization A4 Proof of the Cover-Hart Theorem A5 Proof of Stone’s Theorem A6 Proof of the Vapnik-Chervonenkis Theorem A7 Proof of Convergence of the EM Algorithm A8 Data Sets Used in the Book A8.1 Synthetic Data A8.2 Dengue Fever Prognosis Data Set A8.3 Breast Cancer Prognosis Data Set A8.4 Stacking Fault Energy Data Set A8.5 Soft Magnetic Alloy Data Set A8.6 Ultrahigh Carbon Steel Data Set List of Symbols Bibliography Index