Elements of Data Science, Machine Learning, and Artificial Intelligence Using R

The textbook provides students with tools they need to analyze complex data using methods from data science, machine learning and artificial intelligence. The authors include both the presentation of methods along with applications using the programming language R, which is the gold standard for analyzing data. The authors cover all three main components of data science: computer science; mathematics and statistics; and domain knowledge. The book presents methods and implementations in R side-by-side, allowing the immediate practical application of the learning concepts. Furthermore, this teaches computational thinking in a natural way. The book includes exercises, case studies, Q&A and examples. Preface Contents 1 Introduction to Learning from Data 1.1 What Is Data Science? 1.2 Converting Data into Knowledge 1.2.1 Big Aims: Big Questions 1.2.2 Generating Insights by Visualization 1.3 Structure of the Book 1.3.1 Part I 1.3.2 Part II 1.3.3 Part III 1.4 Our Motivation for Writing This Book 1.5 How to Use This Book 1.6 Summary Part I General Topics 2 General Prediction Models 2.1 Introduction 2.2 Categorization of Methods 2.2.1 Properties of the Data 2.2.2 Properties of the Optimization Algorithm 2.2.3 Properties of the Model 2.2.4 Summary 2.3 Overview of Prediction Models 2.4 Causal Model versus Predictive Model 2.5 Explainable AI 2.6 Fundamental Statistical Characteristics of Prediction Models 2.6.1 Example 2.7 Summary 2.8 Exercises 3 General Error Measures 3.1 Introduction 3.2 Motivation 3.3 Fundamental Error Measures 3.4 Error Measures 3.4.1 True-Positive Rate and True-Negative Rate 3.4.2 Positive Predictive Value and Negative Predictive Value 3.4.3 Accuracy 3.4.4 F-Score 3.4.5 False Discovery Rate and False Omission Rate 3.4.6 False-Negative Rate and False-Positive Rate 3.4.7 Matthews Correlation Coefficient 3.4.8 Cohen's Kappa 3.4.9 Normalized Mutual Information 3.4.10 Area Under the Receiver Operator Characteristic Curve 3.5 Evaluation of Outcome 3.5.1 Evaluation of an Individual Method 3.5.2 Comparing Multiple Binary Decision-Making Methods 3.6 Summary 3.7 Exercises 4 Resampling Methods 4.1 Introduction 4.2 Resampling Methods for Error Estimation 4.2.1 Holdout Set 4.2.2 Leave-One-Out CV 4.2.3 K-Fold Cross-Validation 4.3 Extended Resampling Methods for Error Estimation 4.3.1 Repeated Holdout Set 4.3.2 Repeated K-Fold CV 4.3.3 Stratified K-Fold CV 4.4 Bootstrap 4.4.1 Resampling With versus Resampling Without Replacement 4.5 Subsampling 4.6 Different Types of Prediction Data Sets 4.7 Sampling from a Distribution 4.8 Standard Error 4.9 Summary 4.10 Exercises 5 Data 5.1 Introduction 5.2 Data Types 5.2.1 Genomic Data 5.2.2 Network Data 5.2.3 Text Data 5.2.4 Time-to-Event Data 5.2.5 Business Data 5.3 Summary Part II Core Methods 6 Statistical Inference 6.1 Exploratory Data Analysis and Descriptive Statistics 6.1.1 Data Structure 6.1.2 Data Preprocessing 6.1.3 Summary Statistics and Presentation of Information 6.1.4 Measures of Location 6.1.4.1 Sample Mean 6.1.4.2 Trimmed Sample Mean 6.1.4.3 Sample Median 6.1.4.4 Quartile 6.1.4.5 Percentile 6.1.4.6 Mode 6.1.4.7 Proportion 6.1.5 Measures of Scale 6.1.5.1 Sample Variance 6.1.5.2 Range 6.1.5.3 Interquartile Range 6.1.6 Measures of Shape 6.1.6.1 Skewness 6.1.6.2 Kurtosis 6.1.7 Data Transformation 6.1.8 Example: Summary of Data and EDA 6.2 Sample Estimators 6.2.1 Point Estimation 6.2.2 Unbiased Estimators 6.2.3 Biased Estimators 6.2.4 Sufficiency 6.3 Bayesian Inference 6.3.1 Conjugate Priors 6.3.2 Continuous Parameter Estimation 6.3.2.1 Example: Continuous Bayesian Inference Using R 6.3.3 Discrete Parameter Estimation 6.3.4 Bayesian Credible Intervals 6.3.5 Prediction 6.3.6 Model Selection 6.4 Maximum Likelihood Estimation 6.4.1 Asymptotic Confidence Intervals for MLE 6.4.2 Bootstrap Confidence Intervals for MLE 6.4.3 Meaning of Confidence Intervals 6.5 Expectation-Maximization Algorithm 6.5.1 Example: EM Algorithm 6.6 Summary 6.7 Exercises 7 Clustering 7.1 Introduction 7.2 What Is Clustering? 7.3 Comparison of Data Points 7.3.1 Distance Measures 7.3.2 Similarity Measures 7.4 Basic Principle of Clustering Algorithms 7.5 Non-hierarchical Clustering Methods 7.5.1 K-Means Clustering 7.5.2 K-Medoids Clustering 7.5.3 Partitioning Around Medoids (PAM) 7.6 Hierarchical Clustering 7.6.1 Dendrograms 7.6.2 Two Types of Dissimilarity Measures 7.6.3 Linkage Functions for Agglomerative Clustering 7.6.4 Example 7.7 Defining Feature Vectors for General Objects 7.8 Cluster Validation 7.8.1 External Criteria 7.8.2 Assessing the Numerical Values of Indices 7.8.3 Internal Criteria 7.9 Summary 7.10 Exercises 8 Dimension Reduction 8.1 Introduction 8.2 Feature Extraction 8.2.1 An Overview of PCA 8.2.2 Geometrical Interpretation of PCA 8.2.3 PCA Procedure 8.2.4 Underlying Mathematical Problems in PCA 8.2.5 PCA Using Singular Value Decomposition 8.2.6 Assessing PCA Results 8.2.7 Illustration of PCA Using R 8.2.8 Kernel PCA 8.2.9 Discussion 8.2.10 Non-negative Matrix Factorization 8.2.10.1 NNMF Using the Frobenius Norm as Objective Function 8.2.10.2 NNMF Using the Generalized Kullback-Leibler Divergence as Objective Function 8.2.10.3 Example of NNMF Using R 8.3 Feature Selection 8.3.1 Filter Methods Using Mutual Information 8.4 Summary 8.5 Exercises 9 Classification 9.1 Introduction 9.2 What Is Classification? 9.3 Common Aspects of Classification Methods 9.3.1 Basic Idea of a Classifier 9.3.2 Training and Test Data 9.3.3 Error Measures 9.3.3.1 Error Measures for Multi-class Classification 9.4 Naive Bayes Classifier 9.4.1 Educational Example 9.4.2 Example 9.5 Linear Discriminant Analysis 9.5.1 Extensions 9.6 Logistic Regression 9.7 k-Nearest Neighbor Classifier 9.8 Support Vector Machine 9.8.1 Linearly Separable Data 9.8.2 Nonlinearly Separable Data 9.8.3 Nonlinear Support Vector Machines 9.8.4 Examples 9.9 Decision Tree 9.9.1 What Is a Decision Tree? 9.9.1.1 Three Principal Steps to Get a Decision Tree 9.9.2 Step 1: Growing a Decision Tree 9.9.3 Step 2: Assessing the Size of a Decision Tree 9.9.3.1 Intuitive Approach 9.9.3.2 Formal Approach 9.9.4 Step 3: Pruning a Decision Tree 9.9.4.1 Alternative Way to Construct Optimal Decision Trees: Stopping Rules 9.9.5 Predictions 9.10 Summary 9.11 Exercises 10 Hypothesis Testing 10.1 Introduction 10.2 What Is Hypothesis Testing? 10.3 Key Components of Hypothesis Testing 10.3.1 Step 1: Select Test Statistic 10.3.2 Step 2: Null Hypothesis H0 and AlternativeHypothesis H1 10.3.3 Step 3: Sampling Distribution 10.3.3.1 Examples 10.3.4 Step 4: Significance Level α 10.3.5 Step 5: Evaluate the Test Statistic from Data 10.3.6 Step 6: Determine the p-Value 10.3.7 Step 7: Make a Decision about the Null Hypothesis 10.4 Type 2 Error and Power 10.4.1 Connections between Power and Errors 10.5 Confidence Intervals 10.5.1 Confidence Intervals for a Population Mean with Known Variance 10.5.2 Confidence Intervals for a Population Mean with Unknown Variance 10.5.3 Bootstrap Confidence Intervals 10.6 Important Hypothesis Tests 10.6.1 Student's t-Test 10.6.1.1 One-Sample t-Test 10.6.1.2 Two-Sample t-Test 10.6.1.3 Extensions 10.6.2 Correlation Tests 10.6.3 Hypergeometric Test 10.6.3.1 Null Hypothesis and Sampling Distribution 10.6.3.2 Examples 10.6.4 Finding the Correct Hypothesis Test 10.7 Permutation Tests 10.8 Understanding versus Applying Hypothesis Tests 10.9 Historical Notes and Misinterpretations 10.10 Summary 10.11 Exercises 11 Linear Regression Models 11.1 Introduction 11.1.1 What Is Linear Regression? 11.1.2 Motivating Example 11.2 Simple Linear Regression 11.2.1 Ordinary Least Squares Estimation of Coefficients 11.2.2 Variability of the Coefficients 11.2.3 Testing the Necessity of Coefficients 11.2.4 Assessing the Quality of a Fit 11.3 Preprocessing 11.4 Multiple Linear Regression 11.4.1 Testing the Necessity of Coefficients 11.4.2 Assessing the Quality of a Fit 11.5 Diagnosing Linear Models 11.5.1 Error Assumptions 11.5.2 Linearity Assumption of the Model 11.5.3 Leverage Points 11.5.4 Outliers 11.5.5 Collinearity 11.5.6 Discussion 11.6 Advanced Topics 11.6.1 Interactions 11.6.2 Nonlinearities 11.6.3 Categorical Predictors 11.6.4 Generalized Linear Models 11.6.4.1 How to Determine Which Family to Use When Fitting a GLM 11.6.4.2 Advantages of GLMs over Traditional OLS Regression 11.6.4.3 Example: Poisson Regression 11.6.4.4 Example: Logistic Regression 11.7 Summary 11.8 Exercises 12 Model Selection 12.1 Introduction 12.2 Difference Between Model Selection and Model Assessment 12.3 General Approach to Model Selection 12.4 Model Selection for Multiple Linear Regression Models 12.4.1 R2 and Adjusted R2 12.4.2 Mallow's Cp Statistic 12.4.3 Akaike's Information Criterion (AIC) and Schwarz's BIC 12.4.4 Best Subset Selection 12.4.5 Stepwise Selection 12.4.5.1 Forward Stepwise Selection 12.4.5.2 Backward Stepwise Selection 12.5 Model Selection for Generalized Linear Models 12.5.1 Negative Binomial Regression Model 12.5.2 Zero-Inflated Poisson Model 12.5.3 Quasi-Poisson Model 12.5.4 Comparison of GLMs 12.6 Model Selection for Bayesian Models 12.7 Nonparametric Model Selection for General Models with Resampling 12.8 Summary 12.9 Exercises Part III Advanced Topics 13 Regularization 13.1 Introduction 13.2 Preliminaries 13.2.1 Preprocessing and Norms 13.2.2 Data 13.2.3 R Packages for Regularization 13.3 Ridge Regression 13.3.1 Example 13.4 Non-negative Garrote Regression 13.5 LASSO 13.5.1 Example 13.5.2 Explanation of Variable Selection 13.5.3 Discussion 13.5.4 Limitations 13.6 Ridge Regression 13.7 Dantzig Selector 13.8 Adaptive LASSO 13.8.1 Example 13.9 Elastic Net 13.9.1 Example 13.9.2 Discussion 13.10 Group LASSO 13.10.1 Example 13.10.2 Remarks 13.11 Discussion 13.12 Summary 13.13 Exercises 14 Deep Learning 14.1 Introduction 14.2 Architectures of Classical Neural Networks 14.2.1 Mathematical Model of an Artificial Neuron 14.2.2 Feedforward Neural Networks 14.2.3 Recurrent Neural Networks 14.2.3.1 Hopfield Networks 14.2.3.2 Boltzmann Machine 14.2.4 Overview of General Network Architectures 14.3 Deep Feedforward Neural Networks 14.3.1 Example: Deep Feedforward Neural Networks 14.4 Convolutional Neural Networks 14.4.1 Basic Components of a CNN 14.4.1.1 Convolutional Layer 14.4.1.2 Pooling Layer 14.4.1.3 Fully Connected Layer 14.4.2 Important Variants of CNN 14.4.3 Example: CNN 14.5 Deep Belief Networks 14.5.1 Pre-training Phase: Unsupervised 14.5.2 Fine-Tuning Phase: Supervised 14.6 Autoencoder 14.6.1 Example: Denoising and Variational Autoencoder 14.7 Long Short-Term Memory Networks 14.7.1 LSTM Network Structure with Forget Gate 14.7.2 Peephole LSTM 14.7.3 Applications 14.7.4 Example: LSTM 14.8 Discussion 14.8.1 General Characteristics of Deep Learning 14.8.2 Explainable AI 14.8.3 Big Data versus Small Data 14.8.4 Advanced Models 14.9 Summary 14.10 Exercises 15 Multiple Testing Corrections 15.1 Introduction 15.2 Preliminaries 15.2.1 Formal Setting 15.2.2 Simulations Using R 15.2.3 Focus on Pairwise Correlations 15.2.4 Focus on a Network Correlation Structure 15.2.5 Application of Multiple Testing Procedures 15.3 Motivation of the Problem 15.3.1 Theoretical Considerations 15.3.2 Experimental Example 15.4 Types of Multiple Testing Procedures 15.4.1 Single-Step versus Stepwise Approaches 15.4.2 Adaptive versus Nonadaptive Approaches 15.4.3 Marginal versus Joint Multiple Testing Procedures 15.5 Controlling the FWER 15.5.1 Šidák Correction 15.5.2 Bonferroni Correction 15.5.3 Holm Correction 15.5.4 Hochberg Correction 15.5.5 Hommel Correction 15.5.5.1 Examples 15.5.6 Westfall-Young Procedure 15.6 Controlling the FDR 15.6.1 Benjamini-Hochberg Procedure 15.6.1.1 Example 15.6.2 Adaptive Benjamini-Hochberg Procedure 15.6.3 Benjamini-Yekutieli Procedure 15.6.3.1 Example 15.6.4 Benjamini-Krieger-Yekutieli Procedure 15.6.5 Blanchard-Roquain Procedure 15.6.5.1 BR-1S Procedure 15.6.5.2 BR-2S Procedure 15.7 Computational Complexity 15.8 Comparison 15.9 Summary 15.10 Exercises 16 Survival Analysis 16.1 Introduction 16.2 Motivation 16.2.1 Effect of Chemotherapy: Breast Cancer Patients 16.2.2 Effect of Medication: Agitation 16.3 Censoring 16.4 General Characteristics of a Survival Function 16.5 Nonparametric Estimator for the Survival Function 16.5.1 Kaplan-Meier Estimator for the Survival Function 16.5.2 Nelson-Aalen Estimator for the Survival Function 16.6 Comparison of Two Survival Curves 16.6.1 Log-Rank Test 16.7 Hazard Function 16.7.1 Weibull Model 16.7.2 Exponential Model 16.7.3 Log-Logistic Model 16.7.4 Log-Normal Model 16.7.5 Interpretation of Hazard Functions 16.8 Cox Proportional Hazard Model 16.8.1 Why Is the Model Called a Proportional Hazard Model? 16.8.2 Interpretation of General Hazard Ratios 16.8.3 Adjusted Survival Curves 16.8.4 Testing the Proportional Hazard Assumption 16.8.4.1 Graphical Evaluation 16.8.4.2 Goodness-of-Fit Test 16.8.5 Parameter Estimation of the CPHM via Maximum Likelihood 16.8.5.1 Case Without Ties 16.8.5.2 Case with Ties 16.9 Stratified Cox Model 16.9.1 Testing No-Interaction Assumption 16.9.2 Case of Many Covariates Violating thePH Assumption 16.10 Survival Analysis Using R 16.10.1 Comparison of Survival Curves 16.10.2 Analyzing a Cox Proportional Hazard Model 16.10.3 Testing the PH Assumption 16.10.4 Hazard Ratios 16.11 Further Reading 16.12 Summary 16.13 Exercises 17 Foundations of Learning from Data 17.1 Introduction 17.2 Computational and Statistical Learning Theory 17.2.1 Probabilistic Learnability 17.2.2 Probably Approximately Correct (PAC) Learning 17.2.2.1 Example: Rectangle Learning 17.2.2.2 General Bound for a Finite Hypothesis Space H: The Inconsistent Case 17.2.3 Vapnik-Chervonenkis (VC) Theory 17.2.3.1 Example: One-dimensional Intervals 17.2.3.2 Example: Axis-Aligned Rectangles 17.3 Importance of Bias for Learning 17.4 Learning as Optimization Problem 17.4.1 Empirical Risk Minimization 17.4.2 Structural Risk Minimization 17.5 Fundamental Theorem of Statistical Learning 17.6 Discussion 17.7 Modern Machine Learning Paradigms 17.7.1 Semi-supervised Learning 17.7.1.1 Methodological Approaches 17.7.2 One-Class Classification 17.7.2.1 Methodological Approaches 17.7.3 Positive-Unlabeled Learning 17.7.3.1 Methodological Approaches 17.7.4 Few/One-Shot Learning 17.7.4.1 Methodological Approaches 17.7.5 Transfer Learning 17.7.5.1 Methodological Approaches 17.7.6 Multi-Task Learning 17.7.6.1 Methodological Approaches 17.7.7 Multi-Label Learning 17.7.7.1 Methodological Approaches 17.8 Summary 17.9 Exercises 18 Generalization Error and Model Assessment 18.1 Introduction 18.2 Overall View of Model Diagnosis 18.3 Expected Generalization Error 18.4 Bias-Variance Trade-Off 18.5 Error-Complexity Curves 18.5.1 Example: Linear Polynomial Regression Model 18.5.2 Example: Error-Complexity Curves 18.5.3 Interpretation of Error-Complexity Curves 18.6 Learning Curves 18.6.1 Example: Learning Curves for Linear Polynomial Regression Models 18.6.2 Interpretation of Learning Curves 18.7 Discussion 18.8 Summary 18.9 Outlook 18.10 Exercises References Index