Machine Learning: A Bayesian and Optimization Perspective (Solutions) (Instructor's Solution Manual)

Contents......Page 7 About the Author......Page 20 Preface......Page 21 Acknowledgments......Page 23 Notation......Page 25 1.1 The Historical Context......Page 26 1.2 Artificial Intelligence and Machine Learning......Page 27 1.3 Algorithms Can Learn What Is Hidden in the Data......Page 29 Multimodal Data......Page 31 Autonomous Cars......Page 32 1.5.1 Supervised Learning......Page 33 Classification......Page 34 1.6 Unsupervised and Semisupervised Learning......Page 36 1.7 Structure and a Road Map of the Book......Page 37 References......Page 41 2 Probability and Stochastic Processes......Page 43 2.2.1 Probability......Page 44 Axiomatic Definition......Page 45 Joint and Conditional Probabilities......Page 46 Bayes Theorem......Page 47 2.2.3 Continuous Random Variables......Page 48 2.2.4 Mean and Variance......Page 49 Complex Random Variables......Page 51 2.2.5 Transformation of Random Variables......Page 52 The Bernoulli Distribution......Page 53 The Binomial Distribution......Page 54 The Multinomial Distribution......Page 55 The Gaussian Distribution......Page 56 The Central Limit Theorem......Page 60 The Beta Distribution......Page 61 The Gamma Distribution......Page 62 The Dirichlet Distribution......Page 63 2.4 Stochastic Processes......Page 65 2.4.1 First- and Second-Order Statistics......Page 66 2.4.2 Stationarity and Ergodicity......Page 67 Properties of the Autocorrelation Sequence......Page 70 Power Spectral Density......Page 71 Transmission Through a Linear System......Page 72 Physical Interpretation of the PSD......Page 74 2.4.4 Autoregressive Models......Page 75 2.5 Information Theory......Page 78 Mutual and Conditional Information......Page 80 Entropy and Average Mutual Information......Page 82 2.5.2 Continuous Random Variables......Page 83 2.6 Stochastic Convergence......Page 85 Convergence in the Mean-Square Sense......Page 86 Problems......Page 87 References......Page 89 3.1 Introduction......Page 90 3.2 Parameter Estimation: the Deterministic Point of View......Page 91 3.3 Linear Regression......Page 94 3.4 Classification......Page 98 Generative Versus Discriminative Learning......Page 101 3.5 Biased Versus Unbiased Estimation......Page 103 3.5.1 Biased or Unbiased Estimation?......Page 104 3.6 The Cramér-Rao Lower Bound......Page 106 3.7 Sufficient Statistic......Page 110 3.8 Regularization......Page 112 Inverse Problems: Ill-Conditioning and Overfitting......Page 114 3.9 The Bias-Variance Dilemma......Page 116 3.9.1 Mean-Square Error Estimation......Page 117 3.9.2 Bias-Variance Tradeoff......Page 118 3.10 Maximum Likelihood Method......Page 121 3.10.1 Linear Regression: the Nonwhite Gaussian Noise Case......Page 124 3.11 Bayesian Inference......Page 125 3.11.1 The Maximum a Posteriori Probability Estimation Method......Page 130 3.12 Curse of Dimensionality......Page 131 3.13 Validation......Page 132 Cross-Validation......Page 134 3.14 Expected Loss and Empirical Risk Functions......Page 135 Learnability......Page 136 Problems......Page 137 References......Page 142 4.1 Introduction......Page 144 4.2 Mean-Square Error Linear Estimation: the Normal Equations......Page 145 4.2.1 The Cost Function Surface......Page 146 4.3 A Geometric Viewpoint: Orthogonality Condition......Page 147 4.4 Extension to Complex-Valued Variables......Page 150 4.4.1 Widely Linear Complex-Valued Estimation......Page 152 Circularity Conditions......Page 153 4.4.2 Optimizing With Respect to Complex-Valued Variables: Wirtinger Calculus......Page 155 4.5 Linear Filtering......Page 157 4.6 MSE Linear Filtering: a Frequency Domain Point of View......Page 159 Deconvolution: Image Deblurring......Page 160 4.7.1 Interference Cancelation......Page 163 4.7.2 System Identification......Page 164 4.7.3 Deconvolution: Channel Equalization......Page 166 4.8 Algorithmic Aspects: the Levinson and Lattice-Ladder Algorithms......Page 172 Forward and Backward MSE Optimal Predictors......Page 174 4.8.1 The Lattice-Ladder Scheme......Page 177 Orthogonality of the Optimal Backward Errors......Page 178 4.9 Mean-Square Error Estimation of Linear Models......Page 181 4.9.1 The Gauss-Markov Theorem......Page 183 4.9.2 Constrained Linear Estimation: the Beamforming Case......Page 185 4.10 Time-Varying Statistics: Kalman Filtering......Page 189 Problems......Page 195 MATLAB® Exercises......Page 197 References......Page 199 5 Online Learning: the Stochastic Gradient Descent Family of Algorithms......Page 201 5.1 Introduction......Page 202 5.2 The Steepest Descent Method......Page 203 5.3 Application to the Mean-Square Error Cost Function......Page 206 Time-Varying Step Sizes......Page 212 5.3.1 The Complex-Valued Case......Page 215 5.4 Stochastic Approximation......Page 216 Application to the MSE Linear Estimation......Page 218 5.5 The Least-Mean-Squares Adaptive Algorithm......Page 220 Convergence of the Parameter Error Vector......Page 221 5.5.2 Cumulative Loss Bounds......Page 226 5.6 The Affine Projection Algorithm......Page 228 Orthogonal Projections......Page 230 5.6.1 The Normalized LMS......Page 233 The Widely Linear LMS......Page 235 The Sign-Error LMS......Page 236 Transform-Domain LMS......Page 237 5.9 Simulation Examples......Page 240 5.10 Adaptive Decision Feedback Equalization......Page 243 5.11 The Linearly Constrained LMS......Page 246 5.12 Tracking Performance of the LMS in Nonstationary Environments......Page 247 5.13 Distributed Learning: the Distributed LMS......Page 249 5.13.1 Cooperation Strategies......Page 250 Decentralized Networks......Page 251 5.13.2 The Diffusion LMS......Page 253 5.13.3 Convergence and Steady-State Performance: Some Highlights......Page 259 5.13.4 Consensus-Based Distributed Schemes......Page 262 5.14 A Case Study: Target Localization......Page 263 5.15 Some Concluding Remarks: Consensus Matrix......Page 265 Problems......Page 266 MATLAB® Exercises......Page 268 References......Page 269 6.1 Introduction......Page 274 6.2 Least-Squares Linear Regression: a Geometric Perspective......Page 275 Covariance Matrix of the LS Estimator......Page 278 The LS Estimator Is BLUE in the Presence of White Noise......Page 279 The LS Estimator Achieves the Cramér-Rao Bound for White Gaussian Noise......Page 280 6.4 Orthogonalizing the Column Space of the Input Matrix: the SVD Method......Page 281 Pseudoinverse Matrix and SVD......Page 283 6.5 Ridge Regression: a Geometric Point of View......Page 286 Principal Components Regression......Page 288 6.6 The Recursive Least-Squares Algorithm......Page 289 Time-Iterative Computations......Page 290 Time Updating of the Parameters......Page 291 6.7 Newton's Iterative Minimization Method......Page 292 6.7.1 RLS and Newton's Method......Page 295 6.8 Steady-State Performance of the RLS......Page 296 6.9 Complex-Valued Data: the Widely Linear RLS......Page 298 QR Factorization......Page 300 Fast RLS Versions......Page 301 6.11 The Coordinate and Cyclic Coordinate Descent Methods......Page 302 6.12 Simulation Examples......Page 304 6.13 Total Least-Squares......Page 307 Geometric Interpretation of the Total Least-Squares Method......Page 312 Problems......Page 314 MATLAB® Exercises......Page 317 References......Page 318 7.1 Introduction......Page 321 7.2 Bayesian Classification......Page 322 The Bayesian Classifier Minimizes the Misclassification Error......Page 323 7.2.1 Average Risk......Page 324 7.3 Decision (Hyper)Surfaces......Page 327 7.3.1 The Gaussian Distribution Case......Page 329 Minimum Distance Classifiers......Page 331 7.5 The Nearest Neighbor Rule......Page 335 7.6 Logistic Regression......Page 337 7.7 Fisher's Linear Discriminant......Page 342 7.7.1 Scatter Matrices......Page 343 7.7.2 Fisher's Discriminant: the Two-Class Case......Page 345 7.7.3 Fisher's Discriminant: the Multiclass Case......Page 348 7.8 Classification Trees......Page 349 7.9 Combining Classifiers......Page 353 Some Experimental Comparisons......Page 354 Schemes for Combining Classifiers......Page 355 The AdaBoost Algorithm......Page 357 The Log-Loss Function......Page 361 7.11 Boosting Trees......Page 363 Problems......Page 365 MATLAB® Exercises......Page 367 References......Page 369 8 Parameter Learning: a Convex Analytic Path......Page 371 8.2 Convex Sets and Functions......Page 372 8.2.1 Convex Sets......Page 373 8.2.2 Convex Functions......Page 374 8.3 Projections Onto Convex Sets......Page 377 8.3.1 Properties of Projections......Page 381 8.4 Fundamental Theorem of Projections Onto Convex Sets......Page 385 8.6.1 Regression......Page 389 8.6.2 Classification......Page 393 8.7 Infinitely Many Closed Convex Sets: the Online Learning Case......Page 394 8.7.1 Convergence of APSM......Page 396 Some Practical Hints......Page 398 8.8 Constrained Learning......Page 400 8.9 The Distributed APSM......Page 402 8.10 Optimizing Nonsmooth Convex Cost Functions......Page 404 8.10.1 Subgradients and Subdifferentials......Page 405 The Subgradient Method......Page 408 The Projected Gradient Method (PGM)......Page 411 Projected Subgradient Method......Page 412 8.10.3 Online Learning for Convex Optimization......Page 413 The PEGASOS Algorithm......Page 415 8.11 Regret Analysis......Page 416 Regret Analysis of the Subgradient Algorithm......Page 418 8.12 Online Learning and Big Data Applications: a Discussion......Page 419 Approximation, Estimation, and Optimization Errors......Page 420 Batch Versus Online Learning......Page 422 8.13 Proximal Operators......Page 425 8.13.1 Properties of the Proximal Operator......Page 427 8.13.2 Proximal Minimization......Page 429 Resolvent of the Subdifferential Mapping......Page 431 8.14 Proximal Splitting Methods for Optimization......Page 432 The Proximal Forward-Backward Splitting Operator......Page 433 Alternating Direction Method of Multipliers (ADMM)......Page 434 Mirror Descent Algorithms......Page 435 Problems......Page 437 MATLAB® Exercises......Page 440 References......Page 442 9.1 Introduction......Page 446 9.2 Searching for a Norm......Page 447 9.3 The Least Absolute Shrinkage and Selection Operator (LASSO)......Page 450 9.4 Sparse Signal Representation......Page 455 9.5 In Search of the Sparsest Solution......Page 459 The l2 Norm Minimizer......Page 460 The l1 Norm Minimizer......Page 461 Characterization of the l1 Norm Minimizer......Page 462 Geometric Interpretation......Page 463 9.6 Uniqueness of the l0 Minimizer......Page 466 9.6.1 Mutual Coherence......Page 468 9.7.1 Condition Implied by the Mutual Coherence Number......Page 470 9.7.2 The Restricted Isometry Property (RIP)......Page 471 Constructing Matrices That Obey the RIP of Order k......Page 472 9.8 Robust Sparse Signal Recovery From Noisy Measurements......Page 474 Compressed Sensing......Page 475 9.9.1 Dimensionality Reduction and Stable Embeddings......Page 477 9.9.2 Sub-Nyquist Sampling: Analog-to-Information Conversion......Page 479 9.10 A Case Study: Image Denoising......Page 482 Problems......Page 484 MATLAB® Exercises......Page 487 References......Page 488 10.1 Introduction......Page 492 10.2.1 Greedy Algorithms......Page 493 OMP Can Recover Optimal Sparse Solutions: Sufficiency Condition......Page 496 The LARS Algorithm......Page 497 Compressed Sensing Matching Pursuit (CSMP) Algorithms......Page 498 10.2.2 Iterative Shrinkage/Thresholding (IST) Algorithms......Page 499 10.2.3 Which Algorithm? Some Practical Hints......Page 506 10.3 Variations on the Sparsity-Aware Theme......Page 511 10.4 Online Sparsity Promoting Algorithms......Page 518 10.4.1 LASSO: Asymptotic Performance......Page 519 10.4.2 The Adaptive Norm-Weighted LASSO......Page 521 10.4.3 Adaptive CoSaMP Algorithm......Page 523 10.4.4 Sparse-Adaptive Projection Subgradient Method......Page 524 Projection Onto the Weighted l1 Ball......Page 526 10.5 Learning Sparse Analysis Models......Page 529 10.5.1 Compressed Sensing for Sparse Signal Representation in Coherent Dictionaries......Page 531 10.5.2 Cosparsity......Page 532 Gabor Transform and Frames......Page 535 Time-Frequency Resolution......Page 536 Gabor Frames......Page 537 Time-Frequency Analysis of Echolocation Signals Emitted by Bats......Page 538 Problems......Page 542 MATLAB® Exercises......Page 543 References......Page 544 11 Learning in Reproducing Kernel Hilbert Spaces......Page 549 11.2 Generalized Linear Models......Page 550 11.3 Volterra, Wiener, and Hammerstein Models......Page 551 11.4 Cover's Theorem: Capacity of a Space in Linear Dichotomies......Page 554 11.5 Reproducing Kernel Hilbert Spaces......Page 557 11.5.1 Some Properties and Theoretical Highlights......Page 559 11.5.2 Examples of Kernel Functions......Page 561 Constructing Kernels......Page 564 String Kernels......Page 565 11.6 Representer Theorem......Page 566 11.6.1 Semiparametric Representer Theorem......Page 568 11.7 Kernel Ridge Regression......Page 569 11.8 Support Vector Regression......Page 572 11.8.1 The Linear ε-Insensitive Optimal Regression......Page 573 The Solution......Page 574 Solving the Optimization Task......Page 575 11.9 Kernel Ridge Regression Revisited......Page 579 11.10 Optimal Margin Classification: Support Vector Machines......Page 580 11.10.1 Linearly Separable Classes: Maximum Margin Classifiers......Page 582 The Solution......Page 585 The Optimization Task......Page 586 11.10.2 Nonseparable Classes......Page 587 The Optimization Task......Page 588 11.10.4 Choice of Hyperparameters......Page 592 11.10.5 Multiclass Generalizations......Page 593 11.11 Computational Considerations......Page 594 11.12 Random Fourier Features......Page 595 11.12.1 Online and Distributed Learning in RKHS......Page 597 11.13 Multiple Kernel Learning......Page 598 11.14 Nonparametric Sparsity-Aware Learning: Additive Models......Page 600 11.15 A Case Study: Authorship Identification......Page 602 Problems......Page 605 MATLAB® Exercises......Page 607 References......Page 608 12.1 Introduction......Page 613 12.2 Regression: a Bayesian Perspective......Page 614 12.2.1 The Maximum Likelihood Estimator......Page 615 12.2.2 The MAP Estimator......Page 616 12.2.3 The Bayesian Approach......Page 617 12.3 The Evidence Function and Occam's Razor Rule......Page 623 Laplacian Approximation and the Evidence Function......Page 625 12.4.1 The Expectation-Maximization Algorithm......Page 629 12.5 Linear Regression and the EM Algorithm......Page 631 12.6 Gaussian Mixture Models......Page 634 12.6.1 Gaussian Mixture Modeling and Clustering......Page 638 12.7 The EM Algorithm: a Lower Bound Maximization View......Page 641 12.8 Exponential Family of Probability Distributions......Page 645 12.8.1 The Exponential Family and the Maximum Entropy Method......Page 651 12.9.1 Mixing Linear Regression Models......Page 652 Hierarchical Mixture of Experts......Page 656 12.9.2 Mixing Logistic Regression Models......Page 657 Problems......Page 659 MATLAB® Exercises......Page 661 References......Page 663 13 Bayesian Learning: Approximate Inference and Nonparametric Models......Page 665 13.2 Variational Approximation in Bayesian Learning......Page 666 The Mean Field Approximation......Page 667 13.2.1 The Case of the Exponential Family of Probability Distributions......Page 671 13.3 A Variational Bayesian Approach to Linear Regression......Page 673 Computation of the Lower Bound......Page 678 13.4 A Variational Bayesian Approach to Gaussian Mixture Modeling......Page 679 13.5 When Bayesian Inference Meets Sparsity......Page 683 13.6 Sparse Bayesian Learning (SBL)......Page 685 13.6.1 The Spike and Slab Method......Page 689 13.7.1 Adopting the Logistic Regression Model for Classification......Page 690 13.8 Convex Duality and Variational Bounds......Page 694 13.9 Sparsity-Aware Regression: a Variational Bound Bayesian Path......Page 699 13.10 Expectation Propagation......Page 704 The Expectation Propagation Algorithm......Page 706 13.11 Nonparametric Bayesian Modeling......Page 708 13.11.1 The Chinese Restaurant Process......Page 709 13.11.2 Dirichlet Processes......Page 710 Predictive Distribution and the Pólya Urn Model......Page 713 13.11.3 The Stick Breaking Construction of a DP......Page 715 13.11.4 Dirichlet Process Mixture Modeling......Page 716 Inference......Page 717 13.11.5 The Indian Buffet Process......Page 719 Searching for a Prior on Infinite Binary Matrices......Page 721 Restaurant Construction......Page 724 Stick Breaking Construction......Page 726 Inference......Page 727 13.12 Gaussian Processes......Page 728 13.12.1 Covariance Functions and Kernels......Page 729 13.12.2 Regression......Page 730 Computational Considerations......Page 733 13.12.3 Classification......Page 734 13.13 A Case Study: Hyperspectral Image Unmixing......Page 735 13.13.1 Hierarchical Bayesian Modeling......Page 737 13.13.2 Experimental Results......Page 738 Problems......Page 739 MATLAB® Exercises......Page 744 References......Page 745 14.1 Introduction......Page 749 14.2 Monte Carlo Methods: the Main Concept......Page 750 14.2.1 Random Number Generation......Page 751 14.3 Random Sampling Based on Function Transformation......Page 753 14.4 Rejection Sampling......Page 757 14.5 Importance Sampling......Page 761 14.7 Markov Chain Monte Carlo Methods......Page 763 14.7.1 Ergodic Markov Chains......Page 766 14.8 The Metropolis Method......Page 772 14.8.1 Convergence Issues......Page 774 14.9 Gibbs Sampling......Page 776 14.10 In Search of More Efficient Methods: a Discussion......Page 778 14.11 A Case Study: Change-Point Detection......Page 780 Problems......Page 783 MATLAB® Exercise......Page 785 References......Page 786 15.1 Introduction......Page 788 15.2 The Need for Graphical Models......Page 789 15.3 Bayesian Networks and the Markov Condition......Page 791 15.3.1 Graphs: Basic Definitions......Page 792 15.3.2 Some Hints on Causality......Page 796 15.3.3 d-Separation......Page 798 15.3.4 Sigmoidal Bayesian Networks......Page 802 15.3.6 Multiple-Cause Networks......Page 803 15.3.7 I-Maps, Soundness, Faithfulness, and Completeness......Page 804 15.4 Undirected Graphical Models......Page 805 15.4.1 Independencies and I-Maps in Markov Random Fields......Page 807 15.4.2 The Ising Model and Its Variants......Page 808 15.4.3 Conditional Random Fields (CRFs)......Page 811 15.5 Factor Graphs......Page 812 15.5.1 Graphical Models for Error Correcting Codes......Page 814 15.6 Moralization of Directed Graphs......Page 815 15.7.1 Exact Inference in Chains......Page 816 15.7.2 Exact Inference in Trees......Page 820 15.7.3 The Sum-Product Algorithm......Page 821 15.7.4 The Max-Product and Max-Sum Algorithms......Page 826 Problems......Page 833 References......Page 835 16.1 Introduction......Page 838 16.2 Triangulated Graphs and Junction Trees......Page 839 16.2.1 Constructing a Join Tree......Page 842 16.2.2 Message Passing in Junction Trees......Page 844 16.3 Approximate Inference Methods......Page 847 Multiple-Cause Networks and the Noisy-OR Model......Page 848 The Boltzmann Machine......Page 850 16.3.2 Block Methods for Variational Approximation......Page 852 The Mean Field Approximation and the Boltzmann Machine......Page 853 16.3.3 Loopy Belief Propagation......Page 856 16.4 Dynamic Graphical Models......Page 859 16.5 Hidden Markov Models......Page 861 The Sum-Product Algorithm: the HMM Case......Page 864 16.5.2 Learning the Parameters in an HMM......Page 869 16.5.3 Discriminative Learning......Page 872 16.6.1 Factorial Hidden Markov Models......Page 873 16.7 Learning Graphical Models......Page 876 16.7.1 Parameter Estimation......Page 877 Problems......Page 881 References......Page 884 17.2 Sequential Importance Sampling......Page 887 17.2.1 Importance Sampling Revisited......Page 888 17.2.2 Resampling......Page 889 17.2.3 Sequential Sampling......Page 891 17.3.1 Kalman Filtering: a Bayesian Point of View......Page 894 17.4 Particle Filtering......Page 897 17.4.1 Degeneracy......Page 901 17.4.2 Generic Particle Filtering......Page 902 17.4.3 Auxiliary Particle Filtering......Page 905 Problems......Page 911 MATLAB® Exercises......Page 914 References......Page 915 18 Neural Networks and Deep Learning......Page 917 18.1 Introduction......Page 918 18.2 The Perceptron......Page 920 18.3 Feed-Forward Multilayer Neural Networks......Page 924 18.3.1 Fully Connected Networks......Page 928 18.4 The Backpropagation Algorithm......Page 929 Nonconvexity of the Cost Function......Page 930 18.4.1 The Gradient Descent Backpropagation Scheme......Page 932 Pattern-by-Pattern/Online Scheme......Page 938 Minibatch Schemes......Page 939 18.4.2 Variants of the Basic Gradient Descent Scheme......Page 940 Gradient Descent With a Momentum Term......Page 941 The AdaGrad Algorithm......Page 943 The RMSProp With Nesterov Momentum......Page 944 The Adaptive Moment Estimation Algorithm (Adam)......Page 945 Some Practical Hints......Page 946 Batch Normalization......Page 948 18.4.3 Beyond the Gradient Descent Rationale......Page 950 18.5 Selecting a Cost Function......Page 951 18.6 Vanishing and Exploding Gradients......Page 954 18.6.1 The Rectified Linear Unit......Page 955 18.7 Regularizing the Network......Page 956 Dropout......Page 959 18.8 Designing Deep Neural Networks: a Summary......Page 962 18.9 Universal Approximation Property of Feed-Forward Neural Networks......Page 963 18.10 Neural Networks: a Bayesian Flavor......Page 965 18.11 Shallow Versus Deep Architectures......Page 966 On the Representation Properties of Deep Networks......Page 967 Distributed Representations......Page 969 On the Optimization of Deep Networks: Some Theoretical Highlights......Page 970 On the Generalization Power of Deep Networks......Page 971 18.12.1 The Need for Convolutions......Page 972 The Convolution Step......Page 973 The Pooling Step......Page 979 18.12.2 Convolution Over Volumes......Page 981 Network in Network and 1x1 Convolution......Page 982 18.12.3 The Full CNN Architecture......Page 984 What Deep Neural Networks Learn......Page 985 18.12.4 CNNs: the Epilogue......Page 987 18.13 Recurrent Neural Networks......Page 992 18.13.1 Backpropagation Through Time......Page 994 Vanishing and Exploding Gradients......Page 995 The Long Short-Term Memory (LSTM) Network......Page 996 18.13.2 Attention and Memory......Page 998 18.14 Adversarial Examples......Page 1001 Adversarial Training......Page 1003 18.15.1 Restricted Boltzmann Machines......Page 1004 18.15.2 Pretraining Deep Feed-Forward Networks......Page 1007 18.15.3 Deep Belief Networks......Page 1008 18.15.4 Autoencoders......Page 1010 18.15.5 Generative Adversarial Networks......Page 1011 On the Optimality of the Solution......Page 1014 Problems in Training GANs......Page 1015 The Wasserstein GAN......Page 1017 Which Algorithm Then......Page 1019 18.15.6 Variational Autoencoders......Page 1020 18.16 Capsule Networks......Page 1023 Training......Page 1027 Transfer Learning......Page 1029 Multitask Learning......Page 1030 Geometric Deep Learning......Page 1031 Open Problems......Page 1032 18.18 A Case Study: Neural Machine Translation......Page 1033 18.19 Problems......Page 1039 Computer Exercises......Page 1041 References......Page 1045 19 Dimensionality Reduction and Latent Variable Modeling......Page 1055 19.1 Introduction......Page 1056 19.3 Principal Component Analysis......Page 1057 PCA, SVD, and Low Rank Matrix Factorization......Page 1059 PCA and Information Retrieval......Page 1061 Orthogonalizing Properties of PCA and Feature Generation......Page 1062 Latent Variables......Page 1063 19.4 Canonical Correlation Analysis......Page 1069 Partial Least-Squares......Page 1072 19.5.1 ICA and Gaussianity......Page 1074 19.5.2 ICA and Higher-Order Cumulants......Page 1075 ICA Ambiguities......Page 1076 19.5.3 Non-Gaussianity and Independent Components......Page 1077 19.5.4 ICA Based on Mutual Information......Page 1078 19.5.5 Alternative Paths to ICA......Page 1081 The Cocktail Party Problem......Page 1082 19.6 Dictionary Learning: the k-SVD Algorithm......Page 1085 Dictionary Learning and Dictionary Identifiability......Page 1088 19.7 Nonnegative Matrix Factorization......Page 1090 19.8 Learning Low-Dimensional Models: a Probabilistic Perspective......Page 1092 19.8.1 Factor Analysis......Page 1093 19.8.2 Probabilistic PCA......Page 1094 19.8.3 Mixture of Factors Analyzers: a Bayesian View to Compressed Sensing......Page 1098 19.9.1 Kernel PCA......Page 1101 Laplacian Eigenmaps......Page 1103 Local Linear Embedding (LLE)......Page 1107 Isometric Mapping (ISOMAP)......Page 1108 19.10.1 Matrix Completion......Page 1112 19.10.2 Robust PCA......Page 1116 Matrix Completion......Page 1117 Robust PCA/PCP......Page 1118 19.11 A Case Study: FMRI Data Analysis......Page 1119 MATLAB® Exercises......Page 1123 References......Page 1124 Index......Page 1132 Contents......Page 7 About the Author......Page 20 Preface......Page 21 Acknowledgments......Page 23 Notation......Page 25 1.1 The Historical Context......Page 26 1.2 Artificial Intelligence and Machine Learning......Page 27 1.3 Algorithms Can Learn What Is Hidden in the Data......Page 29 Multimodal Data......Page 31 Autonomous Cars......Page 32 1.5.1 Supervised Learning......Page 33 Classification......Page 34 1.6 Unsupervised and Semisupervised Learning......Page 36 1.7 Structure and a Road Map of the Book......Page 37 References......Page 41 2 Probability and Stochastic Processes......Page 43 2.2.1 Probability......Page 44 Axiomatic Definition......Page 45 Joint and Conditional Probabilities......Page 46 Bayes Theorem......Page 47 2.2.3 Continuous Random Variables......Page 48 2.2.4 Mean and Variance......Page 49 Complex Random Variables......Page 51 2.2.5 Transformation of Random Variables......Page 52 The Bernoulli Distribution......Page 53 The Binomial Distribution......Page 54 The Multinomial Distribution......Page 55 The Gaussian Distribution......Page 56 The Central Limit Theorem......Page 60 The Beta Distribution......Page 61 The Gamma Distribution......Page 62 The Dirichlet Distribution......Page 63 2.4 Stochastic Processes......Page 65 2.4.1 First- and Second-Order Statistics......Page 66 2.4.2 Stationarity and Ergodicity......Page 67 Properties of the Autocorrelation Sequence......Page 70 Power Spectral Density......Page 71 Transmission Through a Linear System......Page 72 Physical Interpretation of the PSD......Page 74 2.4.4 Autoregressive Models......Page 75 2.5 Information Theory......Page 78 Mutual and Conditional Information......Page 80 Entropy and Average Mutual Information......Page 82 2.5.2 Continuous Random Variables......Page 83 2.6 Stochastic Convergence......Page 85 Convergence in the Mean-Square Sense......Page 86 Problems......Page 87 References......Page 89 3.1 Introduction......Page 90 3.2 Parameter Estimation: the Deterministic Point of View......Page 91 3.3 Linear Regression......Page 94 3.4 Classification......Page 98 Generative Versus Discriminative Learning......Page 101 3.5 Biased Versus Unbiased Estimation......Page 103 3.5.1 Biased or Unbiased Estimation?......Page 104 3.6 The Cramér-Rao Lower Bound......Page 106 3.7 Sufficient Statistic......Page 110 3.8 Regularization......Page 112 Inverse Problems: Ill-Conditioning and Overfitting......Page 114 3.9 The Bias-Variance Dilemma......Page 116 3.9.1 Mean-Square Error Estimation......Page 117 3.9.2 Bias-Variance Tradeoff......Page 118 3.10 Maximum Likelihood Method......Page 121 3.10.1 Linear Regression: the Nonwhite Gaussian Noise Case......Page 124 3.11 Bayesian Inference......Page 125 3.11.1 The Maximum a Posteriori Probability Estimation Method......Page 130 3.12 Curse of Dimensionality......Page 131 3.13 Validation......Page 132 Cross-Validation......Page 134 3.14 Expected Loss and Empirical Risk Functions......Page 135 Learnability......Page 136 Problems......Page 137 References......Page 142 4.1 Introduction......Page 144 4.2 Mean-Square Error Linear Estimation: the Normal Equations......Page 145 4.2.1 The Cost Function Surface......Page 146 4.3 A Geometric Viewpoint: Orthogonality Condition......Page 147 4.4 Extension to Complex-Valued Variables......Page 150 4.4.1 Widely Linear Complex-Valued Estimation......Page 152 Circularity Conditions......Page 153 4.4.2 Optimizing With Respect to Complex-Valued Variables: Wirtinger Calculus......Page 155 4.5 Linear Filtering......Page 157 4.6 MSE Linear Filtering: a Frequency Domain Point of View......Page 159 Deconvolution: Image Deblurring......Page 160 4.7.1 Interference Cancelation......Page 163 4.7.2 System Identification......Page 164 4.7.3 Deconvolution: Channel Equalization......Page 166 4.8 Algorithmic Aspects: the Levinson and Lattice-Ladder Algorithms......Page 172 Forward and Backward MSE Optimal Predictors......Page 174 4.8.1 The Lattice-Ladder Scheme......Page 177 Orthogonality of the Optimal Backward Errors......Page 178 4.9 Mean-Square Error Estimation of Linear Models......Page 181 4.9.1 The Gauss-Markov Theorem......Page 183 4.9.2 Constrained Linear Estimation: the Beamforming Case......Page 185 4.10 Time-Varying Statistics: Kalman Filtering......Page 189 Problems......Page 195 MATLAB® Exercises......Page 197 References......Page 199 5 Online Learning: the Stochastic Gradient Descent Family of Algorithms......Page 201 5.1 Introduction......Page 202 5.2 The Steepest Descent Method......Page 203 5.3 Application to the Mean-Square Error Cost Function......Page 206 Time-Varying Step Sizes......Page 212 5.3.1 The Complex-Valued Case......Page 215 5.4 Stochastic Approximation......Page 216 Application to the MSE Linear Estimation......Page 218 5.5 The Least-Mean-Squares Adaptive Algorithm......Page 220 Convergence of the Parameter Error Vector......Page 221 5.5.2 Cumulative Loss Bounds......Page 226 5.6 The Affine Projection Algorithm......Page 228 Orthogonal Projections......Page 230 5.6.1 The Normalized LMS......Page 233 The Widely Linear LMS......Page 235 The Sign-Error LMS......Page 236 Transform-Domain LMS......Page 237 5.9 Simulation Examples......Page 240 5.10 Adaptive Decision Feedback Equalization......Page 243 5.11 The Linearly Constrained LMS......Page 246 5.12 Tracking Performance of the LMS in Nonstationary Environments......Page 247 5.13 Distributed Learning: the Distributed LMS......Page 249 5.13.1 Cooperation Strategies......Page 250 Decentralized Networks......Page 251 5.13.2 The Diffusion LMS......Page 253 5.13.3 Convergence and Steady-State Performance: Some Highlights......Page 259 5.13.4 Consensus-Based Distributed Schemes......Page 262 5.14 A Case Study: Target Localization......Page 263 5.15 Some Concluding Remarks: Consensus Matrix......Page 265 Problems......Page 266 MATLAB® Exercises......Page 268 References......Page 269 6.1 Introduction......Page 274 6.2 Least-Squares Linear Regression: a Geometric Perspective......Page 275 Covariance Matrix of the LS Estimator......Page 278 The LS Estimator Is BLUE in the Presence of White Noise......Page 279 The LS Estimator Achieves the Cramér-Rao Bound for White Gaussian Noise......Page 280 6.4 Orthogonalizing the Column Space of the Input Matrix: the SVD Method......Page 281 Pseudoinverse Matrix and SVD......Page 283 6.5 Ridge Regression: a Geometric Point of View......Page 286 Principal Components Regression......Page 288 6.6 The Recursive Least-Squares Algorithm......Page 289 Time-Iterative Computations......Page 290 Time Updating of the Parameters......Page 291 6.7 Newton's Iterative Minimization Method......Page 292 6.7.1 RLS and Newton's Method......Page 295 6.8 Steady-State Performance of the RLS......Page 296 6.9 Complex-Valued Data: the Widely Linear RLS......Page 298 QR Factorization......Page 300 Fast RLS Versions......Page 301 6.11 The Coordinate and Cyclic Coordinate Descent Methods......Page 302 6.12 Simulation Examples......Page 304 6.13 Total Least-Squares......Page 307 Geometric Interpretation of the Total Least-Squares Method......Page 312 Problems......Page 314 MATLAB® Exercises......Page 317 References......Page 318 7.1 Introduction......Page 321 7.2 Bayesian Classification......Page 322 The Bayesian Classifier Minimizes the Misclassification Error......Page 323 7.2.1 Average Risk......Page 324 7.3 Decision (Hyper)Surfaces......Page 327 7.3.1 The Gaussian Distribution Case......Page 329 Minimum Distance Classifiers......Page 331 7.5 The Nearest Neighbor Rule......Page 335 7.6 Logistic Regression......Page 337 7.7 Fisher's Linear Discriminant......Page 342 7.7.1 Scatter Matrices......Page 343 7.7.2 Fisher's Discriminant: the Two-Class Case......Page 345 7.7.3 Fisher's Discriminant: the Multiclass Case......Page 348 7.8 Classification Trees......Page 349 7.9 Combining Classifiers......Page 353 Some Experimental Comparisons......Page 354 Schemes for Combining Classifiers......Page 355 The AdaBoost Algorithm......Page 357 The Log-Loss Function......Page 361 7.11 Boosting Trees......Page 363 Problems......Page 365 MATLAB® Exercises......Page 367 References......Page 369 8 Parameter Learning: a Convex Analytic Path......Page 371 8.2 Convex Sets and Functions......Page 372 8.2.1 Convex Sets......Page 373 8.2.2 Convex Functions......Page 374 8.3 Projections Onto Convex Sets......Page 377 8.3.1 Properties of Projections......Page 381 8.4 Fundamental Theorem of Projections Onto Convex Sets......Page 385 8.6.1 Regression......Page 389 8.6.2 Classification......Page 393 8.7 Infinitely Many Closed Convex Sets: the Online Learning Case......Page 394 8.7.1 Convergence of APSM......Page 396 Some Practical Hints......Page 398 8.8 Constrained Learning......Page 400 8.9 The Distributed APSM......Page 402 8.10 Optimizing Nonsmooth Convex Cost Functions......Page 404 8.10.1 Subgradients and Subdifferentials......Page 405 The Subgradient Method......Page 408 The Projected Gradient Method (PGM)......Page 411 Projected Subgradient Method......Page 412 8.10.3 Online Learning for Convex Optimization......Page 413 The PEGASOS Algorithm......Page 415 8.11 Regret Analysis......Page 416 Regret Analysis of the Subgradient Algorithm......Page 418 8.12 Online Learning and Big Data Applications: a Discussion......Page 419 Approximation, Estimation, and Optimization Errors......Page 420 Batch Versus Online Learning......Page 422 8.13 Proximal Operators......Page 425 8.13.1 Properties of the Proximal Operator......Page 427 8.13.2 Proximal Minimization......Page 429 Resolvent of the Subdifferential Mapping......Page 431 8.14 Proximal Splitting Methods for Optimization......Page 432 The Proximal Forward-Backward Splitting Operator......Page 433 Alternating Direction Method of Multipliers (ADMM)......Page 434 Mirror Descent Algorithms......Page 435 Problems......Page 437 MATLAB® Exercises......Page 440 References......Page 442 9.1 Introduction......Page 446 9.2 Searching for a Norm......Page 447 9.3 The Least Absolute Shrinkage and Selection Operator (LASSO)......Page 450 9.4 Sparse Signal Representation......Page 455 9.5 In Search of the Sparsest Solution......Page 459 The l2 Norm Minimizer......Page 460 The l1 Norm Minimizer......Page 461 Characterization of the l1 Norm Minimizer......Page 462 Geometric Interpretation......Page 463 9.6 Uniqueness of the l0 Minimizer......Page 466 9.6.1 Mutual Coherence......Page 468 9.7.1 Condition Implied by the Mutual Coherence Number......Page 470 9.7.2 The Restricted Isometry Property (RIP)......Page 471 Constructing Matrices That Obey the RIP of Order k......Page 472 9.8 Robust Sparse Signal Recovery From Noisy Measurements......Page 474 Compressed Sensing......Page 475 9.9.1 Dimensionality Reduction and Stable Embeddings......Page 477 9.9.2 Sub-Nyquist Sampling: Analog-to-Information Conversion......Page 479 9.10 A Case Study: Image Denoising......Page 482 Problems......Page 484 MATLAB® Exercises......Page 487 References......Page 488 10.1 Introduction......Page 492 10.2.1 Greedy Algorithms......Page 493 OMP Can Recover Optimal Sparse Solutions: Sufficiency Condition......Page 496 The LARS Algorithm......Page 497 Compressed Sensing Matching Pursuit (CSMP) Algorithms......Page 498 10.2.2 Iterative Shrinkage/Thresholding (IST) Algorithms......Page 499 10.2.3 Which Algorithm? Some Practical Hints......Page 506 10.3 Variations on the Sparsity-Aware Theme......Page 511 10.4 Online Sparsity Promoting Algorithms......Page 518 10.4.1 LASSO: Asymptotic Performance......Page 519 10.4.2 The Adaptive Norm-Weighted LASSO......Page 521 10.4.3 Adaptive CoSaMP Algorithm......Page 523 10.4.4 Sparse-Adaptive Projection Subgradient Method......Page 524 Projection Onto the Weighted l1 Ball......Page 526 10.5 Learning Sparse Analysis Models......Page 529 10.5.1 Compressed Sensing for Sparse Signal Representation in Coherent Dictionaries......Page 531 10.5.2 Cosparsity......Page 532 Gabor Transform and Frames......Page 535 Time-Frequency Resolution......Page 536 Gabor Frames......Page 537 Time-Frequency Analysis of Echolocation Signals Emitted by Bats......Page 538 Problems......Page 542 MATLAB® Exercises......Page 543 References......Page 544 11 Learning in Reproducing Kernel Hilbert Spaces......Page 549 11.2 Generalized Linear Models......Page 550 11.3 Volterra, Wiener, and Hammerstein Models......Page 551 11.4 Cover's Theorem: Capacity of a Space in Linear Dichotomies......Page 554 11.5 Reproducing Kernel Hilbert Spaces......Page 557 11.5.1 Some Properties and Theoretical Highlights......Page 559 11.5.2 Examples of Kernel Functions......Page 561 Constructing Kernels......Page 564 String Kernels......Page 565 11.6 Representer Theorem......Page 566 11.6.1 Semiparametric Representer Theorem......Page 568 11.7 Kernel Ridge Regression......Page 569 11.8 Support Vector Regression......Page 572 11.8.1 The Linear ε-Insensitive Optimal Regression......Page 573 The Solution......Page 574 Solving the Optimization Task......Page 575 11.9 Kernel Ridge Regression Revisited......Page 579 11.10 Optimal Margin Classification: Support Vector Machines......Page 580 11.10.1 Linearly Separable Classes: Maximum Margin Classifiers......Page 582 The Solution......Page 585 The Optimization Task......Page 586 11.10.2 Nonseparable Classes......Page 587 The Optimization Task......Page 588 11.10.4 Choice of Hyperparameters......Page 592 11.10.5 Multiclass Generalizations......Page 593 11.11 Computational Considerations......Page 594 11.12 Random Fourier Features......Page 595 11.12.1 Online and Distributed Learning in RKHS......Page 597 11.13 Multiple Kernel Learning......Page 598 11.14 Nonparametric Sparsity-Aware Learning: Additive Models......Page 600 11.15 A Case Study: Authorship Identification......Page 602 Problems......Page 605 MATLAB® Exercises......Page 607 References......Page 608 12.1 Introduction......Page 613 12.2 Regression: a Bayesian Perspective......Page 614 12.2.1 The Maximum Likelihood Estimator......Page 615 12.2.2 The MAP Estimator......Page 616 12.2.3 The Bayesian Approach......Page 617 12.3 The Evidence Function and Occam's Razor Rule......Page 623 Laplacian Approximation and the Evidence Function......Page 625 12.4.1 The Expectation-Maximization Algorithm......Page 629 12.5 Linear Regression and the EM Algorithm......Page 631 12.6 Gaussian Mixture Models......Page 634 12.6.1 Gaussian Mixture Modeling and Clustering......Page 638 12.7 The EM Algorithm: a Lower Bound Maximization View......Page 641 12.8 Exponential Family of Probability Distributions......Page 645 12.8.1 The Exponential Family and the Maximum Entropy Method......Page 651 12.9.1 Mixing Linear Regression Models......Page 652 Hierarchical Mixture of Experts......Page 656 12.9.2 Mixing Logistic Regression Models......Page 657 Problems......Page 659 MATLAB® Exercises......Page 661 References......Page 663 13 Bayesian Learning: Approximate Inference and Nonparametric Models......Page 665 13.2 Variational Approximation in Bayesian Learning......Page 666 The Mean Field Approximation......Page 667 13.2.1 The Case of the Exponential Family of Probability Distributions......Page 671 13.3 A Variational Bayesian Approach to Linear Regression......Page 673 Computation of the Lower Bound......Page 678 13.4 A Variational Bayesian Approach to Gaussian Mixture Modeling......Page 679 13.5 When Bayesian Inference Meets Sparsity......Page 683 13.6 Sparse Bayesian Learning (SBL)......Page 685 13.6.1 The Spike and Slab Method......Page 689 13.7.1 Adopting the Logistic Regression Model for Classification......Page 690 13.8 Convex Duality and Variational Bounds......Page 694 13.9 Sparsity-Aware Regression: a Variational Bound Bayesian Path......Page 699 13.10 Expectation Propagation......Page 704 The Expectation Propagation Algorithm......Page 706 13.11 Nonparametric Bayesian Modeling......Page 708 13.11.1 The Chinese Restaurant Process......Page 709 13.11.2 Dirichlet Processes......Page 710 Predictive Distribution and the Pólya Urn Model......Page 713 13.11.3 The Stick Breaking Construction of a DP......Page 715 13.11.4 Dirichlet Process Mixture Modeling......Page 716 Inference......Page 717 13.11.5 The Indian Buffet Process......Page 719 Searching for a Prior on Infinite Binary Matrices......Page 721 Restaurant Construction......Page 724 Stick Breaking Construction......Page 726 Inference......Page 727 13.12 Gaussian Processes......Page 728 13.12.1 Covariance Functions and Kernels......Page 729 13.12.2 Regression......Page 730 Computational Considerations......Page 733 13.12.3 Classification......Page 734 13.13 A Case Study: Hyperspectral Image Unmixing......Page 735 13.13.1 Hierarchical Bayesian Modeling......Page 737 13.13.2 Experimental Results......Page 738 Problems......Page 739 MATLAB® Exercises......Page 744 References......Page 745 14.1 Introduction......Page 749 14.2 Monte Carlo Methods: the Main Concept......Page 750 14.2.1 Random Number Generation......Page 751 14.3 Random Sampling Based on Function Transformation......Page 753 14.4 Rejection Sampling......Page 757 14.5 Importance Sampling......Page 761 14.7 Markov Chain Monte Carlo Methods......Page 763 14.7.1 Ergodic Markov Chains......Page 766 14.8 The Metropolis Method......Page 772 14.8.1 Convergence Issues......Page 774 14.9 Gibbs Sampling......Page 776 14.10 In Search of More Efficient Methods: a Discussion......Page 778 14.11 A Case Study: Change-Point Detection......Page 780 Problems......Page 783 MATLAB® Exercise......Page 785 References......Page 786 15.1 Introduction......Page 788 15.2 The Need for Graphical Models......Page 789 15.3 Bayesian Networks and the Markov Condition......Page 791 15.3.1 Graphs: Basic Definitions......Page 792 15.3.2 Some Hints on Causality......Page 796 15.3.3 d-Separation......Page 798 15.3.4 Sigmoidal Bayesian Networks......Page 802 15.3.6 Multiple-Cause Networks......Page 803 15.3.7 I-Maps, Soundness, Faithfulness, and Completeness......Page 804 15.4 Undirected Graphical Models......Page 805 15.4.1 Independencies and I-Maps in Markov Random Fields......Page 807 15.4.2 The Ising Model and Its Variants......Page 808 15.4.3 Conditional Random Fields (CRFs)......Page 811 15.5 Factor Graphs......Page 812 15.5.1 Graphical Models for Error Correcting Codes......Page 814 15.6 Moralization of Directed Graphs......Page 815 15.7.1 Exact Inference in Chains......Page 816 15.7.2 Exact Inference in Trees......Page 820 15.7.3 The Sum-Product Algorithm......Page 821 15.7.4 The Max-Product and Max-Sum Algorithms......Page 826 Problems......Page 833 References......Page 835 16.1 Introduction......Page 838 16.2 Triangulated Graphs and Junction Trees......Page 839 16.2.1 Constructing a Join Tree......Page 842 16.2.2 Message Passing in Junction Trees......Page 844 16.3 Approximate Inference Methods......Page 847 Multiple-Cause Networks and the Noisy-OR Model......Page 848 The Boltzmann Machine......Page 850 16.3.2 Block Methods for Variational Approximation......Page 852 The Mean Field Approximation and the Boltzmann Machine......Page 853 16.3.3 Loopy Belief Propagation......Page 856 16.4 Dynamic Graphical Models......Page 859 16.5 Hidden Markov Models......Page 861 The Sum-Product Algorithm: the HMM Case......Page 864 16.5.2 Learning the Parameters in an HMM......Page 869 16.5.3 Discriminative Learning......Page 872 16.6.1 Factorial Hidden Markov Models......Page 873 16.7 Learning Graphical Models......Page 876 16.7.1 Parameter Estimation......Page 877 Problems......Page 881 References......Page 884 17.2 Sequential Importance Sampling......Page 887 17.2.1 Importance Sampling Revisited......Page 888 17.2.2 Resampling......Page 889 17.2.3 Sequential Sampling......Page 891 17.3.1 Kalman Filtering: a Bayesian Point of View......Page 894 17.4 Particle Filtering......Page 897 17.4.1 Degeneracy......Page 901 17.4.2 Generic Particle Filtering......Page 902 17.4.3 Auxiliary Particle Filtering......Page 905 Problems......Page 911 MATLAB® Exercises......Page 914 References......Page 915 18 Neural Networks and Deep Learning......Page 917 18.1 Introduction......Page 918 18.2 The Perceptron......Page 920 18.3 Feed-Forward Multilayer Neural Networks......Page 924 18.3.1 Fully Connected Networks......Page 928 18.4 The Backpropagation Algorithm......Page 929 Nonconvexity of the Cost Function......Page 930 18.4.1 The Gradient Descent Backpropagation Scheme......Page 932 Pattern-by-Pattern/Online Scheme......Page 938 Minibatch Schemes......Page 939 18.4.2 Variants of the Basic Gradient Descent Scheme......Page 940 Gradient Descent With a Momentum Term......Page 941 The AdaGrad Algorithm......Page 943 The RMSProp With Nesterov Momentum......Page 944 The Adaptive Moment Estimation Algorithm (Adam)......Page 945 Some Practical Hints......Page 946 Batch Normalization......Page 948 18.4.3 Beyond the Gradient Descent Rationale......Page 950 18.5 Selecting a Cost Function......Page 951 18.6 Vanishing and Exploding Gradients......Page 954 18.6.1 The Rectified Linear Unit......Page 955 18.7 Regularizing the Network......Page 956 Dropout......Page 959 18.8 Designing Deep Neural Networks: a Summary......Page 962 18.9 Universal Approximation Property of Feed-Forward Neural Networks......Page 963 18.10 Neural Networks: a Bayesian Flavor......Page 965 18.11 Shallow Versus Deep Architectures......Page 966 On the Representation Properties of Deep Networks......Page 967 Distributed Representations......Page 969 On the Optimization of Deep Networks: Some Theoretical Highlights......Page 970 On the Generalization Power of Deep Networks......Page 971 18.12.1 The Need for Convolutions......Page 972 The Convolution Step......Page 973 The Pooling Step......Page 979 18.12.2 Convolution Over Volumes......Page 981 Network in Network and 1x1 Convolution......Page 982 18.12.3 The Full CNN Architecture......Page 984 What Deep Neural Networks Learn......Page 985 18.12.4 CNNs: the Epilogue......Page 987 18.13 Recurrent Neural Networks......Page 992 18.13.1 Backpropagation Through Time......Page 994 Vanishing and Exploding Gradients......Page 995 The Long Short-Term Memory (LSTM) Network......Page 996 18.13.2 Attention and Memory......Page 998 18.14 Adversarial Examples......Page 1001 Adversarial Training......Page 1003 18.15.1 Restricted Boltzmann Machines......Page 1004 18.15.2 Pretraining Deep Feed-Forward Networks......Page 1007 18.15.3 Deep Belief Networks......Page 1008 18.15.4 Autoencoders......Page 1010 18.15.5 Generative Adversarial Networks......Page 1011 On the Optimality of the Solution......Page 1014 Problems in Training GANs......Page 1015 The Wasserstein GAN......Page 1017 Which Algorithm Then......Page 1019 18.15.6 Variational Autoencoders......Page 1020 18.16 Capsule Networks......Page 1023 Training......Page 1027 Transfer Learning......Page 1029 Multitask Learning......Page 1030 Geometric Deep Learning......Page 1031 Open Problems......Page 1032 18.18 A Case Study: Neural Machine Translation......Page 1033 18.19 Problems......Page 1039 Computer Exercises......Page 1041 References......Page 1045 19 Dimensionality Reduction and Latent Variable Modeling......Page 1055 19.1 Introduction......Page 1056 19.3 Principal Component Analysis......Page 1057 PCA, SVD, and Low Rank Matrix Factorization......Page 1059 PCA and Information Retrieval......Page 1061 Orthogonalizing Properties of PCA and Feature Generation......Page 1062 Latent Variables......Page 1063 19.4 Canonical Correlation Analysis......Page 1069 Partial Least-Squares......Page 1072 19.5.1 ICA and Gaussianity......Page 1074 19.5.2 ICA and Higher-Order Cumulants......Page 1075 ICA Ambiguities......Page 1076 19.5.3 Non-Gaussianity and Independent Components......Page 1077 19.5.4 ICA Based on Mutual Information......Page 1078 19.5.5 Alternative Paths to ICA......Page 1081 The Cocktail Party Problem......Page 1082 19.6 Dictionary Learning: the k-SVD Algorithm......Page 1085 Dictionary Learning and Dictionary Identifiability......Page 1088 19.7 Nonnegative Matrix Factorization......Page 1090 19.8 Learning Low-Dimensional Models: a Probabilistic Perspective......Page 1092 19.8.1 Factor Analysis......Page 1093 19.8.2 Probabilistic PCA......Page 1094 19.8.3 Mixture of Factors Analyzers: a Bayesian View to Compressed Sensing......Page 1098 19.9.1 Kernel PCA......Page 1101 Laplacian Eigenmaps......Page 1103 Local Linear Embedding (LLE)......Page 1107 Isometric Mapping (ISOMAP)......Page 1108 19.10.1 Matrix Completion......Page 1112 19.10.2 Robust PCA......Page 1116 Matrix Completion......Page 1117 Robust PCA/PCP......Page 1118 19.11 A Case Study: FMRI Data Analysis......Page 1119 MATLAB® Exercises......Page 1123 References......Page 1124 Index......Page 1132 Machine Learning: A Bayesian and Optimization Perspective, 2nd edition, gives a unified perspective on machine learning by covering both pillars of supervised learning, namely regression and classification. The book starts with the basics, including mean square, least squares and maximum likelihood methods, ridge regression, Bayesian decision theory classification, logistic regression, and decision trees. It then progresses to more recent techniques, covering sparse modelling methods, learning in reproducing kernel Hilbert spaces and support vector machines, Bayesian inference with a focus on the EM algorithm and its approximate inference variational versions, Monte Carlo methods, probabilistic graphical models focusing on Bayesian networks, hidden Markov models and particle filtering. Dimensionality reduction and latent variables modelling are also considered in depth. This palette of techniques concludes with an extended chapter on neural networks and deep learning architectures. The book also covers the fundamentals of statistical parameter estimation, Wiener and Kalman filtering, convexity and convex optimization, including a chapter on stochastic approximation and the gradient descent family of algorithms, presenting related online learning techniques as well as concepts and algorithmic versions for distributed optimization. Focusing on the physical reasoning behind the mathematics, without sacrificing rigor, all the various methods and techniques are explained in depth, supported by examples and problems, giving an invaluable resource to the student and researcher for understanding and applying machine learning concepts. Most of the chapters include typical case studies and computer exercises, both in MATLAB and Python. The chapters are written to be as self-contained as possible, making the text suitable for different courses: pattern recognition, statistical/adaptive signal processing, statistical/Bayesian learning, as well as courses on sparse modeling, deep learning, and probabilistic graphical models. New to this edition: Complete re-write of the chapter on Neural Networks and Deep Learning to reflect the latest advances since the 1st edition. The chapter, starting from the basic perceptron and feed-forward neural networks concepts, now presents an in depth treatment of deep networks, including recent optimization algorithms, batch normalization, regularization techniques such as the dropout method, convolutional neural networks, recurrent neural networks, attention mechanisms, adversarial examples and training, capsule networks and generative architectures, such as restricted Boltzman machines (RBMs), variational autoencoders and generative adversarial networks (GANs). Expanded treatment of Bayesian learning to include nonparametric Bayesian methods, with a focus on the Chinese restaurant and the Indian buffet processes. Presents the physical reasoning, mathematical modeling and algorithmic implementation of each method Updates on the latest trends, including sparsity, convex analysis and optimization, online distributed algorithms, learning in RKH spaces, Bayesian inference, graphical and hidden Markov models, particle filtering, deep learning, dictionary learning and latent variables modeling Provides case studies on a variety of topics, including protein folding prediction, optical character recognition, text authorship identification, fMRI data analysis, change point detection, hyperspectral image unmixing, target localization, and more Descripción del editor: "Machine Learning: A Bayesian and Optimization Perspective, 2nd edition, gives a unified perspective on machine learning by covering both pillars of supervised learning, namely regression and classification. The book starts with the basics, including mean square, least squares and maximum likelihood methods, ridge regression, Bayesian decision theory classification, logistic regression, and decision trees. It then progresses to more recent techniques, covering sparse modelling methods, learning in reproducing kernel Hilbert spaces and support vector machines, Bayesian inference with a focus on the EM algorithm and its approximate inference variational versions, Monte Carlo methods, probabilistic graphical models focusing on Bayesian networks, hidden Markov models and particle filtering. Dimensionality reduction and latent variables modelling are also considered in depth.This palette of techniques concludes with an extended chapter on neural networks and deep learning architectures. The book also covers the fundamentals of statistical parameter estimation, Wiener and Kalman filtering, convexity and convex optimization, including a chapter on stochastic approximation and the gradient descent family of algorithms, presenting related online learning techniques as well as concepts and algorithmic versions for distributed optimization.Focusing on the physical reasoning behind the mathematics, without sacrificing rigor, all the various methods and techniques are explained in depth, supported by examples and problems, giving an invaluable resource to the student and researcher for understanding and applying machine learning concepts. Most of the chapters include typical case studies and computer exercises, both in MATLAB and Python.The chapters are written to be as self-contained as possible, making the text suitable for different courses: pattern recognition, statistical/adaptive signal processing, statistical/Bayesian learning, as well as courses on sparse modeling, deep learning, and probabilistic graphical models.New to this edition: Complete re-write of the chapter on Neural Networks and Deep Learning to reflect the latest advances since the 1st edition. The chapter, starting from the basic perceptron and feed-forward neural networks concepts, now presents an in depth treatment of deep networks, including recent optimization algorithms, batch normalization, regularization techniques such as the dropout method, convolutional neural networks, recurrent neural networks, attention mechanisms, adversarial examples and training, capsule networks and generative architectures, such as restricted Boltzman machines (RBMs), variational autoencoders and generative adversarial networks (GANs). Expanded treatment of Bayesian learning to include nonparametric Bayesian methods, with a focus on the Chinese restaurant and the Indian buffet processes." (Elsevier) Machine A Bayesian and Optimization Perspective, 2 nd edition , gives a unified perspective on machine learning by covering both pillars of supervised learning, namely regression and classification. The book starts with the basics, including mean square, least squares and maximum likelihood methods, ridge regression, Bayesian decision theory classification, logistic regression, and decision trees. It then progresses to more recent techniques, covering sparse modelling methods, learning in reproducing kernel Hilbert spaces and support vector machines, Bayesian inference with a focus on the EM algorithm and its approximate inference variational versions, Monte Carlo methods, probabilistic graphical models focusing on Bayesian networks, hidden Markov models and particle filtering. Dimensionality reduction and latent variables modelling are also considered in depth. This palette of techniques concludes with an extended chapter on neural networks and deep learning architectures. The book also covers the fundamentals of statistical parameter estimation, Wiener and Kalman filtering, convexity and convex optimization, including a chapter on stochastic approximation and the gradient descent family of algorithms, presenting related online learning techniques as well as concepts and algorithmic versions for distributed optimization. Focusing on the physical reasoning behind the mathematics, without sacrificing rigor, all the various methods and techniques are explained in depth, supported by examples and problems, giving an invaluable resource to the student and researcher for understanding and applying machine learning concepts. Most of the chapters include typical case studies and computer exercises, both in MATLAB and Python. The chapters are written to be as self-contained as possible, making the text suitable for different pattern recognition, statistical/adaptive signal processing, statistical/Bayesian learning, as well as courses on sparse modeling, deep learning, and probabilistic graphical models. New to this This tutorial text gives a unifying perspective on machine learning by covering both probabilistic and deterministic approaches -which are based on optimization techniques - together with the Bayesian inference approach, whose essence lies in the use of a hierarchy of probabilistic models. The book presents the major machine learning methods as they have been developed in different disciplines, such as statistics, statistical and adaptive signal processing and computer science. Focusing on the physical reasoning behind the mathematics, all the various methods and techniques are explained in depth, supported by examples and problems, giving an invaluable resource to the student and researcher for understanding and applying machine learning concepts. The book builds carefully from the basic classical methods to the most recent trends, with chapters written to be as self-contained as possible, making the text suitable for different courses: pattern recognition, statistical/adaptive signal processing, statistical/Bayesian learning, as well as short courses on sparse modeling, deep learning, and probabilistic graphical models. All major classical techniques: Mean/Least-Squares regression and filtering, Kalman filtering, stochastic approximation and online learning, Bayesian classification, decision trees, logistic regression and boosting methods. The latest trends: Sparsity, convex analysis and optimization, online distributed algorithms, learning in RKH spaces, Bayesian inference, graphical and hidden Markov models, particle filtering, deep learning, dictionary learning and latent variables modeling