Advanced Analytics with Spark : Patterns for Learning From Data at Scale

"In this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example. You'll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniques-classification, collaborative filtering, and anomaly detection among others-to fields such as genomics, security, and finance. If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, you'll find these patterns useful for working on your own data applications. Patterns include: Recommending music and the Audioscrobbler data set, Predicting forest cover with decision trees, Anomaly detection in network traffic with K-means clustering, Understanding Wikipedia with Latent Semantic Analysis, Analyzing co-occurrence networks with GraphX, Geospatial and temporal data analysis on the New York City Taxi Trips data, Estimating financial risk through Monte Carlo simulation, Analyzing genomics data and the BDG project and Analyzing neuroimaging data with PySpark and Thunder." from publisher's website Table of Contents 5 Foreword 9 Preface 11 What鈥檚 in This Book 12 Using Code Examples 12 Safari庐 Books Online 12 How to Contact Us 13 Acknowledgments 13 Chapter 1. Analyzing Big Data 15 The Challenges of Data Science 17 Introducing Apache Spark 18 About This Book 20 Chapter 2. Introduction to Data Analysis with Scala and Spark 23 Scala for Data Scientists 24 The Spark Programming Model 25 Record Linkage 25 Getting Started: The Spark Shell and SparkContext 27 Bringing Data from the Cluster to the Client 32 Shipping Code from the Client to the Cluster 36 Structuring Data with Tuples and Case Classes 37 Aggregations 42 Creating Histograms 43 Summary Statistics for Continuous Variables 44 Creating Reusable Code for Computing Summary Statistics 45 Simple Variable Selection and Scoring 50 Where to Go from Here 51 Chapter 3. Recommending Music and the Audioscrobbler Data Set 53 Data Set 54 The Alternating Least Squares Recommender Algorithm 55 Preparing the Data 57 Building a First Model 60 Spot Checking Recommendations 62 Evaluating Recommendation Quality 64 Computing AUC 65 Hyperparameter Selection 67 Making Recommendations 69 Where to Go from Here 70 Chapter 4. Predicting Forest Cover with Decision Trees 73 Fast Forward to Regression 73 Vectors and Features 74 Training Examples 75 Decision Trees and Forests 76 Covtype Data Set 79 Preparing the Data 80 A First Decision Tree 81 Decision Tree Hyperparameters 85 Tuning Decision Trees 87 Categorical Features Revisited 89 Random Decision Forests 91 Making Predictions 93 Where to Go from Here 93 Chapter 5. Anomaly Detection in Network Traffic with K-means Clustering 95 Anomaly Detection 96 K-means Clustering 96 Network Intrusion 97 KDD Cup 1999 Data Set 98 A First Take on Clustering 99 Choosing k 101 Visualization in R 103 Feature Normalization 105 Categorical Variables 108 Using Labels with Entropy 109 Clustering in Action 110 Where to Go from Here 111 Chapter 6. Understanding Wikipedia with Latent Semantic Analysis 113 The Term-Document Matrix 114 Getting the Data 116 Parsing and Preparing the Data 116 Lemmatization 118 Computing the TF-IDFs 119 Singular Value Decomposition 121 Finding Important Concepts 123 Querying and Scoring with the Low-Dimensional Representation 126 Term-Term Relevance 127 Document-Document Relevance 129 Term-Document Relevance 130 Multiple-Term Queries 131 Where to Go from Here 133 Chapter 7. Analyzing Co-occurrence Networks with GraphX 135 The MEDLINE Citation Index: A Network Analysis 136 Getting the Data 137 Parsing XML Documents with Scala鈥檚 XML Library 139 Analyzing the MeSH Major Topics and Their Co-occurrences 141 Constructing a Co-occurrence Network with GraphX 143 Understanding the Structure of Networks 146 Connected Components 146 Degree Distribution 149 Filtering Out Noisy Edges 152 Processing EdgeTriplets 153 Analyzing the Filtered Graph 154 Small-World Networks 156 Cliques and Clustering Coefficients 157 Computing Average Path Length with Pregel 158 Where to Go from Here 163 Chapter 8. Geospatial and Temporal Data Analysis on the New York City Taxi Trip Data 165 Getting the Data 166 Working with Temporal and Geospatial Data in Spark 167 Temporal Data with JodaTime and NScalaTime 167 Geospatial Data with the Esri Geometry API and Spray 169 Exploring the Esri Geometry API 169 Intro to GeoJSON 171 Preparing the New York City Taxi Trip Data 173 Handling Invalid Records at Scale 174 Geospatial Analysis 178 Sessionization in Spark 181 Building Sessions: Secondary Sorts in Spark 182 Where to Go from Here 185 Chapter 9. Estimating Financial Risk through Monte Carlo Simulation 187 Terminology 188 Methods for Calculating VaR 189 Variance-Covariance 189 Historical Simulation 189 Monte Carlo Simulation 189 Our Model 190 Getting the Data 191 Preprocessing 192 Determining the Factor Weights 195 Sampling 197 The Multivariate Normal Distribution 199 Running the Trials 200 Visualizing the Distribution of Returns 203 Evaluating Our Results 204 Where to Go from Here 206 Chapter 10. Analyzing Genomics Data and the BDG Project 209 Decoupling Storage from Modeling 210 Ingesting Genomics Data with the ADAM CLI 212 Parquet Format and Columnar Storage 218 Predicting Transcription Factor Binding Sites from ENCODE Data 220 Querying Genotypes from the 1000 Genomes Project 227 Where to Go from Here 228 Chapter 11. Analyzing Neuroimaging Data with PySpark and Thunder 231 Overview of PySpark 232 PySpark Internals 233 Overview and Installation of the Thunder Library 235 Loading Data with Thunder 236 Thunder Core Data Types 243 Categorizing Neuron Types with Thunder 245 Where to Go from Here 250 Appendix A. Deeper into Spark 251 Serialization 253 Accumulators 253 Spark and the Data Scientist鈥檚 Workflow 254 File Formats 256 Spark Subprojects 257 MLlib 257 Spark Streaming 258 Spark SQL 259 GraphX 259 Appendix B. Upcoming MLlib Pipelines API 261 Beyond Mere Modeling 261 The Pipelines API 262 Text Classification Example Walkthrough 264 Index 267 About the Authors 275 In this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example.You’ll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniques—classification, collaborative filtering, and anomaly detection among others—to fields such as genomics, security, and finance. If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, you’ll find these patterns useful for working on your own data applications.Patterns include:Recommending music and the Audioscrobbler data setPredicting forest cover with decision treesAnomaly detection in network traffic with K-means clusteringUnderstanding Wikipedia with Latent Semantic AnalysisAnalyzing co-occurrence networks with GraphXGeospatial and temporal data analysis on the New York City Taxi Trips dataEstimating financial risk through Monte Carlo simulationAnalyzing genomics data and the BDG projectAnalyzing neuroimaging data with PySpark and Thunder In this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example. You{u2019}ll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniques{u2014}classification, collaborative filtering, and anomaly detection among others{u2014}to fields such as genomics, security, and finance. If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, you{u2019}ll find these patterns useful for working on your own data applications. Patterns include: Recommending music and the Audioscrobbler data set Predicting forest cover with decision trees Anomaly detection in network traffic with K-means clustering Understanding Wikipedia with Latent Semantic Analysis Analyzing co-occurrence networks with GraphX Geospatial and temporal data analysis on the New York City Taxi Trips data Estimating financial risk through Monte Carlo simulation Analyzing genomics data and the BDG project Analyzing neuroimaging data with PySpark and Thunder In the second edition of this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example. Updated for Spark 2.1, this edition acts as an introduction to these techniques and other best practices in Spark programming. Youll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniquesincluding classification, clustering, collaborative filtering, and anomaly detectionto fields such as genomics, security, and finance. If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, youll find the books patterns useful for working on your own data applications. With this book, you