Beginning Apache Spark 2 : With Resilient Distributed Datasets, Spark SQL, Structured Streaming and Spark Machine Learning Library

Develop applications for the big data landscape with Spark and Hadoop. This book also explains the role of Spark in developing scalable machine learning and analytics applications with Cloud technologies. Beginning Apache Spark 2 gives you an introduction to Apache Spark and shows you how to work with it. Along the way, you’ll discover resilient distributed datasets (RDDs); use Spark SQL for structured data; and learn stream processing and build real-time applications with Spark Structured Streaming. Furthermore, you’ll learn the fundamentals of Spark ML for machine learning and much more. After you read this book, you will have the fundamentals to become proficient in using Apache Spark and know when and how to apply it to your big data applications. What You Will Learn • Understand Spark unified data processing platform • How to run Spark in Spark Shell or Databricks • Use and manipulate RDDs • Deal with structured data using Spark SQL through its operations and advanced functions • Build real-time applications using Spark Structured Streaming • Develop intelligent applications with the Spark Machine Learning library Who This Book Is For Programmers and developers active in big data, Hadoop, and Java but who are new to the Apache Spark platform. Table of Contents 4 About the Author 9 About the Technical Reviewer 10 Chapter 1: Introduction to Apache Spark 11 Overview 11 History 12 Spark Core Concepts and Architecture 13 Spark Clusters and the Resource Management System 14 Spark Application 15 Spark Driver and Executor 15 Spark Unified Stack 16 Spark Core 17 Spark SQL 18 Spark Structured Streaming and Streaming 19 Spark MLlib 20 Spark Graphx 21 SparkR 21 Apache Spark Applications 22 Example Spark Application 22 Summary 23 Chapter 2: Working with Apache Spark 24 Downloading and Installing Spark 24 Downloading Spark 24 Installing Spark 25 Spark Scala Shell 26 Spark Python Shell 27 Having Fun with the Spark Scala Shell 28 Useful Spark Scala Shell Commands and Tips 28 Basic Interactions with Scala and Spark 31 Basic Interactions with Scala 31 Spark UI and Basic Interactions with Spark 34 Spark UI 35 Basic Interactions with Spark 37 Introduction to Databricks 39 Creating a Cluster 42 Creating a Folder 45 Creating a Notebook 48 Setting Up the Spark Source Code 56 Summary 58 Chapter 3: Resilient Distributed Datasets 59 Introduction to RDDs 59 Immutable 60 Fault Tolerant 60 Parallel Data Structures 60 In-Memory Computing 61 Data Partitioning and Placement 61 Rich Set of Operations 62 RDD Operations 62 Creating RDDs 64 Transformations 65 Transformation Examples 66 map(func) 66 flatMap(func) 68 filter(func) 69 mapPartitions(func)/mapPartitionsWithIndex(index, func) 70 union(otherRDD) 73 intersection(otherRDD) 73 substract(otherRDD) 74 distinct() 75 sample(withReplacement, fraction, seed) 75 Actions 76 Action Examples 77 collect() 77 count() 78 first() 78 take(n) 79 reduce(func) 79 takeSample(withReplacement, n, [seed]) 81 takeOrdered(n, [ordering]) 81 top(n, [ordering]) 82 saveAsTextFile(path) 82 Working with Key/Value Pair RDD 82 Creating Key/Value Pair RDD 83 Key/Value Pair RDD Transformations 84 groupByKey([numTasks]) 85 reduceByKey(func, [numTasks]) 86 sortByKey([ascending],[numTasks]) 87 join(otherRDD) 88 Key/Value Pair RDD Actions 89 countByKey() 89 collectAsMap() 90 lookup(key) 90 Understand Data Shuffling 91 Having Fun with RDD Persistence 91 Summary 93 Chapter 4: Spark SQL (Foundations) 95 Introduction to DataFrames 96 Creating DataFrames 97 Creating DataFrames from RDDs 97 Creating DataFrames from a Range of Numbers 100 Creating DataFrames from Data Sources 103 Creating DataFrames by Reading Text Files 105 Creating DataFrames by Reading CSV Files 106 Creating DataFrames by Reading JSON Files 109 Creating DataFrames by Reading Parquet Files 112 Creating DataFrames by Reading ORC Files 113 Creating DataFrames from JDBC 114 Working with Structured Operations 117 Working with Columns 119 Working with Structured Transformations 121 select(columns) 121 selectExpr(expressions) 123 filler(condition), where(condition) 124 distinct, dropDuplicates 126 sort(columns), orderBy(columns) 126 limit(n) 128 union(otherDataFrame) 129 withColumn(colName, column) 130 withColumnRenamed(existingColName, newColName) 131 drop(columnName1, columnName2) 132 sample(fraction), sample(fraction, seed), sample(fraction, seed, withReplacement) 133 randomSplit(weights) 134 Working with Missing or Bad Data 134 describe(columnNames) 137 Working with Structured Actions 137 Introduction to Datasets 138 Creating Datasets 140 Working with Datasets 141 Using SQL in Spark SQL 143 Running SQL in Spark 144 Writing Data Out to Storage Systems 147 The Trio: DataFrames, Datasets, and SQL 150 DataFrame Persistence 151 Summary 152 Chapter 5: Spark SQL (Advanced) 154 Aggregations 154 Aggregation Functions 155 Common Aggregation Functions 155 count(col) 157 countDistinct(col) 158 approx_count_distinct (col, max_estimated_error=0.05) 159 min(col), max(col) 160 sum(col) 161 sumDistinct(col) 161 avg(col) 161 skewness(col), kurtosis(col) 162 variance(col), stddev(col) 163 Aggregation with Grouping 163 Multiple Aggregations per Group 165 Collection Group Values 167 Aggregation with Pivoting 168 Joins 170 Join Expressions and Join Types 171 Working with Joins 172 Inner Joins 173 Left Outer Joins 175 Right Outer Joins 175 Outer Joins (aka Full Outer Joins) 176 Left Anti-Joins 177 Left Semi-Joins 178 Cross (aka Cartesian) 179 Dealing with Duplicate Column Names 180 Use the Original DataFrame 181 Renaming Column Before Joining 181 Using a Joined Column Name 182 Overview of a Join Implementation 182 Shuffle Hash Join 182 Broadcast Hash Join 183 Functions 185 Working with Built-in Functions 185 Working with Date-Time Functions 186 Working with String Functions 190 Working with Math Functions 193 Working with Collection Functions 194 Working with Miscellaneous Functions 197 Working with User-Defined Functions 201 Advanced Analytics Functions 203 Aggregation with Rollups and Cubes 203 Rollups 204 Cube 206 Aggregation with Time Windows 207 Window Functions 210 Catalyst Optimizer 218 Logical Plan 219 Physical Plan 220 Catalyst in Action 220 Project Tungsten 222 Summary 223 Chapter 6: Spark Streaming 225 Stream Processing 226 Concepts 228 Data Delivery Semantics 229 Notion of Time 230 Windowing 232 Stream Processing Engine Landscape 233 Spark Streaming Overview 236 Spark DStream 236 Spark Structured Streaming 238 Overview 239 Core Concepts 241 Data Sources 242 Output Modes 243 Trigger Types 244 Data Sinks 245 Watermarking 247 Structured Streaming Application 248 Streaming DataFrame Operations 254 Selection, Project, and Aggregation Operations 254 Join Operations 255 Working with Data Sources 257 Working with the Socket Data Source 257 Working with the Rate Data Source 259 Working with the File Data Source 262 Working with the Kafka Data Source 263 Working with the Custom Data Source 268 Working with Data Sinks 270 Working with the File Data Sink 270 Working with the Kafka Data Sink 272 Working with the Foreach Data Sink 275 Working with the Console Data Sink 278 Working with the Memory Data Sink 279 Deep Dive on Output Modes 281 Deep Dive on Triggers 286 Summary 291 Chapter 7: Spark Streaming (Advanced) 293 Event Time 293 Fixed Window Aggregation Over an Event Time 295 Sliding Window Aggregation Over an Event Time 297 Aggregation State 301 Watermarking: Limit State and Handle Late Data 302 Arbitrary Stateful Processing 306 Arbitrary Stateful Processing with Structured Streaming 306 Handling State Timeouts 309 Arbitrary State Processing in Action 310 Extracting Patterns with mapGroupsWithState 311 User Sessionization with flatMapGroupsWithState 316 Handling Duplicate Data 322 Fault Tolerance 325 Streaming Application Code Change 326 Spark Runtime Change 326 Streaming Query Metrics and Monitoring 326 Streaming Query Metrics 327 Monitoring Streaming Queries 330 Summary 331 Chapter 8: Machine Learning with Spark 333 Machine Learning Overview 335 Machine Learning Terminologies 336 Machine Learning Types 337 Supervised Learning 338 Unsupervised Learning 340 Reinforcement Learning 341 Machine Learning Process 341 Spark Machine Learning Library 344 Machine Learning Pipelines 344 Transformers 345 Estimators 354 Pipeline 365 Model Tuning 367 Machine Learning Tasks in Action 373 Classification 373 Model Hyperparameters 374 Example 374 Regression 376 Model Hyperparameters 377 Example 377 Recommendation 380 Model Hyperparameters 380 Example 381 Deep Learning Pipeline 387 Summary 389 Index 390 Front Matter ....Pages i-xi Introduction to Apache Spark (Hien Luu)....Pages 1-13 Working with Apache Spark (Hien Luu)....Pages 15-49 Resilient Distributed Datasets (Hien Luu)....Pages 51-86 Spark SQL (Foundations) (Hien Luu)....Pages 87-145 Spark SQL (Advanced) (Hien Luu)....Pages 147-217 Spark Streaming (Hien Luu)....Pages 219-286 Spark Streaming (Advanced) (Hien Luu)....Pages 287-326 Machine Learning with Spark (Hien Luu)....Pages 327-383 Back Matter ....Pages 385-393 A tutorial on the Apache Spark platform written by an expert engineer and trainer, this book will give you the fundamentals to become proficient in using Apache Spark and know when and how to apply it to your big data applications. -- Edited summary from book