آپاچی هادوپ یارن: فراتر رفتن از مپ‌کاهش و پردازش دسته‌ای با آپاچی هادوپ ۲

__“This book is a critically needed resource for the newly released Apache Hadoop 2.0, highlighting YARN as the significant breakthrough that broadens Hadoop beyond the MapReduce paradigm.”__ —From the Foreword by **Raymie Stata, CEO of Altiscale** **The Insider’s Guide to Building Distributed, Big Data Applications with Apache HadoopTM YARN** Apache Hadoop is helping drive the Big Data revolution. Now, its data processing has been completely overhauled: Apache Hadoop YARN provides resource management at data center scale and easier ways to create distributed applications that process petabytes of data. And now in **__**Apache HadoopTM YARN,**__** two Hadoop technical leaders show you how to develop new applications and adapt existing code to fully leverage these revolutionary advances. YARN project founder Arun Murthy and project lead Vinod Kumar Vavilapalli demonstrate how YARN increases scalability and cluster utilization, enables new programming models and services, and opens new options beyond Java and batch processing. They walk you through the entire YARN project lifecycle, from installation through deployment. You’ll find many examples drawn from the authors’ cutting-edge experience—first as Hadoop’s earliest developers and implementers at Yahoo! and now as Hortonworks developers moving the platform forward and helping customers succeed with it. Coverage includes * YARN’s goals, design, architecture, and components—how it expands the Apache Hadoop ecosystem * Exploring YARN on a single node * Administering YARN clusters and Capacity Scheduler * Running existing MapReduce applications * Developing a large-scale clustered YARN application * Discovering new open source frameworks that run under YARN Contents 6 Foreword by Raymie Stata 14 Foreword by Paul Dix 16 Preface 18 Acknowledgments 22 About the Authors 26 1 Apache Hadoop YARN: A Brief History and Rationale 28 Introduction 28 Apache Hadoop 29 Phase 0: The Era of Ad Hoc Clusters 30 Phase 1: Hadoop on Demand 30 HDFS in the HOD World 32 Features and Advantages of HOD 33 Shortcomings of Hadoop on Demand 34 Phase 2: Dawn of the Shared Compute Clusters 36 Evolution of Shared Clusters 36 Issues with Shared MapReduce Clusters 42 Phase 3: Emergence of YARN 45 Conclusion 47 2 Apache Hadoop YARN Install Quick Start 48 Getting Started 49 Steps to Configure a Single-Node YARN Cluster 49 Step 1: Download Apache Hadoop 49 Step 2: Set JAVA_HOME 50 Step 3: Create Users and Groups 50 Step 4: Make Data and Log Directories 50 Step 5: Configure core-site.xml 51 Step 6: Configure hdfs-site.xml 51 Step 7: Configure mapred-site.xml 52 Step 8: Configure yarn-site.xml 52 Step 9: Modify Java Heap Sizes 53 Step 10: Format HDFS 53 Step 11: Start the HDFS Services 54 Step 12: Start YARN Services 55 Step 13: Verify the Running Services Using the Web Interface 55 Run Sample MapReduce Examples 57 Wrap-up 58 3 Apache Hadoop YARN Core Concepts 60 Beyond MapReduce 60 The MapReduce Paradigm 62 Apache Hadoop MapReduce 62 The Need for Non-MapReduce Workloads 64 Addressing Scalability 64 Improved Utilization 65 User Agility 65 Apache Hadoop YARN 65 YARN Components 66 ResourceManager 66 ApplicationMaster 67 Resource Model 68 ResourceRequests and Containers 68 Container Specification 69 Wrap-up 69 4 Functional Overview of YARN Components 70 Architecture Overview 70 ResourceManager 72 YARN Scheduling Components 73 FIFO Scheduler 73 Capacity Scheduler 74 Fair Scheduler 74 Containers 76 NodeManager 76 ApplicationMaster 77 YARN Resource Model 77 Client Resource Request 78 ApplicationMaster Container Allocation 78 ApplicationMaster–Container Manager Communication 79 Managing Application Dependencies 80 LocalResources Definitions 81 LocalResource Timestamps 82 LocalResource Types 82 LocalResource Visibilities 83 Lifetime of LocalResources 84 Wrap-up 84 5 Installing Apache Hadoop YARN 86 The Basics 86 System Preparation 87 Step 1: Install EPEL and pdsh 87 Step 2: Generate and Distribute ssh Keys 88 Script-based Installation of Hadoop 2 89 JDK Options 89 Step 1: Download and Extract the Scripts 90 Step 2: Set the Script Variables 90 Step 3: Provide Node Names 91 Step 4: Run the Script 91 Step 5: Verify the Installation 92 Script-based Uninstall 95 Configuration File Processing 95 Configuration File Settings 95 core-site.xml 95 hdfs-site.xml 96 mapred-site.xml 96 yarn-site.xml 97 Start-up Scripts 98 Installing Hadoop with Apache Ambari 98 Performing an Ambari-based Hadoop Installation 99 Step 1: Check Requirements 100 Step 2: Install the Ambari Server 100 Step 3: Install and Start Ambari Agents 100 Step 4: Start the Ambari Server 101 Step 5: Install an HDP2.X Cluster 102 Wrap-up 111 6 Apache Hadoop YARN Administration 112 Script-based Configuration 112 Monitoring Cluster Health: Nagios 117 Monitoring Basic Hadoop Services 119 Monitoring the JVM 122 Real-time Monitoring: Ganglia 124 Administration with Ambari 126 JVM Analysis 130 Basic YARN Administration 133 YARN Administrative Tools 133 Adding and Decommissioning YARN Nodes 134 Capacity Scheduler Configuration 135 YARN WebProxy 135 Using the JobHistoryServer 135 Refreshing User-to-Groups Mappings 135 Refreshing Superuser Proxy Groups Mappings 136 Refreshing ACLs for Administration of ResourceManager 136 Reloading the Service-level Authorization Policy File 136 Managing YARN Jobs 136 Setting Container Memory 137 Setting Container Cores 137 Setting MapReduce Properties 137 User Log Management 138 Wrap-up 141 7 Apache Hadoop YARN Architecture Guide 142 Overview 142 ResourceManager 144 Overview of the ResourceManager Components 145 Client Interaction with the ResourceManager 145 Application Interaction with the ResourceManager 147 Interaction of Nodes with the ResourceManager 148 Core ResourceManager Components 149 Security-related Components in the ResourceManager 151 NodeManager 154 Overview of the NodeManager Components 155 NodeManager Components 156 NodeManager Security Components 163 Important NodeManager Functions 164 ApplicationMaster 165 Overview 165 Liveliness 166 Resource Requirements 167 Scheduling 167 Scheduling Protocol and Locality 169 Launching Containers 172 Completed Containers 173 ApplicationMaster Failures and Recovery 173 Coordination and Output Commit 173 Information for Clients 174 Security 174 Cleanup on ApplicationMaster Exit 174 YARN Containers 175 Container Environment 175 Communication with the ApplicationMaster 176 Summary for Application-writers 177 Wrap-up 178 8 Capacity Scheduler in YARN 180 Introduction to the Capacity Scheduler 180 Elasticity with Multitenancy 181 Security 181 Resource Awareness 181 Granular Scheduling 181 Locality 182 Scheduling Policies 182 Capacity Scheduler Configuration 182 Queues 183 Hierarchical Queues 183 Key Characteristics 184 Scheduling Among Queues 184 Defining Hierarchical Queues 185 Queue Access Control 186 Capacity Management with Queues 187 User Limits 190 Reservations 193 State of the Queues 194 Limits on Applications 195 User Interface 196 Wrap-up 196 9 MapReduce with Apache Hadoop YARN 198 Running Hadoop YARN MapReduce Examples 198 Listing Available Examples 198 Running the Pi Example 199 Using the Web GUI to Monitor Examples 201 Running the Terasort Test 207 Run the TestDFSIO Benchmark 207 MapReduce Compatibility 208 The MapReduce ApplicationMaster 208 Enabling Application Master Restarts 209 Enabling Recovery of Completed Tasks 209 The JobHistory Server 209 Calculating the Capacity of a Node 209 Changes to the Shuffle Service 211 Running Existing Hadoop Version 1 Applications 211 Binary Compatibility of org.apache.hadoop.mapred APIs 211 Source Compatibility of org.apache.hadoop. mapreduce APIs 212 Compatibility of Command-line Scripts 212 Compatibility Tradeoff Between MRv1 and Early MRv2 (0.23.x) Applications 212 Running MapReduce Version 1 Existing Code 214 Running Apache Pig Scripts on YARN 214 Running Apache Hive Queries on YARN 214 Running Apache Oozie Workflows on YARN 215 Advanced Features 215 Uber Jobs 215 Pluggable Shuffle and Sort 215 Wrap-up 217 10 Apache Hadoop YARN Application Example 218 The YARN Client 218 The ApplicationMaster 235 Wrap-up 253 11 Using Apache Hadoop YARN Distributed-Shell 254 Using the YARN Distributed-Shell 254 A Simple Example 255 Using More Containers 256 Distributed-Shell Examples with Shell Arguments 257 Internals of the Distributed-Shell 259 Application Constants 259 Client 260 ApplicationMaster 263 Final Containers 267 Wrap-up 267 12 Apache Hadoop YARN Frameworks 268 Distributed-Shell 268 Hadoop MapReduce 268 Apache Tez 269 Apache Giraph 269 Hoya: HBase on YARN 270 Dryad on YARN 270 Apache Spark 271 Apache Storm 271 REEF: Retainable Evaluator Execution Framework 272 Hamster: Hadoop and MPI on the Same Cluster 272 Wrap-up 272 A: Supplemental Content and Code Downloads 274 Available Downloads 274 B: YARN Installation Scripts 276 install-hadoop2.sh 276 uninstall-hadoop2.sh 283 hadoop-xml-conf.sh 285 C: YARN Administration Scripts 290 configure-hadoop2.sh 290 D: Nagios Modules 296 check_resource_manager.sh 296 check_data_node.sh 298 check_resource_manager_old_space_pct.sh 299 E: Resources and Additional Information 304 F: HDFS Quick Reference 306 Quick Command Reference 306 Starting HDFS and the HDFS Web GUI 307 Get an HDFS Status Report 307 Perform an FSCK on HDFS 308 General HDFS Commands 308 List Files in HDFS 309 Make a Directory in HDFS 310 Copy Files to HDFS 310 Copy Files from HDFS 311 Copy Files within HDFS 311 Delete a File within HDFS 311 Delete a Directory in HDFS 311 Decommissioning HDFS Nodes 311 Index 314 A 314 B 316 C 316 D 318 E 318 F 319 G 319 H 319 I 320 J 321 K 322 L 322 M 323 N 324 O 324 P 325 Q 326 R 326 S 327 T 329 U 329 V 329 W 329 X 330 Y 330 Z 331 www.it-ebooks.info "This book is a critically needed resource for the newly released Apache Hadoop 2.0, highlighting YARN as the significant breakthrough that broadens Hadoop beyond the MapReduce paradigm." - From the Foreword by Raymie Stata, CEO of Altiscale The Insider's Guide to Building Distributed, Big Data Applications with Apache Hadoop YARN Apache Hadoop is helping drive the Big Data revolution. Now, its data processing has been completely overhauled: Apache Hadoop YARN provides resource management at data center scale and easier ways to create distributed applications that process petabytes of data. And now in Apache Hadoop YARN, two Hadoop technical leaders show you how to develop new applications and adapt existing code to fully leverage these revolutionary advances. YARN project founder Arun Murthy and project lead Vinod Kumar Vavilapalli demonstrate how YARN increases scalability and cluster utilization, enables new programming models and services, and opens new options beyond Java and batch processing. They walk you through the entire YARN project lifecycle, from installation through deployment. You'll find many examples drawn from the authors' cutting-edge experience - first as Hadoop's earliest developers and implementers at Yahoo! and now as Hortonworks developers moving the platform forward and helping customers succeed with it. Coverage includes YARN's goals, design, architecture, and components - how it expands the Apache Hadoop ecosystem Exploring YARN on a single node Administering YARN clusters and Capacity Scheduler Running existing MapReduce applications Developing a large-scale clustered YARN application Discovering new open source frameworks that run under YARN