Big data using Hadoop(tm) and Hive(tm)

This book is the basic guide for developers,architects, engineers, and anyone who wants to start leveraging the open-sourcesoftware Hadoop and Hive to build distributed, scalable concurrent big data applications. Hive will be used for reading, writing, and managing the large, data set files. The book is a concise guide on getting started with an overall understanding onApache Hadoop and Hive and how they work together to speed up development with minimal effort. It will refer to simple concepts and examples, as they are likely to be the best teaching aids. It will explain the logic, code, and configurations needed to build a successful, distributed, concurrent application, as well as the reason behind those decisions. FEATURES: Shows how to leverage the open-source software Hadoop and Hive to build distributed, scalable, concurrent big data applications Includes material on Hive architecture with various storage types and the Hive query language Features a chapter on big data and how Hadoop can be used to solve the changes around it Explains the basic Hadoop setup, configuration, and optimization Cover Half-Title Title Copyright Dedication Contents Preface Chapter 1: Big Data Big Data Challenges for Organizations How We Are Using Big Data Big Data: An Opportunity Hadoop: A Big Data Solution Big Data in the Real World Chapter 2: What is Apache Hadoop? Hadoop History Hadoop Benefits Hadoop's Ecosystem: Components Hadoop Core Component Architecture Summary Chapter 3: The Hadoop Distribution Filesystem HDFS Core Components HDFS Architecture Data Replication Data Locality Data Storage Failure Handling on the HDFS Erasure Coding (EC) HDFS Disk Balancer HDFS Federation HDFS Architecture and Its Challenges Hadoop Federation: A Rescue Benefits of the HDFS Federation HDFS Processes: Read and Write Failure Handling During Read and Write Chapter 4: Getting Started with Hadoop Hadoop Configuration Command-Line Interface Generic Filesystem CLI Command Distributed Copy (distcp) Hadoop's Other User Commands HDFS Permissions HDFS Quotas Guide HDFS Short-Circuit Local Reads Offline Edits Viewer Guide Offline Image Viewer Guide Chapter 5: Interfaces to Access HDFS Files WebHDFS REST API FileSystem URIs Error Responses Authentication Java FileSystem API URI and Path FSDataInputStream FSDataOutputStream FileStatus Directories Delete Files C API libhdfs Chapter 6: Yet Another Resource Negotiator YARN Architecture YARN Process Flow YARN Failures YARN High Availability YARN Schedulers The Fair Scheduler The Capacity Scheduler The YARN Timeline Server Application Timeline Server (ATS) ATS Data Model Structure ATS V2 YARN Federation Chapter 7: MapReduce MapReduce Process Key Features Different Phases in the MapReduce Process MapReduce Architecture MapReduce Sample Program MapReduce Composite Key Operation Mapper Program MapReduce Configuration Chapter 8: Hive Hive History Hive Query Data Storage Data Model Complex Data Types Hive DDL (Data Definition Language) Tables View Partition Bucketing Hive Architecture Serialization/Deserialization (SerDe) Metastore Query Compiler HiveServer2 Chapter 9: Getting Started with Hive Hive Set-up Hive Configuration Settings Loading and Inserting Data into Tables Insert from a Select Query Load Table Data into File Create and Load Data into a Table Hive Transactions Enable Transactions Insert Values Update Delete Merge Locks Hive Select Query Select Basic Query Hive QL File Hive Select on Complex Datatypes Order By and Sort By Distribute By and Cluster By Group By and Having Built-in Aggregate Functions Enhanced Aggregation Table-Generating Functions Built-In Utility Functions Collection Functions Date Functions Conditional Functions String Functions Hive Query Language-Join Chapter 10: File Format File Format Characteristics Columnar Format Schema Evolution Splittable Compression File Formats RC (Row-Columnar) File Input Format Optimized Row Columnar (ORC) File Format Parquet File Format Comparisons ORC vs. Parquet Chapter 11: Data Compression Data Compression Benefits Data Compression in Hadoop Splitting Compression Codec Data Compressions References Index This Book Is The Basic Guide For Developers, Architects, Engineers, And Anyone Who Wants To Start Leveraging The Open-source Software Hadoop And Hive To Build Distributed, Scalable Concurrent Big Data Applications. Hive Will Be Used For Reading, Writing, And Managing The Large, Data Set Files. The Book Is A Concise Guide On Getting Started With An Overall Understanding On Apache Hadoop And Hive And How They Work Together To Speed Up Development With Minimal Effort. It Will Refer To Simple Concepts And Examples, As They Are Likely To Be The Best Teaching Aids. It Will Explain The Logic, Code, And Configurations Needed To Build A Successful, Distributed, Concurrent Application, As Well As The Reason Behind Those Decisions. Features: Shows How To Leverage The Open-source Software Hadoop And Hive To Build Distributed, Scalable, Concurrent Big Data Applications Includes Material On Hive Architecture With Various Storage Types And The Hive Query Language Features A Chapter On Big Data And How Hadoop Can Be Used To Solve The Changes Around It Explains The Basic Hadoop Setup, Configuration, And Optimization This book is the basic guide for developers, architects, engineers, and anyone who wants to start leveraging the open-source software Hadoop and Hive to build distributed, scalable concurrent big data applications. -- Edited summary from book