کلاسترهای داده بزرگ SQL Server: مجازی‌سازی داده، دریاچه داده و پلتفرم هوش مصنوعی

Front Matter ....Pages i-xvii What Are Big Data Clusters? (Benjamin Weissman, Enrico van de Laar)....Pages 1-10 Big Data Cluster Architecture (Benjamin Weissman, Enrico van de Laar)....Pages 11-32 Deployment of Big Data Clusters (Benjamin Weissman, Enrico van de Laar)....Pages 33-83 Loading Data into Big Data Clusters (Benjamin Weissman, Enrico van de Laar)....Pages 85-104 Querying Big Data Clusters Through T-SQL (Benjamin Weissman, Enrico van de Laar)....Pages 105-145 Working with Spark in Big Data Clusters (Benjamin Weissman, Enrico van de Laar)....Pages 147-202 Machine Learning on Big Data Clusters (Benjamin Weissman, Enrico van de Laar)....Pages 203-224 Create and Consume Big Data Cluster Apps (Benjamin Weissman, Enrico van de Laar)....Pages 225-238 Maintenance of Big Data Clusters (Benjamin Weissman, Enrico van de Laar)....Pages 239-252 Back Matter ....Pages 253-260 Table of Contents 5 About the Authors 9 About the Technical Reviewer 10 Acknowledgments 11 Introduction 12 Chapter 1: What Are Big Data Clusters? 15 What Is a SQL Server 2019 Big Data Cluster Really? 16 Data Virtualization 16 Outsource Your Data 20 Reduce Data Redundancy and Development Time 20 A Combined Data Platform Environment 21 Centralized AI Platform 23 Chapter 2: Big Data Cluster Architecture 25 Physical Big Data Cluster Infrastructure 25 Containers 25 SQL Server on Linux 29 Spark 32 HDFS 36 Tying the Physical Infrastructure Parts Together 37 Logical Big Data Cluster Architecture 39 Control Plane 41 Compute Area 42 Data Area 42 Storage Pool 43 SQL Data Pool 44 Summary 45 Chapter 3: Deployment of Big Data Clusters 47 A Little Helper: Chocolatey 47 Installation of an On-Premises PolyBase Instance 49 Using Azure Data Studio to Work with Big Data Clusters 65 What Is Azure Data Studio? 66 Getting and Installing Azure Data Studio 66 Installation of a “Real” Big Data Cluster 67 kubeadm on Linux 67 Azure Kubernetes Service (AKS) 70 Deploy Your Big Data Cluster Through Azure Data Studio 77 What Is azdata? 86 Others 95 Advanced Deployment Options 95 Active Directory Authentication for Big Data Clusters 96 HDFS Tiering in Big Data Clusters 96 Summary 97 Chapter 4: Loading Data into Big Data Clusters 98 Getting Azure Data Studio Fully Ready for Your Big Data Clusters 98 Getting Some Sample Files into the Installation 102 Empty Database 102 Sample Data Within Your Big Data Cluster 102 Restoring Any SQL Server Backup to Your Master Instance 102 Microsoft Sample Data 103 Flight Delay Sample Dataset 104 Azure SQL Database 108 Summary 117 Chapter 5: Querying Big Data Clusters Through T-SQL 118 External Tables 118 Automated External Tables with Biml 131 External Tables from CSV Files in HDFS 136 Accessing Data in an Azure Blob Storage 151 External Tables from Other Data Sources 151 File-Based Data Sources 151 ODBC 153 Others 153 The SqlDataPool 154 Indexes on the SqlDataPool 156 Summary 158 Chapter 6: Working with Spark in Big Data Clusters 159 Loading Data and Creating a Spark Notebook 160 Working with Spark Data Frames 163 More Advanced Data Frame Handling 173 Working with SQL Queries on Spark Data Frames 185 Reading Data from the SQL Server Master Instance 187 Plotting Graphs 190 Data Frame Execution 200 Data Frame Caching 202 Data Frame Partitioning 210 Summary 214 Chapter 7: Machine Learning on Big Data Clusters 215 SQL Server In-Database Machine Learning Services 216 Training Machine Learning Models in the SQL Server Master Instance 217 Scoring Data Using In-Database Machine Learning Models 225 Machine Learning in Spark 229 Summary 236 Chapter 8: Create and Consume Big Data Cluster Apps 237 Create a Big Data Cluster App 238 Consume Big Data Cluster Apps Through REST API 245 Summary 250 Chapter 9: Maintenance of Big Data Clusters 251 Checking the Status of a Big Data Cluster 251 Retrieving a Big Data Cluster’s Status Using azdata 251 Manage a Big Data Cluster Using ADS 253 Metrics (Grafana) 256 Node Metrics 256 SQL Metrics 257 Log Search Analytics (Kibana) 258 Troubleshooting Big Data Clusters 259 Upgrading Big Data Clusters 261 Removing a Big Data Cluster Instance 263 Summary 264 Index 265 Use this guide to one of SQL Server 2019’s most impactful features—Big Data Clusters. You will learn about data virtualization and data lakes for this complete artificial intelligence (AI) and machine learning (ML) platform within the SQL Server database engine. You will know how to use Big Data Clusters to combine large volumes of streaming data for analysis along with data stored in a traditional database. For example, you can stream large volumes of data from Apache Spark in real time while executing Transact-SQL queries to bring in relevant additional data from your corporate, SQL Server database. Filled with clear examples and use cases, this book provides everything necessary to get started working with Big Data Clusters in SQL Server 2019. You will learn about the architectural foundations that are made up from Kubernetes, Spark, HDFS, and SQL Server on Linux. You then are shown how to configure and deploy Big Data Clusters in on-premises environments or in the cloud. Next, you are taught about querying. You will learn to write queries in Transact-SQL—taking advantage of skills you have honed for years—and with those queries you will be able to examine and analyze data from a wide variety of sources such as Apache Spark. Through the theoretical foundation provided in this book and easy-to-follow example scripts and notebooks, you will be ready to use and unveil the full potential of SQL Server 2019: combining different types of data spread across widely disparate sources into a single view that is useful for business intelligence and machine learning analysis. **What You Will Learn*** Install, manage, and troubleshoot Big Data Clusters in cloud or on-premise environments * Analyze large volumes of data directly from SQL Server and/or Apache Spark * Manage data stored in HDFS from SQL Server as if it were relational data * Implement advanced analytics solutions through machine learning and AI * Expose different data sources as a single logical source using data virtualization **Who This Book Is For** Data engineers, data scientists, data architects, and database administrators who want to employ data virtualization and big data analytics in their environments