Advanced Analytics with Transact-SQL : Exploring Hidden Patterns and Rules in Your Data

Table of Contents 4 About the Author 9 About the Technical Reviewer 10 Acknowledgments 11 Introduction 12 Part I: Statistics 17 Chapter 1: Descriptive Statistics 18 Variable Types 18 Demo Data 19 Frequency Distribution of Discrete Variables 23 Frequencies of Nominals 23 Frequencies of Ordinals 25 Descriptive Statistics for Continuous Variables 29 Centers of a Distribution 29 Measuring the Spread 34 Skewness and Kurtosis 39 Conclusion 43 Chapter 2: Associations Between Pairs of Variables 45 Associations Between Continuous Variables 49 Covariance 49 Correlation 52 Interpreting the Correlation 55 Associations Between Discrete Variables 57 Contingency Tables 58 Chi-Squared Test 65 Associations Between Discrete and Continuous Variables 70 Testing Continuous Variable Moments over a Discrete Variable 71 Analysis of Variance 72 Definite Integration 77 Conclusion 80 Part II: Data Preparation and Quality 81 Chapter 3: Data Preparation 82 Dealing with Missing Values 83 NULLs in T-SQL Functions 84 Handling NULLs 86 String Operations 92 Scalar String Functions 92 Aggregating and Splitting Strings 94 Derived Variables and Grouping Sets 97 Adding Computed Columns 98 Efficient Grouping 100 Data Normalization 104 Range and Z-score Normalization 104 Logistic and Hyperbolic Tangent Normalization 107 Recoding Variables 110 Converting Strings to Numerics 110 Discretizing Numerical Variables 113 Conclusion 118 Chapter 4: Data Quality and Information 119 Data Quality 120 Measuring Completeness 122 Finding Inaccurate Data 123 Measuring Data Quality over Time 130 Measuring the Information 135 Introducing Entropy 135 Mutual Information 139 Conditional Entropy 142 Conclusion 145 Part III: Dealing with Time 146 Chapter 5: Time-Oriented Data 147 Application and System Times 147 Inclusion Constraints 148 Demo Data 149 System-Versioned Tables and Issues 154 A Quick Introduction to System-Versioned Tables 154 Querying System-Versioned Tables Surprises 161 Optimizing Temporal Queries 175 Modifying the Filter Predicate 179 Using the Unpacked Form 182 Time Series 184 Moving Averages 186 Conclusion 190 Chapter 6: Time-Oriented Analyses 192 Demo Data 192 Exponential Moving Average 195 Calculating EMA Efficiently 197 Forecasting with EMA 200 ABC Analysis 203 Relational Division 204 Top Customers and Products 208 Duration of Loyalty 211 Survival Analysis 213 Hazard Analysis 217 Conclusion 220 Part IV: Data Science 221 Chapter 7: Data Mining 222 Demo Data 222 Linear Regression 225 Autoregression and Forecasting 227 Association Rules 232 Starting from the Negative Side 233 Frequency of Itemsets 236 Association Rules 239 Look-Alike Modeling 248 Training and Test Data Sets 249 Performing Predictions with LAM 257 Naïve Bayes 259 Training the NB Model 260 Performing Predictions with NB 262 Conclusion 268 Chapter 8: Text Mining 270 Demo Data 270 Introducing Full-Text Search 274 Full-Text Predicates 276 Full-Text Functions 279 Statistical Semantic Search 282 Quantitative Analysis 286 Analysis of Letters 286 Word Length Analysis 288 Advanced Analysis of Text 292 Term Extraction 292 Words Associations 294 Association Rules with Many Items 297 Conclusion 302 Index 303 Learn about business intelligence (BI) features in T-SQL and how they can help you with data science and analytics efforts without the need to bring in other languages such as R and Python. This book shows you how to compute statistical measures using your existing skills in T-SQL. You will learn how to calculate descriptive statistics, including centers, spreads, skewness, and kurtosis of distributions. You will also learn to find associations between pairs of variables, including calculating linear regression formulas and confidence levels with definite integration. No analysis is good without data quality. Advanced Analytics with Transact-SQL introduces data quality issues and shows you how to check for completeness and accuracy, and measure improvements in data quality over time. The book also explains how to optimize queries involving temporal data, such as when you search for overlapping intervals. More advanced time-oriented information in the book includes hazard and survival analysis. Forecasting with exponential moving averages and autoregression is covered as well. Every web/retail shop wants to know the products customers tend to buy together. Trying to predict the target discrete or continuous variable with few input variables is important for practically every type of business. This book helps you understand data science and the advanced algorithms use to analyze data, and terms such as data mining, machine learning, and text mining. Key to many of the solutions in this book are T-SQL window functions. Author Dejan Sarka demonstrates efficient statistical queries that are based on window functions and optimized through algorithms built using mathematical knowledge and creativity. The formulas and usage of those statistical procedures are explained so you can understand and modify the techniques presented. T-SQL is supported in SQL Server, Azure SQL Database, and in Azure Synapse Analytics. There are so many BI features in T-SQL that it might become your primary analytic database language. If you want to learn how to get information from your data with the T-SQL language that you already are familiar with, then this is the book for you. What You Will Learn Describe distribution of variables with statistical measures Find associations between pairs of variables Evaluate the quality of the data you are analyzing Perform time-series analysis on your data Forecast values of a continuous variable Perform market-basket analysis to predict customer purchasing patterns Predict target variable outcomes from one or more input variables Categorize passages of text by extracting and analyzing keywords Who This Book Is For Database developers and database administrators who want to translate their T-SQL skills into the world of business intelligence (BI) and data science. For readers who want to analyze large amounts of data efficiently by using their existing knowledge of T-SQL and Microsoft's various database platforms such as SQL Server and Azure SQL Database. Also for readers who want to improve their querying by learning new and original optimization techniques. Learn about business intelligence (BI) features in T-SQL and how they can help you with data science and analytics efforts without the need to bring in other languages such as R and Python. This book shows you how to compute statistical measures using your existing skills in T-SQL. You will learn how to calculate descriptive statistics, including centers, spreads, skewness, and kurtosis of distributions. You will also learn to find associations between pairs of variables, including calculating linear regression formulas and confidence levels with definite integration. No analysis is good without data quality. Advanced Analytics with Transact-SQL introduces data quality issues and shows you how to check for completeness and accuracy, and measure improvements in data quality over time. The book also explains how to optimize queries involving temporal data, such as when you search for overlapping intervals. More advanced time-oriented information in the book includes hazard and survival analysis. Forecasting with exponential moving averages and autoregression is covered as well. Every web/retail shop wants to know the products customers tend to buy together. Trying to predict the target discrete or continuous variable with few input variables is important for practically every type of business. This book helps you understand data science and the advanced algorithms use to analyze data, and terms such as data mining, machine learning, and text mining. Key to many of the solutions in this book are T-SQL window functions. Author Dejan Sarka demonstrates efficient statistical queries that are based on window functions and optimized through algorithms built using mathematical knowledge and creativity. The formulas and usage of those statistical procedures are explained so you can understand and modify the techniques presented. T-SQL is supported in SQL Server, Azure SQL Database, and in Azure Synapse Analytics. There are so many BI features in T-SQL that it might become your primary analytic database language. If you want to learn how to get information from your data with the T-SQL language that you already are familiar with, then this is the book for you. You will learn to: Describe distribution of variables with statistical measures Find associations between pairs of variables Evaluate the quality of the data you are analyzing Perform time-series analysis on your data Forecast values of a continuous variable Perform market-basket analysis to predict customer purchasing patterns Predict target variable outcomes from one or more input variables Categorize passages of text by extracting and analyzing keywords