Statistics Using Python

This book is designed to offer a fast-paced yet thorough introduction to essential statistical concepts using Python code samples, and aims to assist data scientists in their daily endeavors. The ability to extract meaningful insights from data requires a deep understanding of statistics. The book ensures that each topic is introduced with clarity, followed by executable Python code samples that can be modified and applied according to individual needs. Topics include working with data and exploratoryanalysis, the basics of probability, descriptive and inferential statistics and their applications, metrics for data analysis, probability distributions, hypothesis testing, and more. Appendices on Python and Pandas have been included. From foundational Python concepts to the intricacies of statistics, this book serves as a comprehensive resource for both beginners and seasoned professionals. Statistics Using Python is designed to offer a fast-paced yet thorough introduction to essential statistical concepts using Python code samples, aiming to assist data scientists in their daily endeavors. While the book casts a wide net to cater to a broad audience, it ensures that each topic is introduced with clarity, followed by executable Python code samples that can be modified and applied according to individual needs. Features: - Provides Python code samples to ensure readers can immediately apply what they learn - Covers everything from basic data handling to advanced statistical concepts - Features downloadable companion files with code samples and figures Includes two appendices - An Introduction to Python - Introduction to Pandas as refresher material Target Audience: This book primarily targets data scientists and enthusiasts who have a foundational understanding of statistics but wish to delve deeper. Whether you are a beginner wanting to grasp the basics or someone with intermediate knowledge aiming to broaden your statistical horizon, this book offers a structured approach to various concepts. The interleaving of foundational and advanced topics ensures readers can pace their learning according to their comfort and familiarity. Front Cover 1 Half-Title Page 2 LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY 3 Title Page 4 Copyright Page 5 Dedication 6 Contents 8 Preface 12 CHAPTER 1: Working with Data 14 What is Data Literacy? 14 Exploratory Data Analysis (EDA) 15 Dealing with Data: What Can Go Wrong? 18 An Explanation of Data Types 20 Working with Data Types 25 What is Drift? 26 Discrete Data Versus Continuous Data 27 Binning Data Values 28 Correlation 30 Working with Synthetic Data 32 Summary 40 CHAPTER 2: Introduction to Probability 42 What is Set Theory? 42 Open, Closed, Compact, and Convex Sets (Optional) 44 Concepts in Probability 45 Set Theory and Probability 49 Coin Tossing Probabilities 50 Dice Tossing Probabilities 56 Card Drawing Probabilities 60 Container-Based Probabilities 62 Children-Related Probabilities 65 Summary 66 CHAPTER 3: Introduction to Statistics 68 Introduction to Statistics 68 Basic Concepts in Statistics 69 The Variance and Standard Deviation 75 The Moments of a Function (Optional) 77 Random Variables 80 Multiple Random Variables 82 Sampling Techniques for a Population 83 What is Bias? 85 Two Important Results in Probability 86 Summary 87 CHAPTER 4: Metrics in Statistics 88 The Confusion Matrix 88 The ROC Curve and AUC Curve 100 The sklearn.metrics Module (Optional) 105 Statistical Metrics for Categorical Data 106 Metrics for Continuous Data 110 MAE, MSE, and RMSE 113 Approximating Linear Data with np.linspace() 115 Summary 116 CHAPTER 5: Probability Distributions 118 PDF, CDF, and PMF 118 Two Types of Probability Distributions 120 Discrete Probability Distributions 121 Continuous Probability Distributions 129 Advanced Probability Functions 149 Non-Gaussian Distributions 153 The Best-Fitting Distribution for Data 153 Summary 161 CHAPTER 6: Hypothesis Testing 162 What is Hypothesis Testing? 162 Components of Hypothesis Testing 164 Test Statistics 165 Working with p-values 167 Working with Alpha Values 170 Point Estimation, Confidence Level, and Confidence Intervals 170 What is A/B Testing? 174 The Lifespan of an A/B Test 178 Maximum Likelihood Estimation (MLE) 179 Summary 181 Appendix A: Introduction to Python 182 Tools for Python 182 Python Installation 184 Setting the PATH Environment Variable (Windows Only) 185 Launching Python on Your Machine 185 Identifiers 187 Lines, Indentation, and Multi-Line Statements 187 Quotation Marks and Comments 188 Saving Your Code in a Module 190 Some Standard Modules 191 The help() and dir() Functions 191 Compile Time and Runtime Code Checking 193 Simple Data Types 193 Working with Numbers 194 Working with Fractions 198 Unicode and UTF-8 199 Working with Strings 200 Slicing and Splicing Strings 203 Search and Replace a String in Other Strings 205 Remove Leading and Trailing Characters 206 Printing Text without New Line Characters 207 Text Alignment 208 Working with Dates 209 Exception Handling 211 Handling User Input 213 Python and Emojis (Optional) 216 Command-Line Arguments 217 Summary 218 Appendix B: Introduction to Pandas 220 What is Pandas? 220 A Pandas Data Frame with a NumPy Example 223 Describing a Pandas Data Frame 226 Boolean Data Frames 229 Data Frames and Random Numbers 231 Reading CSV Files in Pandas 233 The loc() and iloc() Methods 235 Converting Categorical Data to Numeric Data 235 Matching and Splitting Strings 240 Converting Strings to Dates 243 Working with Date Ranges 245 Detecting Missing Dates 247 Interpolating Missing Dates 249 Other Operations with Dates 252 Merging and Splitting Columns in Pandas 257 Reading HTML Web Pages 260 Saving a Pandas Data Frame as an HTML Web Page 261 Summary 264 Index 266