Python Data Analysis: Perform data collection, data processing, wrangling, visualization, and model building using Python, 3rd Edition

Understand data analysis pipelines using machine learning algorithms and techniques with this practical guideKey Features\* Prepare and clean your data to use it for exploratory analysis, data manipulation, and data wrangling\* Discover supervised, unsupervised, probabilistic, and Bayesian machine learning methods\* Get to grips with graph processing and sentiment analysisBook DescriptionData analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines.Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask.By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data.What you will learn\* Explore data science and its various process models\* Perform data manipulation using NumPy and pandas for aggregating, cleaning, and handling missing values\* Create interactive visualizations using Matplotlib, Seaborn, and Bokeh\* Retrieve, process, and store data in a wide range of formats\* Understand data preprocessing and feature engineering using pandas and scikit-learn\* Perform time series analysis and signal processing using sunspot cycle data\* Analyze textual data and image data to perform advanced analysis\* Get up to speed with parallel computing using DaskWho this book is forThis book is for data analysts, business analysts, statisticians, and data scientists looking to learn how to use Python for data analysis. Students and academic faculties will also find this book useful for learning and teaching Python data analysis using a hands-on approach. A basic understanding of math and working knowledge of the Python programming language will help you get started with this book. Cover......Page 1 Title Page......Page 2 Copyright and Credits......Page 3 About Packt......Page 4 Contributors......Page 5 Table of Contents......Page 7 Preface......Page 15 Section 1: Foundation for Data Analysis......Page 20 Chapter 1: Getting Started with Python Libraries......Page 21 Understanding data analysis......Page 22 The standard process of data analysis......Page 23 The KDD process......Page 24 SEMMA ......Page 25 CRISP-DM......Page 26 The roles of data analysts and data scientists......Page 28 The skillsets of data analysts and data scientists......Page 29 Python installation and setup on Windows......Page 31 Python installation and setup on Mac OS X with brew......Page 32 Software used in this book......Page 33 Using IPython as a shell......Page 34 Reading manual pages......Page 37 Using JupyterLab......Page 38 Using Jupyter Notebooks......Page 40 Advanced features of Jupyter Notebooks......Page 41 Keyboard shortcuts......Page 42 Installing other kernels......Page 43 Extensions for Notebook......Page 44 Summary......Page 50 Chapter 2: NumPy and pandas......Page 51 Understanding NumPy arrays......Page 52 Array features......Page 55 Selecting array elements......Page 56 NumPy array numerical data types......Page 57 dtype objects......Page 59 Data type character codes......Page 60 dtype attributes......Page 61 Manipulating array shapes......Page 62 The stacking of NumPy arrays......Page 64 Partitioning NumPy arrays......Page 67 Changing the data type of NumPy arrays......Page 69 Creating NumPy views and copies......Page 70 Slicing NumPy arrays......Page 72 Boolean and fancy indexing......Page 74 Broadcasting arrays......Page 75 Creating pandas DataFrames......Page 77 Understanding pandas Series......Page 79 Reading and querying the Quandl data......Page 82 Describing pandas DataFrames......Page 86 Grouping and joining pandas DataFrame......Page 89 Working with missing values......Page 93 Creating pivot tables......Page 95 Dealing with dates......Page 97 References......Page 99 Chapter 3: Statistics......Page 100 Types of attributes......Page 101 Mean......Page 103 Mode......Page 104 Measuring dispersion......Page 105 Skewness and kurtosis......Page 109 Understanding relationships using covariance and correlation coefficients......Page 110 Pearson's correlation coefficient......Page 111 Central limit theorem......Page 112 Collecting samples......Page 114 Performing parametric tests......Page 115 Performing non-parametric tests ......Page 121 Summary......Page 127 Chapter 4: Linear Algebra......Page 128 Fitting to polynomials with NumPy......Page 129 Finding the rank of a matrix......Page 131 Matrix inverse using NumPy......Page 132 Solving linear equations using NumPy......Page 133 Decomposing a matrix using SVD......Page 134 Eigenvectors and Eigenvalues using NumPy......Page 136 Generating random numbers......Page 137 Binomial distribution......Page 138 Normal distribution......Page 140 Testing normality of data using SciPy......Page 141 Creating a masked array using the numpy.ma subpackage......Page 145 Summary......Page 147 Section 2: Exploratory Data Analysis and Data Cleaning......Page 148 Technical requirements......Page 149 Visualization using Matplotlib......Page 150 Accessories for charts......Page 151 Scatter plot......Page 153 Line plot......Page 154 Pie plot......Page 156 Bar plot......Page 157 Histogram plot......Page 158 Bubble plot......Page 160 pandas plotting......Page 162 Advanced visualization using the Seaborn package......Page 164 lm plots......Page 165 Bar plots......Page 168 Distribution plots......Page 169 Box plots......Page 170 KDE plots......Page 171 Violin plots......Page 172 Count plots......Page 173 Joint plots......Page 175 Heatmaps......Page 176 Pair plots......Page 178 Plotting a simple graph......Page 180 Glyphs......Page 182 Layouts......Page 183 Nested layout using row and column layouts......Page 187 Multiple plots......Page 189 Hide click policy......Page 191 Mute click policy......Page 193 Annotations......Page 194 Hover tool......Page 197 Widgets......Page 198 Tab panel......Page 199 Slider......Page 200 Summary......Page 203 Chapter 6: Retrieving, Processing, and Storing Data......Page 204 Reading and writing CSV files with NumPy......Page 205 Reading and writing CSV files with pandas......Page 206 Reading and writing data from Excel......Page 208 Reading and writing data from JSON......Page 209 Reading and writing data from HDF5......Page 210 Reading and writing data from HTML tables......Page 211 Reading and writing data from Parquet......Page 212 Reading and writing data from a pickle pandas object......Page 213 Lightweight access with sqllite3......Page 214 Reading and writing data from MySQL......Page 215 Inserting a whole DataFrame into the database......Page 218 Reading and writing data from MongoDB......Page 219 Reading and writing data from Cassandra......Page 220 Reading and writing data from Redis......Page 221 PonyORM......Page 222 Summary......Page 223 Chapter 7: Cleaning Messy Data......Page 224 Exploring data......Page 225 Filtering data to weed out the noise......Page 228 Column-wise filtration ......Page 229 Row-wise filtration ......Page 231 Handling missing values......Page 234 Filling in a missing value......Page 235 Handling outliers......Page 237 One-hot encoding......Page 240 Label encoding......Page 242 Ordinal encoder......Page 243 Feature scaling......Page 244 Methods for feature scaling......Page 245 Feature transformation......Page 248 Feature splitting......Page 249 Summary......Page 250 Chapter 8: Signal Processing and Time Series......Page 251 The statsmodels modules......Page 252 Moving averages......Page 253 Window functions......Page 256 Defining cointegration......Page 258 STL decomposition......Page 260 Autocorrelation......Page 262 Autoregressive models......Page 264 ARMA models......Page 268 Generating periodic signals......Page 271 Fourier analysis......Page 273 Spectral analysis filtering......Page 276 Summary......Page 278 Section 3: Deep Dive into Machine Learning......Page 279 Chapter 9: Supervised Learning - Regression Analysis......Page 280 Linear regression......Page 281 Understanding multicollinearity......Page 283 Removing multicollinearity......Page 284 Dummy variables......Page 286 Developing a linear regression model......Page 288 R-squared......Page 290 MAE......Page 291 RMSE......Page 292 Fitting polynomial regression......Page 293 Logistic regression......Page 296 Characteristics of the logistic regression model......Page 298 Advantages and disadvantages of logistic regression......Page 299 Implementing logistic regression using scikit-learn......Page 300 Summary......Page 302 Chapter 10: Supervised Learning - Classification Techniques......Page 303 Classification......Page 304 Naive Bayes classification......Page 306 Decision tree classification......Page 310 KNN classification......Page 313 Terminology......Page 316 Holdout......Page 319 Bootstrap method......Page 320 Confusion matrix......Page 321 Accuracy......Page 324 F-measure......Page 325 ROC curve and AUC......Page 326 Summary......Page 329 Chapter 11: Unsupervised Learning - PCA and Clustering......Page 330 Unsupervised learning......Page 331 Reducing the dimensionality of data......Page 332 PCA......Page 333 Performing PCA......Page 334 Clustering......Page 337 Finding the number of clusters......Page 338 The elbow method......Page 339 The silhouette method......Page 341 Partitioning data using k-means clustering......Page 343 Hierarchical clustering......Page 346 DBSCAN clustering......Page 350 Spectral clustering......Page 352 Evaluating clustering performance......Page 355 The silhouette coefficient......Page 356 The Jaccard score......Page 357 The Fowlkes-Mallows score......Page 358 Summary......Page 361 Section 4: NLP, Image Analytics, and Parallel Computing......Page 362 Chapter 12: Analyzing Textual Data......Page 363 Installing NLTK and SpaCy......Page 364 Text normalization......Page 365 Tokenization......Page 366 Removing stopwords......Page 370 Stemming and lemmatization......Page 372 POS tagging......Page 374 Recognizing entities......Page 375 Dependency parsing......Page 376 Creating a word cloud......Page 377 Bag of Words......Page 379 TF-IDF......Page 380 Sentiment analysis using text classification......Page 381 Classification using BoW......Page 382 Classification using TF-IDF......Page 387 Text similarity......Page 390 Jaccard similarity......Page 391 Cosine similarity......Page 392 Summary......Page 393 Chapter 13: Analyzing Image Data......Page 394 Installing OpenCV......Page 395 Binary images......Page 396 Color images......Page 397 Color models......Page 398 Drawing on images......Page 401 Writing on images......Page 406 Resizing images......Page 407 Flipping images......Page 409 Changing the brightness......Page 412 Blurring an image......Page 413 Face detection......Page 417 Summary......Page 421 Chapter 14: Parallel Computing Using Dask......Page 422 Parallel computing using Dask......Page 423 Dask data types......Page 424 Dask Arrays......Page 425 Dask DataFrames......Page 426 DataFrame Indexing......Page 427 Filter data......Page 430 Groupby......Page 431 Converting a Dask DataFrame into a pandas DataFrame......Page 432 Creating a Dask Bag using Python iterable items......Page 433 Creating a Dask Bag using a text file......Page 434 Storing a Dask Bag in a DataFrame......Page 435 Dask Delayed......Page 436 Feature scaling in Dask......Page 438 Feature encoding in Dask......Page 440 Machine learning at scale......Page 442 Parallel computing using scikit-learn......Page 443 Logistic regression......Page 445 Clustering......Page 447 Summary......Page 449 Other Books You May Enjoy......Page 451 Index......Page 454 Reinforce your understanding of data science and data analysis from a statistical perspective to extract meaningful insights from your data using Python programmingKey FeaturesWork your way through the entire data analysis pipeline with statistics concerns in mind to make reasonable decisionsUnderstand how various data science algorithms functionBuild a solid foundation in statistics for data science and machine learning using Python-based examplesBook DescriptionStatistics remain the backbone of modern analysis tasks, helping you to interpret the results produced by data science pipelines. This book is a detailed guide covering the math and various statistical methods required for undertaking data science tasks. The book starts by showing you how to preprocess data and inspect distributions and correlations from a statistical perspective. You'll then get to grips with the fundamentals of statistical analysis and apply its concepts to real-world datasets. As you advance, you'll find out how statistical concepts emerge from different stages of data science pipelines, understand the summary of datasets in the language of statistics, and use it to build a solid foundation for robust data products such as explanatory models and predictive models. Once you've uncovered the working mechanism of data science algorithms, you'll cover essential concepts for efficient data collection, cleaning, mining, visualization, and analysis. Finally, you'll implement statistical methods in key machine learning tasks such as classification, regression, tree-based methods, and ensemble learning. By the end of this Essential Statistics for Non-STEM Data Analysts book, you'll have learned how to build and present a self-contained, statistics-backed data product to meet your business goals.What you will learnFind out how to grab and load data into an analysis environmentPerform descriptive analysis to extract meaningful summaries from dataDiscover probability, parameter estimation, hypothesis tests, and experiment design best practicesGet to grips with resampling and bootstrapping in PythonDelve into statistical tests with variance analysis, time series analysis, and A/B test examplesUnderstand the statistics behind popular machine learning algorithmsAnswer questions on statistics for data scientist interviewsWho this book is forThis book is an entry-level guide for data science enthusiasts, data analysts, and anyone starting out in the field of data science and looking to learn the essential statistical concepts with the help of simple explanations and examples. If you're a developer or student with a non-mathematical background, you'll find this book useful. Working knowledge of the Python programming language is required. Understand data analysis concepts to make accurate decisions based on data using Python programming and Jupyter NotebookKey FeaturesFind out how to use Python code to extract insights from data using real-world examplesWork with structured data and free text sources to answer questions and add value using dataPerform data analysis from scratch with the help of clear explanations for cleaning, transforming, and visualizing dataBook DescriptionData literacy is the ability to read, analyze, work with, and argue using data. Data analysis is the process of cleaning and modeling your data to discover useful information. This book combines these two concepts by sharing proven techniques and hands-on examples so that you can learn how to communicate effectively using data.After introducing you to the basics of data analysis using Jupyter Notebook and Python, the book will take you through the fundamentals of data. Packed with practical examples, this guide will teach you how to clean, wrangle, analyze, and visualize data to gain useful insights, and you'll discover how to answer questions using data with easy-to-follow steps.Later chapters teach you about storytelling with data using charts, such as histograms and scatter plots. As you advance, you'll understand how to work with unstructured data using natural language processing (NLP) techniques to perform sentiment analysis. All the knowledge you gain will help you discover key patterns and trends in data using real-world examples. In addition to this, you will learn how to handle data of varying complexity to perform efficient data analysis using modern Python libraries.By the end of this book, you'll have gained the practical skills you need to analyze data with confidence.What you will learnUnderstand the importance of data literacy and how to communicate effectively using dataFind out how to use Python packages such as NumPy, pandas, Matplotlib, and the Natural Language Toolkit (NLTK) for data analysisWrangle data and create DataFrames using pandasProduce charts and data visualizations using time-series datasetsDiscover relationships and how to join data together using SQLUse NLP techniques to work with unstructured data to create sentiment analysis modelsDiscover patterns in real-world datasets that provide accurate insightsWho this book is forThis book is for aspiring data analysts and data scientists looking for hands-on tutorials and real-world examples to understand data analysis concepts using SQL, Python, and Jupyter Notebook. Anyone looking to evolve their skills to become data-driven personally and professionally will also find this book useful. No prior knowledge of data analysis or programming is required to get started with this book. Understand data analysis pipelines using machine learning algorithms and techniques with this practical guide Key Features Prepare and clean your data to use it for exploratory analysis, data manipulation, and data wrangling Discover supervised, unsupervised, probabilistic, and Bayesian machine learning methods Get to grips with graph processing and sentiment analysis Book Description Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data. What you will learn Explore data science and its various process models Perform data manipulation using NumPy and pandas for aggregating, cleaning, and handling missing values Create interactive visualizations using Matplotlib, Seaborn, and Bokeh Retrieve, process, and store data in a wide range of formats Understand data preprocessing and feature engineering using pandas and scikit-learn Perform time series analysis and signal processing using sunspot cycle data Analyze textual data and image data to perform advanced analysis Get up to speed with parallel computing using Dask Who this book is for This book is for data analysts, business analysts, statisticians, and data scientists looking .. Reinforce your understanding of data science and data analysis from a statistical perspective to extract meaningful insights from your data using Python programming Key Features Work your way through the entire data analysis pipeline with statistics concerns in mind to make reasonable decisions Understand how various data science algorithms function Build a solid foundation in statistics for data science and machine learning using Python-based examples Book Description Statistics remain the backbone of modern analysis tasks, helping you to interpret the results produced by data science pipelines. This book is a detailed guide covering the math and various statistical methods required for undertaking data science tasks. The book starts by showing you how to preprocess data and inspect distributions and correlations from a statistical perspective. You'll then get to grips with the fundamentals of statistical analysis and apply its concepts to real-world datasets. As you advance, you'll find out how statistical concepts emerge from different stages of data science pipelines, understand the summary of datasets in the language of statistics, and use it to build a solid foundation for robust data products such as explanatory models and predictive models. Once you've uncovered the working mechanism of data science algorithms, you'll cover essential concepts for efficient data collection, cleaning, mining, visualization, and analysis. Finally, you'll implement statistical methods in key machine learning tasks such as classification, regression, tree-based methods, and ensemble learning. By the end of this Essential Statistics for Non-STEM Data Analysts book, you'll have learned how to build and present a self-contained, statistics-backed data product to meet your business goals. What you will learn Find out how to grab and load data into an analysis environment Perform descriptive analysis to extract meaningful summaries from data Discover probability, parameter estimation, hypothesis tests, and experiment design best practices Get to grips with resampling and bootstrapping in Python Delve into statistical tests with variance analysis, time series analysis, and A/B test examples Understand the statistics behind popular machine learning algorithms Answer questions on statistics for data scientist interviews Who this book is for This book is an entry-level guide for data science enthusiasts, data analysts, and anyone starting out in the field of data sc.. Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you’ll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you’ll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You’ll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you’ll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you’ll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you’ll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data. See Also: • https://github.com/PacktPublishing/Python-Data-Science-Essentials-Third-Edition Data analysis enables one to generate value from small and big data by discovering new patterns and trends. Python is a popular tool for analyzing a wide variety of data. This books instructs how to get up and running with using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines The book will take you on a journey through the evolution of data analysis explaining each step in the process in a very simple and easy to understand manner. You will learn how to use various Python libraries to work with data. Learn how to sift through the many different types of data, clean it, and analyze it to gain useful insights. Put your data science knowledge to work with this practical guide to statistics. You'll understand the working mechanism of each method used and find out how data science algorithms function. This book will help you learn the statistical techniques required for key model building and functioning using Python.